Description
Join Kobie's India Tech Hub as a Lead AI QA Engineer. You'll design test strategies and evaluation harnesses for production-grade, agentic AI systems, and work with ETL. You need 6+ years of experience, strong Python skills, and hands-on experience testing LLM-powered features.
Responsibilities:
- Design and build evaluation harnesses for agentic systems in Python
- Author automated test suites for prompts, tools, structured outputs, and end-to-end agent workflows
- Validate guardrails around tool execution
- Wire evaluations into CI using Dataiku Evaluations, GitHub Actions, or Jenkins
- Build observability into testing
- Own quality end-to-end
- Partner with data engineers and platform teams
- Help shape internal QA standards for AI & Data engineering
Requirements:
- 3+ years of professional QA / SDET experience
- 1+ years of hands-on experience testing LLM or AI features in production
- Working knowledge of evaluation frameworks such as RAGAS, DeepEval, LangSmith, Langfuse
- Strong Python and PyTest skills
- Solid SQL skills and comfort with at least one cloud platform
- Fluency with Git, Docker, REST APIs, and at least one CI tool
- Solid understanding of data security and responsible AI practices
Nice to Have:
- Hands-on experience with Dataiku DSS
- Experience with Dataiku LLM Mesh, Knowledge Banks, Prompt Studio, or Visual / Code Agents
- Experience with Snowflake, Snowpark, or Snowflake Cortex
- Experience with red-teaming, prompt-injection testing, or adversarial test generation for LLMs