IN-Manager_AI-Testing_D&A_ Advisory_Pune

Job Description & Summary: A career within…. A career within Data and Analytics services will provide you with the opportunity to help organizations uncover enterprise insights and drive business results using smarter data analytics. We focus on a collection of organizational technology capabilities, including business intelligence, data management, and data assurance that help our clients drive innovation, growth, and change within their organizations in order to keep up with the changing nature of customers and technology. We make impactful decisions by mixing mind and machine to leverage data, understand and navigate risk, and help our clients gain a competitive edge.  

Responsibilities: 

AI Testing Strategy & Leadership 

• Define and implement enterprise-wide AI testing and quality assurance strategies. 

• Lead a team of SDETs and QA engineers supporting AI/ML and GenAI applications. 

• Establish quality standards, testing frameworks, and governance processes for AI solutions. 

• Drive continuous improvement in test automation, quality metrics, and release processes. 

Test Automation & Framework Development 

• Build and maintain scalable automation frameworks for AI applications and APIs. 

• Develop automated validation pipelines for machine learning models. 

• Integrate automated testing into CI/CD workflows. 

• Design automated regression, functional, integration, performance, and security tests. 

AI/ML Model Validation 

• Validate model accuracy, precision, recall, F1-score, and business outcomes. 

• Test model drift, data drift, concept drift, and retraining workflows. 

• Perform bias, fairness, explainability, and hallucination testing. 

• Validate LLM responses using automated evaluation frameworks. 

Generative AI Quality Engineering 

• Design testing strategies for chatbots, copilots, agents, and RAG applications. 

• Evaluate prompt effectiveness and response quality. 

• Implement automated prompt testing and output validation. 

• Test grounding, toxicity, hallucination, and safety controls. 

Performance & Reliability Testing 

• Conduct load, stress, scalability, and resilience testing for AI systems. 

• Analyze latency, throughput, token consumption, and inference performance. 

• Ensure high availability and reliability of AI services. 

Data Quality Testing 

• Validate data pipelines, feature engineering, and training datasets. 

• Identify data quality issues affecting model performance. 

• Implement automated data validation and monitoring controls. 

DevOps & CI/CD Integration 

• Integrate AI testing practices into DevSecOps pipelines. 

• Automate test execution across cloud environments. 

• Work with engineering teams to enable shift-left quality practices. 

Mandatory skill sets: 

• Playwright 

• Selenium 

• MLflow  

• OpenAI Evaluations 

• CI/CD Pipelines 

Preferred skill sets: 

• Experience testing LLMs, RAG architectures, AI Agents, and Copilot solutions.  

• Knowledge of Responsible AI principles.  

•  Experience with MLOps and ModelOps practices.  

• AI/ML certifications from Microsoft, AWS, Google, or equivalent.  

• Experience in enterprise-scale cloud environments. 

Years of experience required: 

8-12 

Education qualification: 

B.Tech / M.Tech / MBA / MCA