Employer description & source details
Design benchmarks, rubrics and metrics, and choose between human, automated and hybrid evaluation based on reliability, cost and validity trade-offs. Deploy and serve models through APIs and self-hosted GPU inference, with versioned configs, logging and controls for reproducible results at scale. Analyse evaluation data with sound statistical methods (agreement, uncertainty, significance, disaggregation), and validate automated judges against human ground truth. Build and maintain modular evaluation pipelines and leaderboard infrastructure, covering sampling, inference, task packaging, scoring and reporting, with tested, documented code. Prototype new evaluation methods, and turn results into publications, technical reports and public releases. 4+ years in ML engineering or applied ML research, shipping production-quality systems. Strong software engineering, including clean, tested Python and end-to-end data and ML pipelines. Deep working knowledge of modern AI architectures and inference, including quality, latency and cost trade-offs. Rigorous grounding in probability and statistics on noisy real-world data. Ability to independently turn ambiguous questions into rigorous, executable technical work. Master's degree in Computer Science, Machine Learning, NLP or a related field preferred; equivalent industry or research experience will be considered. Experience building benchmarks or evaluation systems. Experience with speech or multilingual NLP, especially Indian languages. Publications or open-source work in AI or related domains. Sceptical of numbers until you know how they were produced. Thrives in interdisciplinary teams spanning engineering, research, social science, operations and domain experts. Comfortable with ambiguity, messy data and evolving requirements, a
Extracted skills: python, machine learning, operations