Employer description & source details
We are looking for a dedicated ML Model Evals & Benchmarking Engineer to bridge the gap between machine learning research and high-performance production systems. In this role, you will own the end-to-end evaluation lifecycle, designing rigorous offline and online evaluation frameworks, defining robust model performance metrics, and architecting the scalable infrastructure needed to benchmark throughput, latency, resource utilization, and statistical accuracy across diverse ML/AI models. Own model quality before production: Establish standard evaluation frameworks using statistical metrics (e.g., Precision, Recall, F1-Score, ROC-AUC) tailored to specific business domains. Build validation frameworks for computer vision models (preferably satellite or geospatial). Catch regressions in statistical metrics before they reach customers. Build version-aware model regression: Set up pipelines that compare inference outputs across model versions on curated golden datasets. Every release should come with a clear, data-backed go/no-go signal. Make data quality a first-class gate: Validate training and evaluation datasets for label quality, class balance, geographic and seasonal coverage, leakage, and drift. Ensure models are tested on data that reflects real-world conditions. Validate inference at scale: Design, build, and scale distributed evaluation pipelines and testing harnesses capable of running large-scale batch and real-time model benchmarks. Harden the platform end-to-end: Benchmark model inference servers for throughput, latency (p50, p99), and hardware efficiency (GPU/CPU/Memory footprints) under heavy concurrent loads. Software Engineering & Scripting: Advanced proficiency in Python (NumPy, Pandas, Scikit-learn, PyTorch/TensorFlow evaluation utilities) for building cu
Extracted skills: python, go, sql, aws, kubernetes, machine learning