Employer description & source details
Operate the provisioning state machine in production: own the health of the host lifecycle pipeline, discovery, inference bring-up, GPU driver/CUDA validation, health checks, decommission/RMA, triaging stuck or failed state transitions at fleet scale and feeding real failure patterns back into how the states and transitions are designed. Triage and extend automated self-healing: when the node-scanning and remediation platform misses something, a new failure mode, an InfiniBand issue in disaggregated prefill/decode serving, a GPU Xid variant, a hero-model or LoRA-specific fault, diagnose it live, drive the fix to production, and turn it into a new detection/remediation plugin so it's automatic the next time. Own end-to-end reliability of the pipeline: hold the SLOs and error budgets for provisioning and fleet health, lead incident response when idempotency, retry, rollback, or drift assumptions break down in practice, and drive the systemic fix, not just the hotfix. Partner with the inference/ML platform team from the production side: be the one who knows how real cluster topologies, interconnects, and scheduling constraints actually fail in the field, and feed those failure modes back as first-class requirements into the platform's abstractions. Work alongside machine learning researchers, engineers, and product managers to bring our latest technologies into production, enabling advanced research through awesome engineering. Introduce new techniques, tools, and architecture that improve the performance, latency, throughput, and efficiency of our model inference stack. Build tools to give us visibility into our bottlenecks and sources of instability, then design and implement solutions to address the highest-priority issues. Lead reliability cross-functionally: build coa
Extracted skills: kubernetes, machine learning