Senior ML Infrastructure Engineer

Job Description

Design and operate production-grade inference infrastructure on RDU architecture to optimize performance and cost. Own the public API surface and build accuracy verification systems to ensure the reliability of inference features.

Requirements: Requires a B.S. in Computer Science or a related field and over 5 years of experience building large-scale distributed systems. Proficiency in Python and familiarity with modern LLM serving stacks like vLLM or TensorRT-LLM are essential.

Key Skills: ML Infrastructure, Distributed Systems, LLM Inference, Python, Systems Design, vLLM, Speculative Decoding, Constrained Decoding, API Design, Concurrency, Algorithms, Data Structures, RDU Architecture, Prompt Caching, Long-context Serving, Accuracy Verification

Benefits: Medical Insurance, Dependent Insurance, Health Savings Account (HSA), Dental Insurance, Vision Insurance, Short Term Disability, Long Term Disability, Basic Life Insurance, Voluntary Life Insurance, AD&D Insurance, Flexible Spending Account (FSA), Headspace Subscription, Gympass+ Membership, One Medical Membership, Employee Assistance Program, Equity