Staff Software Engineer, Model Infrastructure

Job Description

Lead the design and implementation of Harvey's Model Infrastructure platform to ensure high availability and low latency for AI inference. Develop systems for model routing, capacity management, and observability across multiple AI providers.

Requirements: Requires over 7 years of experience building large-scale distributed systems and strong proficiency in languages like Go, Java, Python, Rust, or C++. Candidates must have a deep understanding of cloud infrastructure and experience leading cross-functional technical projects.

Key Skills: Distributed Systems, Go, Java, Python, Rust, C++, Cloud Infrastructure, Networking, Observability, Kubernetes, LLM Serving, Model Routing, Capacity Planning, SRE Best Practices, API Integration, Technical Leadership