Senior Software Engineer, Inference

Job Description

Design and implement key components of the LLM runtime, including engine integration, batching, and KV cache management. Contribute to distributed execution capabilities and the Kubernetes orchestration layer to ensure high efficiency and low latency.

Requirements: Requires at least 8 years of software engineering experience, with 1-2+ years specifically in LLM inference runtimes or production model serving. Candidates must possess strong proficiency in Go, Python, and Kubernetes, along with deep knowledge of inference internals and GPU memory hierarchies.

Key Skills: LLM inference, Kubernetes, Go, Python, C++, CUDA, vLLM, TensorRT-LLM, Continuous batching, KV cache management, Distributed execution, Tensor parallelism, Pipeline parallelism, GPU scheduling, Profiling

Benefits: Health and wellbeing benefits, Personal and professional development programs, Flexible work arrangements