Senior Software Engineer, Inference

Job Description

Design and implement key components of the LLM runtime, including engine integration, batching, and KV cache management. Partner with engineering teams to optimize inference performance, including latency, throughput, and distributed execution capabilities.

Requirements: Requires at least 8 years of software engineering experience with 1-2+ years specifically in LLM inference runtimes or production model serving. Candidates must possess strong proficiency in Go, Python, Kubernetes, and deep knowledge of inference engine internals.

Key Skills: LLM inference, Kubernetes, Go, Python, C++, CUDA, vLLM, TensorRT-LLM, Continuous batching, KV cache management, Distributed execution, GPU optimization, Tensor parallelism, Pipeline parallelism, Performance profiling

Benefits: Health and wellbeing benefits, Professional development programs, Flexible work arrangements