Senior Software Engineer, Inference

Job Description

Design and implement key components of the LLM runtime, including engine integration, batching, and KV cache management. Collaborate with engineering teams to optimize inference performance, throughput, and latency on customer-owned hardware.

Requirements: Requires at least 8 years of software engineering experience with 1-2+ years specifically in LLM inference runtimes or production model serving. Proficiency in Go, Python, Kubernetes, and deep knowledge of inference engine internals are essential.

Key Skills: LLM inference, Kubernetes, Go, Python, C++, CUDA, vLLM, TensorRT-LLM, Continuous batching, KV cache management, Distributed execution, Tensor parallelism, Pipeline parallelism, GPU utilization, Profiling, Debugging

Benefits: Health & Wellbeing, Personal & Professional Development