# Senior Software Engineer, Inference

**Company**: Hewlett Packard Enterprise
**Work arrangement**: hybrid
**Experience**: senior
**Job type**: full-time
**Salary**: $144,000 - $273,000 per year
**Category**: Engineering
**Industry**: Technology

**Apply**: https://hpe.wd5.myworkdayjobs.com/en-US/Jobsathpe/job/Spring-Texas-United-States-of-America/Senior-Software-Engineer--Inference_1211356-2?utm_source=yubhub.co&utm_medium=jobs_feed&utm_campaign=apply
**Canonical**: https://yubhub.co/jobs/job_2be53c1b-53d

## Description

Hewlett Packard Enterprise is seeking a Senior Software Engineer to build and evolve the model runtime within HPE AI Essentials, the inference platform used by enterprises to operate large language models on infrastructure they own.

The Senior Software Engineer will design and implement key components of the runtime – engine integration, batching, KV cache management, and distributed execution – together with the Kubernetes orchestration layer that supports it.

Responsibilities:

- Design, implement, and own major components of the LLM serving deployment, including engine integration, continuous batching, KV cache management and reuse, and quantized execution

- Partner with inference engineering teams and contribute to improving time-to-first-token, inter-token latency, throughput per GPU, and P95/P99 tail latency

- Build and operate distributed execution capabilities, including disaggregated prefill/decode, tensor and pipeline parallelism, and KV cache offload across GPU memory, host memory, and RDMA-attached storage

- Evaluate emerging runtimes, quantization schemes, speculative decoding, and mixture-of-experts serving, and make well-supported recommendations on adoption

- Contribute to the orchestration layer supporting the runtime, including model admission, GPU scheduling and partitioning, cache-aware request routing, and autoscaling

- Triage and resolve customer issues end-to-end, identifying root causes and improving systems and processes to prevent recurrence

- Provide insightful code and design reviews, mentor team members, and lead by example on engineering practices within the team

Knowledge and Skills:

- Familiar with LLM inference engines such as vLLM, SGLang, TensorRT-LLM, TGI, or NVIDIA NIM, including modification of engine internals

- Strong understanding of inference internals, including continuous batching, paged attention, KV cache reuse and prefix caching, chunked prefill, quantization, and speculative decoding

- Working knowledge of tensor and pipeline parallelism, NCCL collective operations, and the GPU memory hierarchy and interconnect characteristics that govern them

- Advanced proficiency in Kubernetes platform architectures, including operators, custom resources, controllers, and scheduling

- Strong programming proficiency in Go and Python, with the ability to read, debug, and profile C++/CUDA using tools such as Nsight

Benefits:

- Comprehensive suite of benefits that supports physical, financial, and emotional wellbeing

- Investment in career development and growth

- Unconditional inclusion and celebration of individual uniqueness

Salary: $144,000 - $273,000 per year (USD)

## Skills

### Required
- LLM inference engines
- Kubernetes platform architectures
- Go programming
- Python programming
- C++/CUDA programming

### Nice to have
- Upstream contribution to vLLM, SGLang, TensorRT-LLM, llm-d, LMCache, or KServe
- Disaggregated prefill/decode serving
- RDMA, GPUDirect Storage, InfiniBand, or RoCE
- MIG, fractional GPU allocation, and multi-tenant GPU isolation

---

Source: [Apply at hpe.wd5.myworkdayjobs.com](https://hpe.wd5.myworkdayjobs.com/en-US/Jobsathpe/job/Spring-Texas-United-States-of-America/Senior-Software-Engineer--Inference_1211356-2?utm_source=yubhub.co&utm_medium=jobs_feed&utm_campaign=apply)