SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Baseten powers mission-critical inference for leading AI companies including Cursor, Notion, OpenEvidence, Abridge, Clay, Gamma, and Writer. The company recently raised $1.5B in Series F funding and is building the platform engineers rely on to ship AI products at scale.
You will join a small, high-impact team building the distributed runtime that powers large-scale LLM inference. This role spans the full stack—from the developer experience customers use to deploy models, through libraries supporting tool calling and reasoning, down to Kubernetes orchestration and traffic routing systems. Your focus will be ensuring every model on the platform is fast, reliable, and cost-efficient.
Key responsibilities include:
- Building infrastructure and orchestration systems for distributed LLM inference, including routing, autoscaling, scheduling, and runtime management
- Designing and operating Model APIs with advanced inference capabilities: structured outputs (JSON mode, grammar-constrained generation), tool/function calling, and multimodal serving
- Implementing platform fundamentals such as API versioning, validation, usage metering, quotas, and authentication
- Instrumenting deep observability (metrics, traces, logs) and building repeatable benchmarks for speed, reliability, and quality
- Debugging and hardening complex production systems spanning Kubernetes, distributed runtimes, networking, and GPU workloads
- Partnering with Inference Performance engineers to make optimizations broadly available and easy to configure
- Owning projects end-to-end from architecture through deployment, monitoring, and customer feedback iteration
You'll work on initiatives like the Baseten Inference Stack, KV cache-aware routing for 2x faster inference, and multi-cloud capacity management unifying deployments across cloud, self-hosted, and hybrid environments.
REQUIREMENTS:
- Bachelor's, Master's, or Ph.D. in Computer Science, Engineering, or related field, or equivalent practical experience
- 3+ years building and operating distributed systems, backend infrastructure, or large-scale APIs where reliability, latency, and scale are first-class concerns
- Proven track record owning low-latency, reliable backend services including rate limiting, auth, quotas, metering, and migrations
- Infrastructure instincts with profiling, tracing, capacity planning, and SLO management experience
- Comfort debugging performance and reliability issues across multiple stack layers from application to runtime and infrastructure
- Strong sense of developer experience and how systems are used
- Eagerness to learn new languages, frameworks, and systems with real interest in inference engineering (prior ML/LLM experience not required, but model serving or inference systems experience is a plus)
- Excellent written communication and collaboration skills including design doc writing and cross-functional work
NICE TO HAVE:
- Experience with LLM inference engines: vLLM, SGLang, TensorRT-LLM, TGI, or Dynamo
- Deep Kubernetes experience including operators, custom resources, service meshes, or API gateways
- Distributed scheduling, autoscaling, or service orchestration experience
- Production GPU workload operations
- Developer-facing infrastructure or API background, or open-source infrastructure/ML contributions
- Observability tooling, CI/CD systems, or release automation familiarity
