SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
The Analytics Data Platform Batch team at Datadog owns the platform powering batch data processing across the company's internal data infrastructure. The team executes 800k+ Spark jobs daily, along with PySpark, Ray, and Python-based workloads, serving 50+ internal teams across billing, cost management, security intelligence, and applied AI.
As a platform engineering team, your customers are Datadog's internal data engineers and scientists. You'll build the infrastructure, abstractions, and tooling they depend on, operating at the intersection of distributed systems, developer experience, cost engineering, and AI-assisted automation.
Key responsibilities include:
- Designing and implementing AI-driven version migration automation for Spark and JVM systems
- Building agentic tooling that analyzes compatibility, generates pull requests, and orchestrates multi-datacenter testing
- Developing engine-agnostic batch execution capabilities
- Ensuring platform reliability and scale across large workloads
- Implementing Spark Pod Autosizing for optimal CPU and memory allocations
- Analyzing and understanding tradeoffs across different execution models
Datadog operates as a hybrid workplace, valuing office culture for collaboration and creativity while enabling work-life harmony.
REQUIREMENTS:
- 6+ years of software engineering experience
- Significant backend programming experience in one or more of: Go, Java, Rust, Python, Scala
- Deep knowledge of Spark internals and infrastructure (executor lifecycle, memory model, dynamic allocation, resource tuning)
- Demonstrated experience building platform or infrastructure that other engineers depend on
- Distributed systems engineering background with experience building high-performance, large-scale systems
- Kubernetes fundamentals
- Excellent English communication skills
- High proactivity and cross-team collaboration experience
- Excitement about leveraging AI tools to enhance coding, problem-solving, and building (or eagerness to learn)
- Demonstrated ability to use AI coding tools in day-to-day workflows, validate, critique, and refine AI-generated output
- Bonus: AI/LLM integration in engineering workflows; observability and cost attribution experience
