SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Salary: USD 295,250 - 345,040 / annual
Roblox operates a platform serving 200M+ daily active users. The Data Platform Orchestration team owns the system that schedules and runs the company's data workloads—Apache Airflow and the Kubernetes (EKS) infrastructure beneath it, executing 8,000+ DAGs daily, most of them Spark jobs.
As Principal Software Engineer, you will be the technical leader setting long-term architecture for how Roblox orchestrates data at scale. You will be hands-on in making the platform faster, more reliable, and easier for hundreds of engineers to use.
Key responsibilities:
- Define the multi-year orchestration strategy, including scheduler and executor design, multi-tenancy, upgrades and migrations, and platform evolution roadmap.
- Scale and harden Airflow by removing scaling limits across the scheduler, metadata database, executors, and DAG parsing to ensure thousands of daily DAGs run with predictable latency and strong SLAs.
- Lead the design and operation of EKS clusters for orchestration and Spark workloads, including autoscaling, resource isolation, cost efficiency, and deployment automation.
- Design APIs, custom operators and providers, libraries, and self-service tooling that enable data teams to build, deploy, and monitor pipelines safely.
- Establish SLOs, alerting, and diagnostics for pipelines and platform components; lead incident reviews and fix root causes in code.
- Work across data platform, governance, data science, and product engineering teams to align roadmaps, guide technical decisions, and build consensus on complex initiatives.
- Set standards for code quality, design, and operational readiness; mentor staff, senior, and mid-level engineers.
The role is based in San Mateo with hybrid work: onsite Tuesday, Wednesday, and Thursday, with optional presence on Monday and Friday.
Requirements:
- B.S. equivalent in CS or sufficient equivalent experience.
- 10+ years designing, building, and operating production-grade, large-scale distributed systems.
- Deep, hands-on experience designing, running, and scaling a production data-related workflow orchestrator or job scheduler (thousands of workflows or DAGs daily). Apache Airflow is ideal; equivalent experience with Dagster, Prefect, Temporal, or large in-house schedulers also welcome. Must understand scheduler and executor design, metadata store performance, task isolation, retries and idempotency, and safe upgrades and migrations.
- Proven experience managing and scaling production Kubernetes clusters on AWS (EKS), including Helm, autoscaling (Karpenter or Cluster Autoscaler), and GitOps-style deployment.
- Strong proficiency in Python, plus Go or Java, with a record of writing clean, well-tested platform and tooling code. Must be a software engineer who builds and improves systems, not primarily a ticket/operations handler.
- Working knowledge of how Spark jobs run on Kubernetes (Spark on K8s, spark-operator, or EMR on EKS) and how orchestration interacts with them.
- Experience building platforms used by other engineers, with strong instincts for APIs, observability, reliability, and developer experience.
- Track record of influencing direction across a large engineering organization and driving multi-quarter projects from concept to production.
- Extreme ownership: define problems when requirements are unclear, unblock dependencies, and see complex work through to stable production.
Bonus:
- Contributions to Apache Airflow or related open-source projects (Airflow providers, Helm chart, spark-operator, Karpenter, Argo CD).
- Experience with other orchestration systems (Dagster, Temporal, Argo Workflows, Flyte) and judgment to know when each fits.
- Experience with Terraform or infrastructure-as-code, Prometheus/OpenTelemetry, and cost optimization or chargeback for shared platforms.
- Experience operating infrastructure at consumer-internet scale.
