SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Salary: USD 173,000 - 255,000 / annual
Affirm is seeking a Senior Software Engineer to join the Workload Safety team, which is responsible for defining and enforcing standards across Affirm's 85+ Amazon EKS clusters. This is a high-impact infrastructure role focused on ensuring workload safety, availability, and cost efficiency at scale.
In this role, you will:
- Define and evolve workload safety standards for Affirm's EKS fleet, including CPU and memory settings, workload isolation and placement, disruption tolerance, and graceful termination patterns.
- Own standards end-to-end: research and prototyping, white-glove onboarding for early adopters, self-service guidance, and automation/policy guardrails to ensure compliance from day one.
- Lead cross-team initiatives from design through fleet-wide rollout, mentor other engineers, and set high bars for design, code review, and operational readiness.
- Isolate workloads by criticality and resource profile to prevent misbehaving workloads from degrading critical paths such as checkout.
- Troubleshoot stuck workloads and node lifecycle problems, including pods that ignore termination signals, pod disruption budgets blocking node cycling, and Karpenter timeout issues.
- Determine safe operating limits for each cluster (pod count, node count, API server request volume) and develop control plane scaling patterns for bursty workloads like scheduled batch jobs and large deployments.
- Analyze workload resource profiles across the fleet (CPU, memory, disk, network) to guide EC2 instance type strategy and automate analysis as workloads evolve.
- Measure each standard's impact on availability, latency, risk, and cost to prioritize the roadmap.
- Participate in on-call for Kubernetes runtime issues and convert recurring failure patterns into roadmap items.
- Collaborate closely with teams running services, batch jobs, and ML workloads on EKS, as well as deployment, capacity, and SRE teams.
- Stay current with Kubernetes community best practices and adapt them to Affirm's needs.
Affirm is a fintech company that provides flexible payment solutions. The engineering team builds large-scale, highly available, global infrastructure shared across multiple financial products, with a strong culture of technical design review, operational excellence, and incident response.
REQUIREMENTS:
- 5+ years of experience running Kubernetes in production at scale, ideally Amazon EKS across many clusters.
- Deep understanding of Kubernetes resource management and scheduling: requests and limits, OOM and CPU throttling behavior, affinity and anti-affinity, taints and tolerations, and topology spread constraints.
- Solid grasp of the pod lifecycle and disruption management, including graceful termination, pod disruption budgets, and node draining.
- Experience enforcing configuration standards through admission control and policy-as-code (e.g., OPA Gatekeeper or Kyverno).
- Proficiency in Go or Python, including building production infrastructure software such as Kubernetes controllers, operators, or admission webhooks.
- Strong production debugging skills and demonstrated ability to turn recurring incidents into systemic fixes.
- Track record of leading technical initiatives across multiple teams, driving adoption of shared standards, and mentoring other engineers.
- Experience quantifying the impact of infrastructure changes on availability, latency, and cost.
- Strong written communication skills, including ability to create standards and guidance that teams can adopt independently.
- Bachelor's degree in a related field or equivalent practical experience.
