SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Salary: USD 201,300 - 352,300 / annual
Moveworks (ServiceNow) is hiring a Senior Staff Software Engineer to build the runtime infrastructure powering AI agents that orchestrate, execute, and deliver responses to millions of enterprise users in real time. This is a distributed systems engineering role at the core of the agentic AI wave, not an ML role.
You will own and build:
**Agent Orchestration Engine** — A state machine managing long-running agent sessions, coordinating planning, execution, and user interaction across multiple LLM calls and tool invocations.
**Distributed Session Management** — Lease-based ownership using DynamoDB conditional writes, heartbeat protocols, and crash recovery via checkpointing to ensure correctness and resilience.
**Event-Driven Message Pipeline** — SQS FIFO queues for ordered delivery, Kafka consumers for event processing, and real-time streaming via gRPC and Socket.IO.
**Structured Concurrency** — Python asyncio TaskGroups running multiple concurrent tasks per session (message polling, lease heartbeats, output publishing, orchestrator execution) with fail-fast semantics and graceful cancellation.
**Observability Infrastructure** — OpenTelemetry instrumentation, distributed trace context propagation across async boundaries, and custom span lifecycle management for sessions spanning minutes.
**Caching and State Layers** — Redis and DynamoDB KV stores with per-org/per-bot scoping, batch read optimization, and hot-reload configuration.
The role requires comfort with ambiguity—these are novel problems without textbook solutions. You'll design and operate systems handling real traffic at scale, maintaining low latency and high correctness.
**Requirements:**
- 10+ years building production backend/infrastructure systems
- Deep experience in at least 3 of: distributed systems (consistency models, idempotency, exactly-once delivery, distributed locking/leasing); concurrent/async programming (Python asyncio, Go goroutines, structured concurrency, cancellation handling); event-driven architectures (SQS, Kafka, pub/sub, backpressure, delivery guarantees); database systems for infrastructure (DynamoDB conditional writes/transactions, Redis connection pooling/pub/sub); observability (OpenTelemetry, distributed tracing, span context propagation, Prometheus metrics); gRPC/protobuf (streaming RPCs, service interface design, error handling)
- Strong in Python or Go (ideally both)
- Experience designing and operating systems handling real traffic at scale
- Comfort with ambiguity and novel problem-solving
