SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Vanta is seeking a Staff Software Engineer to lead the re-platforming of the data layer powering its compliance product. This is a high-impact migration spanning multiple teams that must preserve every public API contract and customer evidence without interruption.
The Foundations team ingests, stores, catalogs, and queries security and compliance data from customer environments—currently processing tens of thousands of resources per second with single-customer bursts reaching millions. The team is migrating from a Mongo-centric architecture to a schema-aware, Postgres-backed architecture on Kafka and S3 pipelines that decouple data fetching from processing. Both pipelines run in parallel today; the core challenges are correctness under migration, eventual consistency, and multi-tenancy in a domain where "mostly right" is unacceptable.
In this role, you will lead the migration of Vanta's resource data model while running both generations in parallel without breaking customer integrations. You'll drive solutions across dependent teams, design for correctness under eventual consistency with idempotent session handling and conditional writes, and evolve the data ingestion and pipeline architecture to reliably process terabyte-scale data streams. You'll take the Query API from internal use to a production platform with schema versioning, joins, exports, per-tenant isolation, and predictable latency. You'll build auditable data infrastructure where correctness can be externally verified, diagnose production-only failures (hot partitions, unbounded fan-out, online table rewrites), and set architectural direction for streaming (Kafka), caching (Redis), and persistent storage (Postgres, MongoDB) across the stack.
You'll mentor senior engineers, raise the technical bar through design reviews and architectural guidance, champion engineering excellence in reliability and observability, and leverage AI tools to accelerate your work and improve the systems you build.
Success requires a proven track record leading platform migrations while maintaining backward compatibility and availability, strong command of streaming infrastructure (Kafka), deep distributed systems and data pipeline experience at enterprise scale, fluency in relational and document data modeling (Postgres and MongoDB), experience collaborating with leadership on multi-quarter roadmaps, solid AWS fundamentals, comfort with TypeScript and Node, and the ability to articulate complex tradeoffs to technical and non-technical audiences.
