SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Vanta is a Series D security and compliance platform that helps businesses earn and prove trust through continuous security monitoring and verification. The Foundations team owns Vanta's data platform, ingesting tens of thousands of security and compliance resources per second from customer environments, storing and cataloging the data, and transforming it into auditable evidence for SOC 2 and FedRAMP compliance.
As a Staff Software Engineer on the Foundations team, you will lead a major re-platforming initiative: migrating Vanta's data layer from a Mongo-centric architecture to a schema-aware, Postgres-backed architecture running on Kafka and S3 pipelines. This is a complex, high-stakes migration that must preserve every public API contract, maintain zero data loss for customers, and run both old and new systems in parallel during the transition.
Key responsibilities include:
- Lead the resource data model migration from MongoDB to Postgres while maintaining backward compatibility and customer integrations
- Drive cross-team solutions for platform dependencies you don't directly own
- Design for correctness under eventual consistency, including idempotent session handling, conditional writes, and reconciliation patterns
- Design and evolve the data ingestion and pipeline architecture for reliable, high-throughput processing of terabyte-scale streams
- Evolve the Query API from internal tool to production platform with schema versioning, joins, exports, per-tenant isolation, and predictable latency
- Build auditable data infrastructure where correctness can be externally verified
- Diagnose production failures at scale (hot partitions, unbounded fan-out, online table rewrites)
- Set architectural direction for streaming (Kafka), caching (Redis), and storage (Postgres, MongoDB) across Foundations
- Mentor senior engineers, raise technical bar through design reviews and architectural guidance
- Champion reliability, observability, and operational excellence across data infrastructure
- Explore and leverage AI tools to accelerate your work and improve systems
Success requires proven experience leading platform migrations while maintaining availability, deep expertise in streaming infrastructure (Kafka), distributed systems and data pipeline engineering at enterprise scale, fluency in relational and document data modeling (Postgres and MongoDB), ability to collaborate with leadership on multi-quarter roadmaps, AWS fundamentals, TypeScript/Node proficiency, and strong communication skills for complex technical tradeoffs.
