SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Salary: USD 192,000 - 240,000 / annual
Husky is Datadog's distributed, petabyte-scale columnar event store powering the Event Platform and dozens of Datadog's most popular products including Logs, RUM, APM, Cloud Network Monitoring, and Netflow. Built from the ground up at Datadog to store and query massive volumes of event data at low cost with data fully queryable within seconds of arrival.
As a Senior Software Engineer on the Husky Storage team, you will work hands-on with a distributed storage system at petabyte scale and help evolve the platform to support the next level of scale for the AI era. The team is migrating from a custom storage format toward open standards (Parquet, Arrow, DataFusion) to enable on-premises solutions, increasingly complex analytical and batch workloads, and power AI observability sustainably into the future.
Key responsibilities include:
- Drive Husky adoption by migrating the storage format to open-source Parquet and reimagining compaction on a format-agnostic, DataFusion-based substrate
- Drive concrete cost-reduction work including block-level inverted index compression, combining the database with external objects at the row level, and storage tiering/caching
- Scale the solution to enable customers to manage event versioning at scale using temporal-database techniques
- Play an influential role in operating and deciding how to best operate the service going forward
- Work across the write path with the Ingestion team and toward a cleaner Iceberg-based handoff to the Lakehouse team for batch and analytical workloads
You will partner closely with Husky Ingestion, Metadata, and Platform teams, and increasingly with Lakehouse on the Iceberg-compatible boundary. You will also work with the open-source community to build the next generation of reusable software solutions that can be operated at scale.
Datadog's culture is engineering-driven, bottom-up, and high-agency. The company emphasizes blameless culture, open-source advocacy, and flexibility in remote work while valuing in-person collaboration across offices.
Requirements:
- Enthusiastic and experienced with large-scale distributed systems and/or storage engines
- Comfortable with improving and operating large distributed systems that customers rely on for critical business functions
- Strong fundamentals in columnar/file storage formats (Parquet, ORC, Arrow) or comparable systems
- Bonus experience: Go, Java, or Rust; compaction or compression/encoding schemes (RLE, dictionary, delta, gorilla); object storage (S3/GCS/Azure Blob) at scale including caching and cost optimization; Kafka and stream processing; or Kubernetes
