SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
LanceDB is an AI-native Multimodal Lakehouse that enables researchers to curate and manage petabytes of video, audio, and derived signals with minimal code, accelerating AI research and model training. The company powers frontier AI development at organizations like Runway, Midjourney, and Netflix.
You will be a Senior Software Engineer focused on expanding Lance and LanceDB's integration within the broader data infrastructure ecosystem. Working at the intersection of high-performance computing, big data systems, and open-source infrastructure, you'll drive scale, performance, and usability improvements across the platform.
Key responsibilities include:
- Designing and maintaining efficient distributed Lance dataset operations
- Building high-performance indices to enable predicate pushdown and accelerate queries in Spark, Ray, or Trino
- Contributing to table formats, data encodings, and the Lance format specification in Rust
- Leading open-source community efforts to integrate Lance with Spark, Hive Metastore, Presto, Trino, Ray, and other data infrastructure systems
- Operating and improving internal data processing infrastructure
- Promoting the Lance format in open-source communities and at Big Data conferences
This is an individual contributor role with significant technical scope and community leadership responsibilities. You will work independently and collaboratively with a high-caliber engineering team on problems spanning distributed systems, query optimization, and ecosystem integration.
REQUIREMENTS:
- 10+ years of experience building high-performance databases, big data systems, or large-scale data services
- Deep understanding of internals of open-source Big Data or AI training systems (e.g., Hadoop, Spark, Flink, Ray, Iceberg, Delta Lake, Hudi, ClickHouse, Trino, Presto, PyTorch, or JAX)
- Strong experience with high-performance computing in C++, Java, and/or Scala
- Experience with Rust or willingness to learn it
- Proven ability to move fast, work independently, and collaborate with a high-caliber team
NICE TO HAVE:
- Contributor, committer, or PMC member in Apache or other large open-source projects
- Experience with Apache Arrow, DataFusion, Parquet, Iceberg, or Delta Lake
- Track record of driving large features or integrations in distributed systems
- Strong community presence and passion for open-source collaboration
