SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Wynd Network builds infrastructure that delivers massive amounts of web data to companies training frontier AI models. The company operates Grass, a bandwidth-sharing network powering a distributed crawler with unique access to high-quality public web data at global scale. They've also built pipelines for ingesting, segmenting, and annotating billions of videos, transcripts, and audio files for dataset creation.
As a Data Engineer, you'll maintain, optimize, and troubleshoot database queries and data systems supporting efficient data access and processing. You'll create and improve data pipelines used to collect, process, transform, validate, and deliver large-scale datasets. Your responsibilities include supporting web scraping and data collection initiatives by developing and maintaining scripts and tools for gathering publicly available data. You'll monitor and troubleshoot pipeline issues, identify data quality concerns, and implement timely fixes to maintain accuracy and operational continuity.
You'll document engineering work including database queries, pipeline processes, scraping workflows, and technical decisions. You'll also participate in R&D projects to improve the company's data products and workflows.
Required skills: advanced Python with async programming, multiprocessing, and production-grade code for long-running jobs; hands-on web scraping at scale with proxies, rate limiting, and anti-bot evasion; distributed data pipelines using task queues (Celery, Kafka, RabbitMQ); data warehousing with columnar/analytical databases (Databend, ClickHouse, BigQuery); Docker & Kubernetes for containerization and deployments; Linux and bare-metal operations; CI/CD for data workflows (GitHub Actions, ArgoCD); and scalable API writing.
The team is lean, technical, and moves fast with no red tape. They prioritize low ego and high output, seeking people who make everyone around them better. This is a fully remote position with competitive salary, benefits, and equity.
