SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Pattern Group is seeking a Senior Data Engineer to own and extend Destiny, an automated ads optimizer that discovers high-value keywords, assembles feature stores, runs machine-learning models, and pushes campaign edits to marketplace APIs every 15 minutes. This is a high-stakes system where data correctness directly impacts customer advertising spend—a silently wrong join or unguarded backfill becomes a customer-facing incident, not a dashboard discrepancy.
You will develop, deploy, and support automated, scalable batch data pipelines from diverse sources into the lakehouse. You'll own and extend Airflow orchestration for multi-DAG, cross-triggered daily pipelines and 15-minute action pipelines, including branching, parallel task groups, cross-DAG triggers, backfill and full-refresh paths, and safe reruns. You'll write and tune large analytical SQL—multi-hundred-column joins, window functions, incremental merges—and perform warehouse-sizing and query-profile work to keep daily runs within time and budget windows.
You will extend the feature store by adding new features and labels, wiring them through the join layer, and preserving leakage and data-completeness conventions that make models trainable. You'll orchestrate model training and batch inference on SageMaker from Airflow: build training and scoring datasets, manage S3 and Parquet round-trips, containerized training images, instance sizing, and load predictions and metrics back into the warehouse.
You'll develop and implement data auditing strategies and processes to ensure data quality, including blocking data-quality checks in front of outward writes, and set thresholds that catch bad data without needlessly halting live bidding. You'll identify and resolve problems in large-scale data processing workflows, maintain pipeline processes, troubleshoot failures, and participate in on-call triage when a run breaks before market open.
You'll guard the safety properties of an outward-writing system: idempotency, new-data detection, action validation and invalidation, and audit trails for every change pushed to marketplace. You'll collaborate with data scientists, advertising strategists, and platform teams to specify data requirements and provide data access. You'll translate business and analytics requirements—ROAS targets, budget pacing, playbook rules, branded versus non-branded strategy—into comprehensive data models and pipelines.
You'll foster data expertise and own data quality for assigned areas; work with data infrastructure to triage issues and drive to resolution. You'll mentor and provide technical direction to other data engineers and review their SQL and DAG changes.
In your first 30 days, you'll read the pipeline end to end, shadow a daily run, ship small SQL and DAG fixes, take your first on-call triage, and explain how a bid becomes an edit on marketplace. Days 31–60, you'll own a stage, add features to the feature store, tune slow tasks, and add or re-threshold data-quality checks. Days 61–90, you'll lead a cross-pipeline change—a new signal, playbook rule, or reliability improvement—with tests, monitoring, and rollback stories.
REQUIREMENTS:
- Bachelor's degree in Data Science, Data Analytics, Information Management, Computer Science, Information Technology, or related field, or equivalent professional experience
- 4+ years overall professional experience
- 4+ years hands-on experience with SQL and Python, including advanced SQL (window and analytic functions, complex joins, incremental merges, query tuning)
- 3+ years building production data pipelines on modern data architectures, with real ownership of scheduling, dependencies, retries, backfills, at scale across many source systems
- 2+ years working with cloud data warehouses (Snowflake, Redshift, or BigQuery)
- Production experience with a workflow orchestrator—Airflow strongly preferred—including debugging failed runs in live systems
- Experience orchestrating ML training and batch inference from a scheduler on SageMaker or equivalent platform
- Working knowledge of applied machine learning fundamentals: dataset construction, leakage, evaluation metrics (MAE, RMSE, MAPE, WMAPE), and model lifecycle operations
- Comfort with AWS (minimum S3 and IAM) and columnar file formats
- Demonstrated ownership of data quality: testing, monitoring, alerting, and root-cause analysis on pipelines others depend on
- Excellent software engineering and scripting practice (version control, code review, modular changes)
- Strong communication skills in presentation and comprehension, with aptitude for cross-collaboration across data management, data science, and analytics
- Ability to lead and mentor a team of data engineers
PREFERRED:
- Experience with digital advertising, bidding, or auction systems (Amazon Ads, Google Ads, DSP)
- Advanced Snowflake (streams, tasks, stored procedures, UDFs, clustering, cost/performance tuning)
- Experience with time-series data, forecasting, hourly or day-parted grains
- Background in big data, non-relational databases, machine learning, or data mining
- Experience with data-quality frameworks (Soda, Great Expectations, dbt tests)
- Experience with open-source and distributed data platforms (Spark, Hive, Trino/Presto, Cassandra, DynamoDB, Elasticsearch)
- Broader cloud experience (SNS, SQS, SES, Lambda, Glue, ECR, containerized workloads)
- Expertise in data governance
- Experience working productively with AI coding agents on large existing codebases
