SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
EdCast is seeking a Data Engineer to design, develop, and maintain scalable cloud-based data pipelines and platforms. This role focuses on building reliable ETL/ELT pipelines, real-time data ingestion, data transformation, monitoring, and performance/cost optimization across AWS, GCP, Kafka, and Databricks.
Key responsibilities include:
- Design and develop ETL/ELT data pipelines using AWS Glue, Databricks, and Spark
- Build and maintain batch and real-time data pipelines using Apache Kafka or Confluent Kafka
- Develop CDC-based data ingestion and event-driven data processing
- Work with AWS services including Glue, S3, Lambda, IAM, KMS, and Secrets Manager
- Develop and optimize data solutions on GCP, including BigQuery, Cloud Storage, and Cloud Functions
- Implement data processing using Databricks, PySpark, Delta Lake, and Bronze/Silver/Gold architecture
- Manage and optimize Elasticsearch/OpenSearch data ingestion, indexing, queries, and cluster performance
- Implement data quality, validation, monitoring, logging, and alerting across pipelines
- Troubleshoot production issues involving Kafka, schema compatibility, CDC, Spark, Glue, Databricks, and cloud infrastructure
- Optimize pipeline performance and cloud infrastructure costs
- Implement secure data solutions using IAM, encryption, network controls, and least-privilege access
- Collaborate with application, DevOps, QA, analytics, and product teams
Requirements:
- 5+ years of experience in Data Engineering or Big Data Engineering
- Strong hands-on experience with AWS, Databricks, and Kafka
- Experience working in multi-cloud environments (AWS/GCP)
- Strong understanding of distributed systems, streaming, CDC, schema evolution, and data governance
- Experience troubleshooting high-volume production data pipelines
- Experience with cloud performance and cost optimization
