Software Engineer, Data Ingestion

Job Description

Debug and improve large-scale Spark-based ingestion pipelines, resolving corrupt data, stalled jobs, queues, and complex workflow failures. Increase pipeline resilience, throughput, reliability, cost efficiency, and operability while prioritizing datasets for downstream annotation, data science, and model training.

Requirements: Candidates should have strong production experience with Apache Spark and Python, plus experience building, operating, or debugging large-scale ingestion, ETL, or distributed data-processing pipelines. They should be able to diagnose production failures and optimize systems for reliability, performance, and cost; experience with Airflow, Flyte, Databricks, Delta Lake, Scala, or Java is advantageous.

Key Skills: Apache Spark, Python, Data Ingestion, ETL, Distributed Data Processing, Pipeline Debugging, Data Quality, Batch Processing, Pipeline Optimization, Monitoring, Failure Recovery, Workflow Orchestration, Airflow, Flyte, Databricks, Delta Lake

Benefits: Equity, Relocation Support, Visa Sponsorship, Hybrid Working, Learning And Development Budget, Health Insurance, Dental Insurance, Enhanced Maternity And Paternity Leave, Retirement Or Pension, Therapist Access, Wellbeing Partnerships, Team Socials