Staff Data Engineer

This is a Staff Data Engineer (Data Lake) opportunity with a Series C New York healthcare data company whose platform gives life sciences organisations data and AI-driven insight on doctors and trials. You’ll design and scale distributed ETL and ELT pipelines and ingestion frameworks for structured and unstructured healthcare data, evolve the data lake architecture for scale, reliability and cost, improve data quality and normalisation across thousands of global sources, build batch and near-real-time processing on cloud-native platforms, optimise Spark workloads, and mentor engineers and lead architecture reviews. The stack includes Python (PySpark), Java, Scala, SQL, Spark, AWS (EMR, Glue, S3, Athena, Redshift), Argo or Airflow, Docker and Kubernetes. The ideal candidate will have 8+ years in data or software engineering building distributed data platforms, advanced SQL, deep Spark experience and technical leadership. Large-scale data cleaning and healthcare, life sciences or entity-resolution datasets are preferred. This is a hybrid position in New York City.

Requirements