Staff Data Engineer - Emerald

This is a Staff Data Engineer opportunity (Emerald team) with a Series C New York healthcare data company whose platform gives life sciences organisations data and AI-driven insight on doctors and trials. Emerald is the entity resolution platform that links external datasets (publications, clinical trials, conferences, web data) to canonical physician and organisation profiles. You’ll design and scale distributed Spark and PySpark pipelines for entity resolution, own automatching, identity mapping, deduplication and enrichment, build processing frameworks for publication and clinical trial data, improve throughput, observability and cloud cost, work with AI/ML teams on matching models, and mentor engineers. The stack includes Spark, AWS (EMR, S3), Kafka, Argo, Hudi, Docker, Kubernetes, Terraform and PostgreSQL or Redshift. The ideal candidate will have 8+ years building large-scale distributed data systems, deep Spark and AWS big data experience, technical leadership and mentoring experience. Entity resolution and healthcare or life sciences data experience are strongly preferred. This is a hybrid position in New York City.

Requirements