Staff Data Engineer - Emerald

About the role

This is a Staff Data Engineer opportunity (Emerald team) with a Series C New York healthcare data company whose platform gives life sciences organisations data and AI-driven insight on doctors and trials. Emerald is the entity resolution platform that links external datasets (publications, clinical trials, conferences, web data) to canonical physician and organisation profiles. You’ll design and scale distributed Spark and PySpark pipelines for entity resolution, own automatching, identity mapping, deduplication and enrichment, build processing frameworks for publication and clinical trial data, improve throughput, observability and cloud cost, work with AI/ML teams on matching models, and mentor engineers. The stack includes Spark, AWS (EMR, S3), Kafka, Argo, Hudi, Docker, Kubernetes, Terraform and PostgreSQL or Redshift. The ideal candidate will have 8+ years building large-scale distributed data systems, deep Spark and AWS big data experience, technical leadership and mentoring experience. Entity resolution and healthcare or life sciences data experience are strongly preferred. This is a hybrid position in New York City. We're looking for someone curious, pragmatic, and eager to work across the stack with a small senior team. You'll own end-to-end delivery of major features, influence architecture, and mentor engineers.

What you'll do

• Ship customer-facing features on a weekly cadence.
• Partner with design and product on scoping and trade-offs.
• Drive technical quality through review, testing, and docs.
• Mentor one or two engineers on your team.

Requirements