Lead Data Engineer
Data Engineering Position Overview
We are seeking a talented and experienced Lead Data Engineer specializing in healthcare data pipelines to join our team. This role will primarily focus on utilizing Databricks with Spark, Python, Scala, and SQL for building and optimizing data pipelines, as well as performing data engineering for machine learning and AI output.
Job Responsibilities
- Deliver a world-class data platform from the ground up
- Designing and developing data engineering solutions
- Support analytics, data science and data platform teams and understand their unique needs and challenges
Requirements
- 3+ years Data Engineering experience
- 2+ years hands-on experience working with large, structured/unstructured datasets using partitioned cloud storage architecture using query engines such as Spark, Delta Lake
- Experience with conceptual, logical and/or physical database designs is required
- Good knowledge in Linux and shell scripting is highly desired
Benefits of this Role
- Work closely with other engineering teams to troubleshoot issues with API's and data exchanges between multiple systems
- Develop datasets and models to meet the requirements of analysts, data scientists and key stakeholders; optimizing query compute and data storage patterns for cost and performance
What You Need To Succeed
- Strong communication skills to relay complex data integration requirements to team members
Benefits We Offer
- Flexible, hybrid work environment
- Competitive pay
- Paid time off and holidays
Additional Requirements
- Familiarity with machine learning concepts and tools is a plus
- Experience working in an Agile development environment is preferred
This job requires a strong foundation in data engineering and a willingness to learn about healthcare data.