Data Integration Specialist
We are seeking an experienced Data Engineer specializing in healthcare data pipelines to join our team.
Job Description
This role will primarily focus on utilizing Databricks with Spark, Python, Scala, and SQL for building and optimizing data pipelines, as well as performing data engineering for machine learning and AI output.
Key Responsibilities:
* Designing and developing robust data pipelines for healthcare data
* Utilizing Databricks with Spark, Python, Scala, and SQL
* Performing data engineering for machine learning and AI output
Required Skills and Qualifications
- 5+ years Data Engineering experience
- 3+ years hands-on experience working with large, structured/unstructured datasets using partitioned cloud storage architecture using query engines such as Spark, Delta Lake.
- 3+ years experience designing, developing, deploying and testing in Databricks.
- 3+ years of hands-on experience in Python/Pyspark/SparkSQL.
- 3+ years of experience with Scala programming language
- 2+ years experience on Big data pipelines/DAG tools like Airflow and dbt
- 2+ years of SQL experience, specifically to write complex, highly optimized queries across large volumes of data.
- Experience in the AWS computing environment and storage services such as s3/glacier is required.
- Experience with conceptual, logical and/or physical database designs is required.
- Good knowledge in Linux and shell scripting is highly desired.
- Experience with Data Visualization tools like Looker, Tableau is desired.
- Strong communication skills to relay complex data integration requirements to team members.
Benefits
We provide a total compensation package that includes: pension contribution, flexible hybrid work environment, and opportunities for professional growth and development.
Additional Information
We are committed to building a diverse team of professionals who are all responsible for stewarding a high-performance culture in which all team members belong and thrive. We are proud to be an Equal Employment Opportunity employer.