About the Role
We are seeking a skilled professional to lead our data engineering efforts, driving innovation and excellence in data solutions across both on-prem and cloud platforms.
This is an exciting opportunity for a talented engineer to take charge of architecting, building, and optimizing robust data pipelines and platforms, while leading a team of engineers working at the intersection of Big Data and Cloud innovation.
Key Responsibilities
* Design, build, and maintain scalable data pipelines and architectures for both on-prem (Hadoop ecosystem) and Databricks Cloud environments.
* Lead and mentor a team of data engineers, providing technical guidance, conducting code reviews, and fostering team growth.
* Ensure data quality, reliability, and governance across all stages of data processing.
* Work closely with data scientists, analysts, and stakeholders to deliver impactful data solutions.
* Drive best practices in CI/CD using Jenkins and Git, and manage deployment pipelines for seamless production releases.
* Optimize workflows using Airflow for orchestration and ensure performance across HDFS, Hive, Impala, and Spark.
* Utilize Databricks features including Delta Tables, SQL Warehouses, DLT pipelines, and Unity Catalog for secure, performant data management.
* Champion team collaboration and continuous improvement in both engineering practices and team development.