Senior / Lead Data Engineer
opportunity with leading data consultancy based in Dublin 2 (hybrid working 2-3 days). Our client delivers cutting-edge analytics and AI solutions to enterprise-level customers.
Responsibilities
* Technical project delivery and hands-on execution of complex Databricks and Azure solutions
* Develop and performance tune data pipelines using SQL and PySpark.
* Implement and manage Databricks Unity Catalog.
* Develop reports and reconciliation dashboards to validate data.
* Manage data quality frameworks
* Manage the implementation of Git and DevOps (CI/CD) standards
* Work directly with clients technical stakeholders to manage expectations regarding timelines and technical feasibility.
* Lead daily stand-ups and sprint planning for your project team
Experience
* 5+ years hands-on experience in Data Engineering
* 2+ years exp working on large-scale Databricks deployments on Azure
* Proficiency in PySpark and SQL for complex data transformation and analysis
* Proven experience as a lead on projects, sprint planning and managing client communication.
* Demonstrable experience leading the technical implementation of Git integration, Databricks, and CI/CD pipelines within an Azure DevOps environment.
* Strong communication skills and confidence working directly with clients
* Relevant certifications in Azure or Databricks are desirable