Senior Data Engineer - Databricks
6 month Daily rate contract
Dublin - Hybrid
As a Data Engineer at you will be responsible for designing and implementing data solutions using Microsoft Azure and Databricks, ensuring scalable, efficient, and secure data pipelines while meeting the needs of both technical and business stakeholders. You will work closely with cross-functional teams to understand data requirements and design & develop cloud-based solutions that support real-time and batch data processing, advanced analytics, and machine learning workflows.
The role
1. Design, develop and implement robust, scalable, and efficient data pipelines in our Azure/Databricks cloud platform, for real-time and batch workloads
2. Implement ELT processes to integrate data from various sources into the CDP
3. Implement data quality checks, monitoring systems, and alerts for pipeline health
4. Manage and optimize data workloads in the data platform, ensuring performance, cost-efficiency, resiliency and security
5. Collaborate with data scientists, analysts, and other engineers to understand data needs and deliver optimal solutions from ingestion through to visualisation
6. Ensure data solutions comply with data governance, security, and privacy requirements
7. Understand cloud infrastructure provisioning using Infrastructure as Code (IaC) tools e.g. Terraform
8. Ensure data integrity, security, governance, and compliance in all data solutions, adhering to industry best practices and regulations
9. Mentor and guide junior data engineers leveraging industry wide best practices and technical excellence
10. Troubleshoot and resolve data-related issues, ensuring data accuracy and performant pipelines
11. Stay current with the latest technologies and trends in cloud computing & data engineering particularly within the Microsoft Azure and Databricks ecosystems
Skills Required
oStrong experience in Microsoft Azure, including Azure Data Factory, Azure SQL Database, Azure Storage, Azure Key Vault, Function & Logic Apps, Cost Analysis and other related Azure cloud services
oExperience with Databricks, including data engineering, data processing, and analytics development
oStrong understanding of performance optimisation best practices & data governance frameworks (e.g Databricks Unity Catalog)
oStrong knowledge of ETL/ELT processes, data modelling, data warehousing concepts and medallion architecture
oExperience with real-time data processing frameworks (e.g., Apache Kafka)
oProficiency in Python programming language
oExpertise in PySpark, SQL and experience working with large-scale datasets
oKnowledge of CI/CD DevOps practices and tools (Git, Azure DevOps)
oExperience working with Agile methods of delivery
oCloud infrastructure provisioning using Terraform Infrastructure as Code (IaC), desirable
oKnowledge of machine learning integration with ML Ops desirable
For more information call Michael on 01- or
#LI-MF7