Job Overview
As a skilled Data Engineer, you will be responsible for designing and implementing scalable ETL/ELT pipelines using Azure Data Factory (ADF). Your expertise in mapping data flows and parameterized pipelines will enable seamless data integration across systems. You will also collaborate with cross-functional teams to understand integration requirements and deliver robust solutions.
* Design and implement scalable ETL/ELT pipelines using Azure Data Factory (ADF).
* Integrate with Microsoft Dataverse and Dynamics 365 using ADF's native connectors, OData endpoints, and REST APIs.
* Collaborate with cross-functional teams to understand integration requirements and deliver robust solutions.
* Develop and optimize dataflows for transformation logic, including column reduction, lookups, and upsert strategies.
* Implement delta load strategies and batch endpoints to manage large-scale data synchronization efficiently.
* Ensure data consistency and integrity across systems by leveraging unique identifiers and business keys.
Key Responsibilities:
* Collaborate with business analysts and architects to translate business requirements into technical solutions.
* Monitor, troubleshoot, and optimize pipeline performance using ADF's built-in diagnostics and logging tools.
* Contribute to data governance, security, and compliance by implementing best practices in access control and data handling.
* Collaborate on the development of new standards and practices to improve the quality, capability, and velocity of the team.
* Mentor and coach junior team members new to Azure to ensure that the team builds up a strong capability.
Requirements:
* ADFAzure Data Engineer Experts required for Dublin role
* Azure Developer certification is highly desirable.
* Excellent problem-solving skills and the ability to work collaboratively in a team environment.
* Dynamics 365 experience is a must.