This is a hybrid role, requiring the successful candidate to attend our Tralee office.
Role Description SMBC is seeking an Azure Cloud Data Engineer who has a strong passion for Production Support and is interested in building a career at a fast growing and reputable Bank.
The Azure Cloud Data Engineer will have strong Azure Data Factory and Databricks knowledge.
They will be given opportunities to participate in providing support for Azure based data integration and analytics pipelines built using Azure Data Factory and Azure Databricks ensuring the uptime and performance of critical pipelines and workflows.
The successful candidate will have knowledge of complex interface process development and is required to troubleshoot issues.
This role will report to the Executive Director of Production Support.
Role Objectives: Expertise
• Monitor, troubleshoot and support ADF pipelines and Databricks notebooks/jobs in Production.
• Extensive experience on Cloud solutions, specifically in Azure
• Experience with Azure cloud services, Azure Data Factory, Gen 2, Azure Databases, Functions, Databricks, or similar technology.
• Good understanding of ETL/ELT
• Analyze pipeline failures, spark job issues, data mismatches, cluster timeouts, resource unavailability and latency bottlenecks.
• Root cause analysis for incidents and outages.
• Understand the ADF Components and architecture.
• Knowledge of data integration techniques and best practices.
• Experience with connection to various data sources and destinations.
• Ability to orchestrate complex data workflows and transformation.
• Monitoring and troubleshooting data pipeline executions.
• Familiarity with ADF data flow activities for data transformations.
• Version control and deployment management using Azure DevOps or similar tools
• Awareness of ADF Integration with Azure services like Azure Data Lake Storage, Azure Databricks, etc.
• Skills in implementing streaming and batch data ingestion using Delta Lake.
• Skills in implementing data pipelines and workflows in Databricks.
• Familiarity with Databricks notebooks for interactive data exploration and development.
• Integrating Databricks with Azure Services like ADLS Gen2, Azure SQLdb.
• Monitoring and optimizing Databricks jobs for cost efficiency.
• Proficiency in Git for managing code repositories, including branching, merging and pull requests.
• Support CI/CD pipelines for deployment using Azure DevOps.
• Participate in on-call rotation and ensure business continuity via proper DR strategies.
• Experience with RDBS systems like Azure SQL DB, Oracle, and NoSQL Databases like MongoDB
• Understanding of indexing, partitioning, and other optimization techniques.
• Experience with stored procedure, functions, and triggers.
• Ensure High Availability (HA) of ADF pipelines and auto-scaling/failover readiness of Databricks clusters.
• Manage alerts, incidents and escalations using ServiceNow, Azure Monitor, Log Analytics etc.,
• Proficiency in Git for managing code repositories, including branching, merging and pull requests.
• Experience with Confluence, ServiceNow