Data Engineering Lead
This is a key role for developing, maintaining, and optimizing data pipelines and systems that support the acquisition, storage, transformation, and analysis of large volumes of data.
As a Data Engineer Lead, you will design, develop, and maintain data pipelines in Azure for ingesting, transforming, and loading data from various sources into centralized Azure data lakes, Databricks Delta Lake and Snowflake. You will ensure data quality and integrity throughout the process.
The ideal candidate has strong technical expertise in data engineering principles, database management, and programming skills. They also have hands-on experience with Azure services such as Azure Synapse Analytics, Azure Data Factory, Azure Databricks, Azure SQL Database, etc., and are familiar with CI/CD tools and DevOps practices.
You will work closely with cross-functional teams to understand data requirements and translate them into technical solutions. This includes designing and implementing efficient data models and database schemas that support the storage and retrieval of structured and unstructured data. You will also optimize data storage and access for performance and scalability.
In addition, you will implement knowledge of modern data processing principles to streamline data import/transformation processes. You will leverage modern data pipeline tools to reduce human attention during ETL process and ensure the efficiency and reliability of data ingestion and processing.
Your responsibilities include collaborating with stakeholders to define and enforce data governance standards and policies, identifying performance bottlenecks in data pipelines and database systems and optimizing queries, data structures, and infrastructure configurations to improve overall system performance and scalability.
Furthermore, you will implement appropriate security measures to protect sensitive data and ensure compliance with data privacy regulations. You will also monitor and address data security vulnerabilities and risks. Your strong background in integrating continuous delivery (CD) with Kubernetes using tools such as Argo, GitLab, Spinnaker and strong Git experience, development methodologies, trunk-based develop vs. git flow, Helm etc. will be valuable assets in this role.
Finally, you will stay updated with the latest trends, tools, and technologies in the field of data engineering. You will proactively identify opportunities to improve data engineering practices and contribute to the evolution of data infrastructure.