Job Title: Data Engineer Lead
We are seeking a skilled Data Engineer Lead to join our team and play a key role in the design, development, and implementation of our data infrastructure. The ideal candidate will have expertise in designing and implementing scalable, efficient, and robust data pipelines in cloud platforms.
Key Responsibilities:
* Design, develop, and implement data pipelines that meet business requirements and ensure data accuracy, security, and compliance.
* Lead a team of data engineers to achieve project objectives and maintain high-quality standards.
* Collaborate with cross-functional teams, including data scientists, analysts, and stakeholders, to understand data needs and deliver optimal solutions.
* Ensure data integrity, security, governance, and compliance in all data solutions, adhering to industry best practices and regulations.
* Stay up-to-date with the latest technologies and trends in cloud computing and data engineering.
Required Skills and Qualifications:
* Strong experience in Microsoft Azure, including Azure Data Factory, Azure SQL Database, Azure Storage, Azure Key Vault, Function & Logic Apps, Cost Analysis, and other related Azure cloud services.
* Experience with Databricks, including data engineering, data processing, and analytics development.
* Strong understanding of performance optimization best practices and data governance frameworks (e.g., Databricks Unity Catalog).
* Strong knowledge of ETL/ELT processes, data modeling, data warehousing concepts, and medallion architecture.
* Experience with real-time data processing frameworks (e.g., Apache Kafka).
* Proficiency in Python programming language.
* Expertise in PySpark, SQL, and experience working with large-scale datasets.
* Knowledge of CI/CD DevOps practices and tools (Git, Azure DevOps).
* Experience working with Agile methods of delivery.
* Cloud infrastructure provisioning using Terraform Infrastructure as Code (IaC), desirable.
* Knowledge of machine learning integration with ML Ops, desirable.
Benefits:
We offer a competitive salary and benefits package, including opportunities for professional growth and development.
What We Offer:
We provide a dynamic work environment, collaborative team culture, and cutting-edge technology stack.