Job Title: Data Engineering Expert
We are seeking a highly skilled Data Engineer to join our team.
About the Job:
* This role will involve designing, developing and maintaining scalable data pipelines using Databricks and Apache Spark.
* You will collaborate with data analysts and scientists to understand data needs and provide appropriate solutions.
* The ideal candidate will have experience in ETL pipeline development and optimisation, as well as strong knowledge of SQL, Python and Scala programming languages.
* Cloud experience is also essential, with Azure preferred but AWS also suitable.
Your Key Responsibilities:
1. Data Pipeline Development: Design and develop scalable data pipelines using Databricks and Apache Spark to ensure efficient data processing and analysis.
2. Data Quality Assurance: Ensure data quality by implementing validation and testing procedures to guarantee accuracy and reliability.
3. ETL Optimisation: Optimize existing ETL processes for enhanced performance and efficiency to meet business requirements.
4. Data Modelling and Design: Assist in data modelling and design of data warehouses to support business intelligence initiatives and drive strategic decision-making.
5. Performance Monitoring and Troubleshooting: Monitor and troubleshoot data pipeline performance issues to identify areas for improvement.
6. Code Reviews and Best Practices: Participate in code reviews and contribute to best practices and standards in data engineering to maintain high-quality code and ensure consistency.
7. Documentation and Knowledge Sharing: Document data processes, architecture, and workflows for knowledge sharing across the team to facilitate collaboration and innovation.
Your Requirements:
* Databricks & Apache Spark
* ETL pipeline development and optimisation
* SQL, Python/Scala
* Cloud experience (Azure preferred, AWS also suitable)
* Data warehousing & modelling
About Our Organisation:
We are a forward-thinking organisation with a passion for harnessing the power of data to drive business growth and success. We offer a dynamic work environment that fosters collaboration, innovation, and continuous learning.