Lead Data Engineer – 12‑Month Contract (Hybrid)
*For this position, you need to be based in Ireland and hold an EU/UK passport or Stamp 4!
The Role
Lead the end‑to‑end design, development, and optimisation of scalable Spark/PySpark data pipelines on Databricks, shaping robust, secure, and high‑quality data architectures. Collaborate with cross‑functional teams – data science, compliance, and product – to deliver actionable intelligence from complex data ecosystems.
Your Responsibilities
* Design, build, and optimise Spark/PySpark data pipelines on Databricks (batch and streaming).
* Define and enforce data lakehouse architecture and governance standards.
* Architect secure, compliant AWS infrastructure (S3, Glue, IAM, Lake Formation, etc.).
* Drive CI/CD, testing, and automation for data workflows.
* Mentor engineers, lead design reviews, and drive continuous improvement.
About You
* 10+ years’ experience as a Data Engineer, with strong Databricks, Spark, Python, and SQL expertise.
* Deep knowledge of AWS data services: S3, Glue, Lake Formation, and orchestration tools Airflow, Databricks Workflows.
* Strong grasp of data governance, lineage, and compliance (Unity Catalog, OpenLineage).
* Experience in large‑scale distributed data systems within financial services or similar industries.
* Excellent communication and leadership skills, with a pragmatic approach to delivery and mentoring.
Seniority Level
Mid‑Senior level
Employment Type
Full‑time
Job Function
Information Technology
Industries
Banking
Referrals increase your chances of interviewing at E‑Frontiers by 2x.
#J-18808-Ljbffr