The Information Lab Ireland
About The Information Lab:
The Information Lab is a leading data consultancy specialising in delivering innovative data-driven solutions to help businesses optimise performance, enhance decision-making, and drive growth. We partner with organisations across industries to harness the power of data through cutting-edge analytics, AI, and cloud solutions.
Role Overview:
As a Senior Data Engineer at The Information Lab, you will lead the technical delivery of small-to-medium engagements or own critical workstreams on large enterprise projects. Reporting to a Lead Data Engineer, you will act as the primary technical point of contact for clients on your projects, managing junior associates and ensuring code quality, while collaborating with the Lead Engineer for high-level architectural sign-off. You will lead project teams, coordinate tasks, and be accountable for the quality and timeliness of deliverables. This role is ideal for an individual with deep Databricks platform expertise, exceptional attention to detail, and a passion for leading technical teams through example.
Key Responsibilities
* Technical Project Delivery Ownership: Own and drive the hands-on implementation and execution of complex Databricks and Azure solutions, ensuring high standards of quality and architectural alignment, attention to detail, and timeliness.
* Technical Delivery: Apply deep technical knowledge in data engineering and solution design and to build end-to-end data solutions. This includes:
o Developing, managing and performance tuning data pipelines across a medallion architecture (Bronze, Silver, Gold) using SQL and PySpark.
o Oversee and enforce the creation of detailed source-to-target mapping documentation to ensure data lineage and clarity.
o Implement and manage Databricks Unity Catalog.
o Implementing robust metadata tracking and designing for schema evolution.
o Developing reports and reconciliation dashboards to validate data accuracy.
* Technical Quality Assurance & Testing: Have accountability for code quality, accuracy and completeness of all delivered notebooks and pipelines. This includes implementing and managing data quality frameworks (e.g. Great Expectations).
* Governance and Best Practices:
o Manage the implementation of Git and DevOps (CI/CD) standards for version control and automated deployments.
o Design and implement workspace governance, security protocols, and comprehensive documentation for client implementations.
o Create and maintain clear and concise detailed technical runbooks and low-level design documentation for all aspects of the project delivery.
* Client Communication:
o Work directly with clients to identify and align on business value, technical matters, clarifying requirements, presenting work, and incorporating feedback.
o Manage day-to-day client technical stakeholders, managing expectations regarding timelines and technical feasibility.
o Identify potential scope creep on your projects and escalate commercial implications to the Consulting Director where necessary.
* Delivery Leadership & Mentorship:
o Lead daily stand-ups and sprint planning for your project team, delegating tasks effectively to Data Engineers.
o Perform the first line of technical review and QA for your team's code, ensuring it meets The Information Lab's standards before final sign-off.
o Mentor junior team members on-the-job, helping them upskill in PySpark, SQL, and consulting soft skills.
* Issue Resolution: Proactively identify risks and challenges in the delivery process, escalating to the Director as appropriate.
* Learning & Development: Stay current with emerging technologies and maintain certifications relevant to Databricks, Microsoft Fabric, Python, and data engineering, with a progression path towards becoming a Databricks and Azure Solution Architect.
* Knowledge Sharing: Contribute to internal documentation, delivery standards, and community initiatives within the consultancy.
Qualifications & Experience
* Recommended 3+ years hands-on experience in Data Engineering/Consulting with at least +1 year focused on large-scale Databricks deployments on Azure, including Unity Catalog and Delta Live Tables.
* Demonstrable proficiency in PySpark and SQL for complex data transformation and analysis, and performance tuning ideally within Databricks Notebooks or an Azure environment.
* Proven experience acting as a senior data engineer on projects, including breaking down requirements into tasks for other developers, and managing client communication.
* Strong familiarity with data modelling techniques, specifically Medallion architecture and Kimball standards.
* Demonstrable experience owning the technical implementation of Git integration, Databricks Repos, and CI/CD pipelines within an Azure DevOps environment.
* Demonstrable experience defining and implementing data quality frameworks and testing methodologies.
* Exceptional attention to detail with a track record of high-quality, production-ready code and robust technical documentation.
* Excellent organisational skills with the ability to manage multiple workstreams.
* Strong communication skills and confidence working directly with clients on complex technical deliverables.
* Relevant certifications in Azure or Databricks (e.g. Databricks Data Engineer Professional, DP-203, DP-700) are highly preferred.
* A bachelor's or master's degree in a relevant field is preferred but not required.
Key Performance Indicators (KPIs)
* Delivery Accuracy: Quality and correctness of deliverables.
* Timeliness: Adherence to delivery schedules and project milestones.
* Client Feedback: Satisfaction from clients regarding technical execution and solution reliability.
* Team Effectiveness: Ability to manage junior team members and collaborate efficiently.
* Code Quality & Governance: Adherence to coding standards, version control practices, and documentation completeness.
* Technical Certification & Learning: Completion and maintenance of relevant technical certifications.
* Commercial Engagement: Support Consulting Partner, Directors, and Senior Managers in successful pre-sale client engagements, and identification of additional opportunities with existing clients.
* Attention to Detail: Demonstrated rigour in analysis, documentation, and solution development.
What we Offer
* Competitive Salary
* 25 days annual leave per year
* Pension and Healthcare subject to successful probationary period
* Learning and development days
* Flexible working arrangements
* Team Building Days
* Great Team Culture
For more information, please contact
Type of Person
Package and Rewards
Role Purpose
Role Requirements
Our Values
Role Responsibilities
Competencies
Experience