Location: Limerick City — Hybrid (minimum 3 days/week onsite during probation)
Employment Type: Full-time
Experience Level: Senior
Position Responsibilities
Design, develop, and optimize large-scale distributed data pipelines using Apache Spark as the core processing engine.
Implement robust batch and streaming solutions using modern Lakehouse patterns (e.g., Delta Lake, streaming ingestion, structured transformations).
Develop and maintain data workflows in our Databricks environment (or ramp up quickly if new to Databricks).
Apply deep Spark expertise for performance tuning, partitioning strategies, job optimisation, and troubleshooting distributed workloads.
Ingest data from structured, semi-structured, and unstructured sources into the Lakehouse environment.
Technical leadership
Serve as a senior specialist in Spark-based data engineering, adhering to best practices, patterns, and quality standards
Lead and participate in peer code reviews, promoting high engineering standards across coding, testing, version control, and documentation.
Help define patterns and guidelines for producers sending data to the Lakehouse to ensure quality, consistency, and reliability.
Collaboration & Delivery
Work closely with the Data Solutions Architect and Product Owner to design solutions that meet technical and business requirements.
Collaborate with cross-functional engineers to ensure data workflows integrate smoothly with applications, services, and platform components.
Communicate technical concepts clearly to both technical and non-technical stakeholders.
Ensure solutions comply with modern development processes, deployment workflows, and change control.
Team Contribution
Share knowledge openly and mentor less experienced engineers, particularly in Databricks, Apache Spark, and distributed data engineering.
Learn from colleagues about the broader system architecture and domain, contributing to a strong, cohesive engineering culture.
Promote a culture of teamwork, continuous learning, and shared ownership within the engineering group.
Required Skills
8+ years of professional experience in software or data engineering, with a strong focus on distributed data processing and cloud-based data platforms.
Deep hands-on expertise with Apache Spark including Data Frame API, Spark SQL, shuffle optimisation, execution plan analysis, and debugging distributed workloads.
Experience with Databricks or the ability to ramp up quickly - candidates with strong Spark experience can easily transition into Databricks.
Strong proficiency in Python and SQL.
Solid understanding of big data architectures, relational database design, and cloud-native data patterns.
Experience designing and maintaining ETL/ELT pipelines.
Familiarity with version control systems such as Git/TFS/SVN.
Experience with CI/CD pipelines, including Octopus Deploy, Git Hub Actions, or Azure Dev Ops.
Excellent communication skills and proven experience working within Agile methodologies.
Fluency in written and spoken English
Preferred Skills
Production experience with Databricks, including notebooks, workflows/jobs, Delta Lake, and cluster/runtime configuration.
Experience with Azure data services (e.g., ADLS Gen2, Azure Data Factory).
Knowledge of Delta Live Tables and/or Delta Sharing.
Experience with Spark Structured Streaming and No SQL data stores.
Familiarity with Infrastructure-as-Code tools such as Terraform or ARM templates.
Understanding of data governance, data cataloguing, and data management best practices.
Exposure to machine learning or data science concepts.
Experience with Azure Event Hub or Event Grid for real-time or streaming ingestion.
Why Join us
Opportunity to play a major role in shaping a modern cloud-native Lakehouse platform.
Work in a collaborative team that values creativity, ownership, and continuous learning.
Strong focus on ongoing professional development and certification support.
Hybrid working model with flexibility.
Regular social, sporting, and community events.
Benefits
Private Health Insurance
Dental Insurance
Matched Pension Contribution
25 Days Annual Leave
This job description is not designed to contain a comprehensive listing of activities, duties, or responsibilities that are required.
Nothing in this job description restricts management's right to assign or reassign duties and responsibilities at any time.
Transact Campus Inc.
is an equal employment opportunity employer and considers qualified applicants for employment without regard to race, gender, age, color, religion, national origin, marital status, disability, sexual orientation, protected military/veteran status, or any other protected factor.
#J-18808-Ljbffr