Senior Data Engineer – (Apache Spark)We are seeking an experiencedSenior Data Engineerto join a high-performing team. This role is central to designing, building, and evolving large-scale data platforms that power analytics, insights, and operational decision-making. You will bring strong expertise inApache Spark, contributing to the development and optimisation of distributed data pipelines within a modernLakehouse architecture. Working in a cloud-native environment built on technologies such asDelta Lake and Databricks, you'll play a key role in scaling and enhancing a data ecosystem that supports both batch and streaming workloads.Location:Limerick (Hybrid – minimum 3 days onsite per week during probation)ResponsibilitiesData Engineering & ArchitectureDesign, develop, and optimise scalable distributed data pipelines using Apache SparkBuild reliable batch and streaming solutions following modern Lakehouse design patternsDevelop and maintain workflows in a Databricks environment (or quickly ramp up if new)Apply advanced Spark techniques for performance tuning, partitioning strategies, and job optimisationIngest and process structured, semi-structured, and unstructured data sourcesTechnical LeadershipAct as a subject-matter expert for Spark-based data engineering best practicesLead and contribute to code reviews, ensuring high standards in quality, testing, and documentationHelp define data ingestion standards and guidelines to ensure consistency and reliabilityCollaboration & DeliveryPartner closely with architects and product stakeholders to translate business requirements into scalable data solutionsCollaborate with application and platform engineers to ensure seamless integration of data workflowsCommunicate technical concepts clearly to both technical and non-technical audiencesEnsure adherence to development processes, deployment standards, and change controlRequired Experience & Skills8+ years' experience in software or data engineering, with a strong focus on distributed data systemsDeep hands-on expertise with Apache Spark (DataFrames, Spark SQL, execution plans, shuffle optimisation)Experience with Databricks, or the ability to transition quickly with strong Spark fundamentalsStrong proficiency in Python and SQLSolid understanding of big data architectures, ETL/ELT pipelines, and cloud-native data patternsExperience with relational databases and version control systems (Git/TFS/SVN)Familiarity with CI/CD pipelines (e.g. Azure DevOps, GitHub Actions, Octopus Deploy)Strong communication skills and experience working in Agile environmentsFluent written and spoken EnglishPreferred SkillsHands-on production experience with Databricks (notebooks, workflows, Delta Lake, cluster configuration)Experience with Azure data services such as ADLS Gen2 and Azure Data FactoryKnowledge of Delta Live Tables, Delta Sharing, and Spark Structured StreamingExperience with NoSQL data stores and event-based ingestion (EventHub / EventGrid)Familiarity with Infrastructure-as-Code tools (Terraform, ARM templates)Understanding of data governance, cataloguing, and data management best practicesExposure to machine learning or data science concepts