Data & Databricks Test Automation EngineerJoin to apply for the Data & Databricks Test Automation Engineer role at The Citco Group LimitedCitco is a global leader in financial services, delivering innovative solutions to some of the world's largest institutional clients. We are seeking a Test Automation Engineer specializing in Databricks and data platforms to ensure the quality and reliability of our data solutions.ResponsibilitiesAs a Data & Databricks Test Automation Engineer, you will be responsible for developing and implementing automated testing frameworks for Databricks-based solutions, data pipelines, and data quality validation.You will work closely with data engineering teams to ensure data accuracy and reliability across our Lakehouse architecture.Databricks Testing: Design and implement automated testing for Databricks notebooks and workflowsCreate test frameworks for Delta Lake tables and ACID transactionsDevelop automated validation for structured streaming pipelinesTest MLflow integrations and model trackingValidate Delta Live Tables implementationsData Pipeline Testing: Automate testing for ETL/ELT processes in DatabricksImplement Spark job testing and optimization validationCreate test cases for data ingestion and processing workflowsDevelop automated checks for data transformationsTest Unity Catalog features and access controlsAWS Integration Testing: Implement automated testing for Databricks-AWS integrationsCreate test cases for S3, Glue catalog, and Lambda functionsValidate data lake storage and access patternsTest secure networking configurationsQuality Assurance: Design and execute data quality test strategiesImplement automated data reconciliation processesDevelop performance testing for large-scale Spark jobsCreate cluster configuration testingMonitoring & Reporting: Implement pipeline monitoring test frameworksCreate automated test dashboardsGenerate quality metrics and testing reportsMaintain comprehensive test documentationQualificationsBachelor's degree in Computer Science, Data Science, or related fieldRelevant certifications in Databricks or data testing are a plus2+ years hands-on experience with Databricks (Spark)Strong programming skills in Python (PySpark) and SQLExperience with data testing frameworks and toolsKnowledge of AWS services (S3, Glue, Lambda)Understanding of Delta Lake and Lakehouse architectureExperience with version control systems (Git)Strong analytical and problem-solving abilitiesExperience with large-scale data processingKnowledge of data quality best practicesUnderstanding of data governance and compliance requirementsExperience with Agile methodologiesDatabricks workspace and notebook developmentDelta Lake and Delta Live TablesMLflow for ML pipeline testingUnity Catalog for governance testingSpark optimization and performance testingOur BenefitsYour well being is of paramount importance to us, and central to our success. We provide a range of benefits, training and education support, and flexible working arrangements to help you achieve success in your career while balancing personal needs. Ask us about specific benefits in your location.We embrace diversity, prioritizing the hiring of people from diverse backgrounds. Our inclusive culture is a source of pride and strength, fostering innovation and mutual respect.Citco welcomes and encourages applications from people with disabilities. Accommodations are available upon request for candidates taking part in all aspects of the selection.Seniority levelEntry levelEmployment typeFull-timeJob functionQuality AssuranceReferrals increase your chances of interviewing at The Citco Group Limited by 2xGet notified about new Quality Assurance Automation Engineer jobs in Cork, County Cork, Ireland.
#J-18808-Ljbffr