Job DescriptionCitco is a global leader in financial services, delivering innovative solutions to some of the world's largest institutional clients. We are seeking a Test Automation Engineer specializing in Databricks and data platforms to ensure the quality and reliability of our data solutions.ResponsibilitiesAs a Data & Databricks Test Automation Engineer, you will be responsible for developing and implementing automated testing frameworks for Databricks-based solutions, data pipelines, and data quality validation. You will work closely with data engineering teams to ensure data accuracy and reliability across our Lakehouse architecture.Key ResponsibilitiesDatabricks TestingDesign and implement automated testing for Databricks notebooks and workflowsCreate test frameworks for Delta Lake tables and ACID transactionsDevelop automated validation for structured streaming pipelinesTest MLflow integrations and model trackingValidate Delta Live Tables implementationsData Pipeline TestingAutomate testing for ETL/ELT processes in DatabricksImplement Spark job testing and optimization validationCreate test cases for data ingestion and processing workflowsDevelop automated checks for data transformationsTest Unity Catalog features and access controlsAWS Integration TestingImplement automated testing for Databricks-AWS integrationsCreate test cases for S3, Glue catalog, and Lambda functionsValidate data lake storage and access patternsTest secure networking configurationsQuality AssuranceDesign and execute data quality test strategiesImplement automated data reconciliation processesDevelop performance testing for large-scale Spark jobsCreate cluster configuration testingMonitoring & ReportingImplement pipeline monitoring test frameworksCreate automated test dashboardsGenerate quality metrics and testing reportsMaintain comprehensive test documentationQualificationsEducational BackgroundBachelor's degree in Computer Science, Data Science, or related fieldRelevant certifications in Databricks or data testing are a plusTechnical Experience2+ years hands-on experience with Databricks (Spark)Strong programming skills in Python (PySpark) and SQLExperience with data testing frameworks and toolsKnowledge of AWS services (S3, Glue, Lambda)Understanding of Delta Lake and Lakehouse architectureExperience with version control systems (Git)Additional SkillsStrong analytical and problem-solving abilitiesExperience with large-scale data processingKnowledge of data quality best practicesUnderstanding of data governance and compliance requirementsExperience with Agile methodologiesPlatform KnowledgeDatabricks workspace and notebook developmentDelta Lake and Delta Live TablesMLflow for ML pipeline testingUnity Catalog for governance testingSpark optimization and performance testingOur BenefitsYour well being is of paramount importance to us, and central to our success. We provide a range of benefits, training and education support, and flexible working arrangements to help you achieve success in your career while balancing personal needs. Ask us about specific benefits in your location.We embrace diversity, prioritizing the hiring of people from diverse backgrounds. Our inclusive culture is a source of pride and strength, fostering innovation and mutual respect.Citco welcomes and encourages applications from people with disabilities. Accommodations are available upon request for candidates taking part in all aspects of the selection.