DescriptionData & Databricks Test Automation EngineerCompany OverviewCitco is a global leader in financial services, delivering innovative solutions to some of the world's largest institutional clients. We are seeking a Test Automation Engineer specializing in Databricks and data platforms to ensure the quality and reliability of our data solutions.Role DescriptionAs a Data & Databricks Test Automation Engineer, you will be responsible for developing and implementing automated testing frameworks for Databricks-based solutions, data pipelines, and data quality validation. You will work closely with data engineering teams to ensure data accuracy and reliability across our Lakehouse architecture.
Key Responsibilities1. Databricks TestingDesign and implement automated testing for Databricks notebooks and workflowsCreate test frameworks for Delta Lake tables and ACID transactionsDevelop automated validation for structured streaming pipelinesValidate Delta Live Tables implementations2. Data Pipeline TestingAutomate testing for ETL/ELT processes in DatabricksImplement Spark job testing and optimization validationCreate test cases for data ingestion and processing workflowsDevelop automated checks for data transformationsTest Unity Catalog features and access controls3. Quality AssuranceDesign and execute data quality test strategiesImplement automated data reconciliation processesDevelop performance testing for large-scale Spark jobs4. Monitoring & ReportingImplement pipeline monitoring test frameworksCreate automated test dashboardsGenerate quality metrics and testing reportsMaintain comprehensive test documentation
Requirements & Qualifications1. Educational BackgroundBachelor's degree in Computer Science, Data Science, or related fieldRelevant certifications in Databricks or data testing are a plus2. Technical Experience2+ years hands-on experience with Databricks (Spark)Strong programming skills in Python (PySpark) and SQLExperience with data testing frameworks and toolsKnowledge of AWS services (S3, Glue, Lambda)Understanding of Delta Lake and Lakehouse architectureExperience with version control systems (Git)3. Additional SkillsStrong analytical and problem-solving abilitiesExperience with large-scale data processingKnowledge of data quality best practicesUnderstanding of data governance and compliance requirementsExperience with Agile methodologies4. Platform KnowledgeDatabricks workspace and notebook developmentDelta Lake and Delta Live TablesUnity Catalog for governance testingSpark optimization and performance testing