Data & Databricks Test Automation Engineer Company Overview Our client is a global leader in financial and technology-driven services, delivering innovative data solutions to some of the world's largest institutional clients.
They are seeking a Test Automation Engineer with a strong focus on Databricks and modern data platforms to ensure the quality, performance, and reliability of their data solutions.
Role Overview As a Data & Databricks Test Automation Engineer, you will be responsible for developing and implementing automated testing frameworks across Databricks-based data platforms.
You'll collaborate closely with data engineering teams to validate pipelines, ensure data quality, and strengthen the integrity of a modern Lakehouse architecture.
Key Responsibilities Databricks Testing Design and implement automated testing for Databricks notebooks and workflows.
Develop test frameworks for Delta Lake tables and ACID transactions.
Build automated validation processes for structured streaming pipelines.
Validate Delta Live Tables implementations.
Data Pipeline Testing Automate testing for ETL/ELT processes within Databricks.
Implement Spark job testing and performance validation.
Create and manage test cases for data ingestion and transformation workflows.
Test Unity Catalog configurations, access controls, and governance models.
Quality Assurance Design and execute data quality test strategies and reconciliation processes.
Implement performance testing for large-scale Spark jobs.
Ensure compliance with internal data governance and quality standards.
Monitoring & Reporting Develop test monitoring frameworks and dashboards.
Automate quality reporting and produce actionable test metrics.
Maintain clear test documentation and version control across projects.
Requirements & Qualifications Education Bachelor's degree in Computer Science, Data Science, Engineering, or related field.
Certifications in Databricks or data testing tools are advantageous.
Technical Skills 2+ years' hands-on experience with Databricks (Spark).
Strong programming experience with Python (Py Spark) and SQL.
Exposure to data testing frameworks and tools.
Familiarity with AWS services (S3, Glue, Lambda) or similar cloud platforms.
Knowledge of Delta Lake and Lakehouse architectures.
Proficiency with version control systems such as Git.
Additional Skills Strong analytical and problem-solving mindset.
Experience in large-scale data processing environments.
Understanding of data governance, compliance, and data quality best practices.
Previous experience working within Agile or Dev Ops teams.
Platform Knowledge Databricks workspace and notebook development.
Delta Lake and Delta Live Tables.
Unity Catalog testing for governance and permissions.
Spark optimization and performance analysis.