Job Description
As a Test Automation Engineer, you will be responsible for designing and implementing automated testing frameworks for Databricks-based solutions, data pipelines, and data quality validation. You will work closely with data engineering teams to ensure data accuracy and reliability across our Lakehouse architecture.
Databricks Testing:
* Design and implement automated testing for Databricks notebooks and workflows
* Create test frameworks for Delta Lake tables and ACID transactions
* Develop automated validation for structured streaming pipelines
* Test MLflow integrations and model tracking
* Validate Delta Live Tables implementations
Data Pipeline Testing:
* Automate testing for ETL/ELT processes in Databricks
* Implement Spark job testing and optimization validation
* Create test cases for data ingestion and processing workflows
* Develop automated checks for data transformations
* Test Unity Catalog features and access controls
AWS Integration Testing:
* Implement automated testing for Databricks-AWS integrations
* Create test cases for S3, Glue catalog, and Lambda functions
* Validate data lake storage and access patterns
* Test secure networking configurations
Quality Assurance:
* Design and execute data quality test strategies
* Implement automated data reconciliation processes
* Develop performance testing for large-scale Spark jobs
* Create cluster configuration testing
Monitoring & Reporting:
* Implement pipeline monitoring test frameworks
* Create automated test dashboards
* Generate quality metrics and testing reports
* Maintain comprehensive test documentation