Job Title: Databricks Engineer
We are seeking a highly skilled Databricks engineer to join our team. As a key member of our data engineering team, you will be responsible for designing and implementing automated testing for Databricks notebooks and workflows.
Key Responsibilities:
* Databricks Testing: Design and implement automated testing frameworks for Delta Lake tables and ACID transactions.
* Data Pipeline Testing: Automate testing for ETL/ELT processes in Databricks and implement Spark job testing and optimization validation.
* AWS Integration Testing: Implement automated testing for Databricks-AWS integrations and create test cases for S3, Glue catalog, and Lambda functions.
* Quality Assurance: Design and execute data quality test strategies and implement automated data reconciliation processes.
* Monitoring & Reporting: Implement pipeline monitoring test frameworks and create automated test dashboards.
Requirements:
* Programming Skills: Proficiency in languages such as Python, Scala, or Java.
* Databricks Experience: Experience working with Databricks notebooks and workflows.
* Testing Frameworks: Knowledge of testing frameworks such as Pytest or Unittest.
* Cloud Computing: Experience working with cloud-based platforms such as AWS or Azure.
* Collaboration: Strong communication and collaboration skills.
Benefits:
* Competitive Salary
* Opportunities for Growth
* Collaborative Work Environment
* Professional Development