Senior Data Architect Opportunity
As a seasoned data architect, you will be responsible for leading the design and implementation of scalable, secure, and high-performance data solutions on Databricks and AWS.
* Key Responsibilities:
* Design and manage Databricks-based Lakehouse platforms (Delta Lake, Spark, MLflow)
* Integrate with AWS services including S3, Glue, Lambda, and Step Functions
* Develop and optimize scalable ETL/ELT pipelines using Spark (Python/Scala)
* Automate infrastructure with Terraform or CloudFormation
* Ensure robust performance tuning of Spark jobs and cluster configurations
* Implement strong security governance using IAM, VPC, and Unity Catalog
Requirements
* Extensive experience in designing and implementing data solutions on Databricks and AWS
* Advanced knowledge of AWS services, including S3, Glue, Lambda, VPC, IAM, and EMR
* Strong coding skills in Python (PySpark), Scala, and SQL
* Expertise in CI/CD pipelines, Git-based workflows, and automated testing
* Familiarity with data modeling and warehousing (e.g., Redshift, Postgres)
* Proficient in orchestration and workflow tools (e.g., Airflow, Step Functions)