Responsibilities
1. Databricks Platform & Architecture Architect and maintain Databricks Lakehouse solutions using Delta Lake for ACID transactions and efficient data versioning. Leverage Databricks SQL Analytics for interactive querying and report generation. Manage cluster lifecycle (provisioning, sizing, scaling) and optimize Spark jobs for cost and performance. Implement structured streaming pipelines for near real-time data ingestion and processing. Configure and administer Databricks Repos, notebooks, and job scheduling/orchestration to streamline development workflows.
2. AWS Cloud Integration Integrate Databricks with AWS S3 as the primary data lake storage layer. Design and implement ETL/ELT pipelines using AWS Glue catalog, AWS Lambda, and AWS Step Functions where needed. Ensure proper networking configuration (VPC, security groups, private links) for secure andpliant data access. Automate infrastructure deployment and scaling using AWS CloudFormation or Terraform.
3. Data Pipeline & Workflow Management Develop and maintain scalable, reusable ETL frameworks using Spark (Python/Scala). Orchestrateplex workflows, applying CI/CD principles (Git-based version control, automated testing). Implement Delta Live Tables or similar frameworks to handle real-time data ingestion and transformations. Integrate with MLflow (if applicable) for experiment tracking and model versioning, ensuring data lineage and reproducibility.
4. Performance Tuning & Optimization Conduct advanced Spark job tuning (caching strategies, shuffle partitions, broadcast joins, memory optimization). Fine-tune Databricks clusters (autoscaling policies, instance types) to manage cost withoutpromising performance. Optimize I/O performance and concurrency for large-scale data sets.
5. Security &ernance Implement Unity Catalog or equivalent Databricks features for centralizedernance, access control, and data lineage. Ensurepliance with industry standards (, GDPR, SOC, ISO) and internal security policies. Apply IAM best practices across Databricks and AWS to enforce least-privilege access.
6. Technical Leadership & Mentorship Lead and mentor a team of data engineers, conducting code reviews, design reviews, and knowledge-sharing sessions. Champion Agile or Scrum development practices, coordinating sprints and deliverables. Serve as a primary technical liaison, working closely with product managers, data scientists, DevOps, and external stakeholders.
7. Monitoring & Reliability Configure observability solutions (, Datadog, CloudWatch, Prometheus) to proactively identify performance bottlenecks. Set up alerting mechanisms for latency, cost overruns, and cluster health. Maintain SLAs and KPIs for data pipelines, ensuring robust data quality and reliability.
8. Innovation & Continuous Improvement Stay updated on Databricks roadmap and emerging data engineering trends (, Photon, Lakehouse features). Evaluate new tools and technologies, driving POCs to improve data platform capabilities. Collaborate with business units to identify data-driven opportunities and craft solutions that align with strategic goals.
Qualifications
Qualifications
9. Educational Background Bachelor's or Master's degree inputer Science, Data Science, Engineering, or equivalent experience.
10. Technical Experience Databricks Expertise: 5+ years of hands-on Databricks (Spark) experience, with a focus on building and maintaining production-grade pipelines. AWS Services: Proven track record with AWS S3, EC2, Glue, EMR, Lambda, Step Functions, and security best practices (IAM, VPC). Programming Languages: Strong proficiency in Python (PySpark) or Scala; SQL for analytics and data modeling. Data Warehousing & Modeling: Familiarity with RDBMS (, Postgres, Redshift) and dimensional modeling techniques. Infrastructure as Code: Hands-on experience using Terraform or AWS CloudFormation to manage cloud infrastructure. Version Control & CI/CD: Git-based workflows (GitHub/GitLab), Jenkins or similar CI/CD tools for automated builds and deployments.
11. Leadership & Soft Skills Demonstrated experience leading a team of data engineers in aplex, high-traffic data environment. Outstandingmunication and stakeholder management skills, with the ability to translate technical jargon into business insights. Adept at problem-solving, with a track record of quickly diagnosing and resolving data performance issues.
12. Certifications (Preferred) Databricks Certified Associate/Professional (, Databricks Certified Professional Data Engineer). AWS Solutions Architect (Associate or Professional).
Job ID 300001274736084