As a senior data engineering leader, you will oversee the design and implementation of large-scale data solutions on Databricks and AWS. Your team will focus on developing cutting-edge data platforms that meet the organization's growing demands for analytics and insights.
Key Responsibilities
* Platform Engineering: You will be responsible for architecting and managing Databricks-based Lakehouse platforms (Delta Lake, Spark, MLflow) to ensure seamless integration with existing systems.
* AWS Integration: Integrate our applications with various AWS services including S3, Glue, Lambda, and Step Functions to enable efficient data processing and analytics.
* Pipeline Development: Design and optimize scalable ETL/ELT pipelines using Spark (Python/Scala) to process large volumes of data in real-time.
* Infrastructure Automation: Automate infrastructure setup using Terraform or CloudFormation to reduce manual errors and increase deployment speed.
* Performance Tuning: Ensure robust performance tuning of Spark jobs and cluster configurations to minimize latency and maximize throughput.
* Security Governance: Implement strong security governance using IAM, VPC, and Unity Catalog to protect sensitive data and prevent unauthorized access.
* Team Leadership: Lead a high-performing engineering team through Agile delivery cycles, providing guidance and mentorship to ensure successful project execution.
Required Skills
* Leadership Experience: Proven track record of leading cross-functional teams in a technical environment.
* Data Engineering Expertise: In-depth knowledge of data engineering principles, tools, and technologies.
* Cloud Computing: Extensive experience in designing and implementing cloud-based data solutions on AWS.
* Programming Skills: Strong coding skills in Python (PySpark), Scala, and SQL.
* CI/CD Pipelines: Expertise in setting up and maintaining CI/CD pipelines using tools like Git, Jenkins, or CircleCI.
* Orchestration Tools: Familiarity with orchestration tools such as Airflow, Step Functions, or Apache NiFi.
At our company, we offer a dynamic work environment that fosters innovation, collaboration, and growth. If you are passionate about building cutting-edge data solutions and leading high-performing teams, we encourage you to apply for this exciting opportunity.