Overview
Senior Data Engineer (DATABRICKS) role at Mastercard
Get AI-powered advice on this job and more exclusive features.
Role Overview
As a Senior Databricks Administrator, you will be responsible for the setup, configuration, administration, and optimization of the Databricks Platform on AWS. This role will play a critical part in managing secure, scalable, and high-performing Databricks environments, with a strong focus on governance, user access management, cost optimization, and platform operations. You will collaborate closely with engineering, infrastructure, and compliance teams to ensure that the Databricks platform meets enterprise data and regulatory requirements.
Key Responsibilities
* Databricks Platform Administration (AWS): Provision, configure, and manage Databricks Workspaces across multiple environments (Dev, UAT, Prod).
* Define and enforce cluster policies, compute tagging, and usage controls to optimize cost and resource utilization.
* Administer Unity Catalog, including creation and management of catalogs, schemas, access control, and data lineage.
* Manage SCIM integrations for user/group provisioning with IdPs like Okta or Azure/AWS SSO.
* Configure and maintain CI/CD pipelines for notebook deployment and workflow promotion using GitHub, GitLab, or similar tools.
* Security, Governance & Compliance: Implement and enforce role-based access controls (RBAC) and fine-grained data permissions using Unity Catalog and Lake Formation.
* Ensure auditability and lineage tracking for compliance with Open Banking, GDPR, and PSD2 regulations.
* Configure and manage token scopes, secrets management, and credential passthrough for secure access to underlying data.
* Work with InfoSec and compliance teams to ensure platform security aligns with regulatory frameworks.
* Monitoring, Support & Troubleshooting: Monitor workspace performance, cluster health, job execution, and user activity using Databricks native tools, CloudWatch, and third-party integrations.
* Provide L2/L3 support for Databricks usage issues, including notebook errors, job failures, and workspace anomalies.
* Maintain operational runbooks, automation scripts, and alerting mechanisms for platform health and governance events.
* Automation & Best Practices: Build and manage Terraform/IaC scripts for environment provisioning and infrastructure consistency.
* Define naming conventions, resource tagging standards, and workspace governance guidelines.
* Promote standardization of reusable notebook templates, shared cluster configurations, and workflow orchestrations.
* Drive internal knowledge sharing, onboarding, and platform enablement sessions.
Must-have Skills
* 6+ years of experience in Databricks administration on AWS or multi-cloud environments.
* Deep understanding of Databricks workspace architecture, Unity Catalog, and cluster configuration best practices.
* Strong experience in managing IAM policies, SCIM integration, and access provisioning workflows.
* Hands-on experience with monitoring, cost optimization, and governance of large-scale Databricks deployments.
* Hands-on experience with infrastructure-as-code (Terraform) and CI/CD pipelines.
* Experience with ETL orchestration and collaboration with engineering teams (Databricks Jobs, Workflows, Airflow).
Good-to-Have Skills
* Understanding of Delta Lake, Lakehouse architecture, and data governance patterns.
* Experience in AWS Glue and database technologies SQL (Amazon Aurora) and NoSQL (MongoDB/DynamoDB).
* Exposure to regulated industries such as banking, fintech, or healthcare.
* Experience with incident response, audit logging, and security remediation.
Preferred Certifications
* Databricks Certified Data Engineer Associate/Professional
* Databricks Certified Lakehouse Platform Administrator
* AWS Certified Solutions Architect – Associate
Corporate Security Responsibility
All activities involving access to Mastercard assets, information, and networks comes with an inherent risk to the organization and, therefore, it is expected that every person working for, or on behalf of, Mastercard is responsible for information security and must:
* Abide by Mastercard’s security policies and practices;
* Ensure the confidentiality and integrity of the information being accessed;
* Report any suspected information security violation or breach; and
* Complete all periodic mandatory security trainings in accordance with Mastercard’s guidelines.
Seniority level
* Mid-Senior level
Employment type
* Full-time
Job function
* Information Technology
Industries
* Financial Services, IT Services and IT Consulting, and Technology, Information and Internet
#J-18808-Ljbffr