Overview
Lead Data Engineer
Mastercard is a global technology company. Our mission is to connect and power an inclusive, digital economy that benefits everyone, everywhere by making payment and data transactions safe, simple, smart, and accessible. The ML Engineering team leads AI/ML deployments across Mastercard platforms and is responsible for planning the implementation of solutions, choosing the right technologies, and evaluating the evolution of the architecture as needs change. Data is key for AI/ML. This role focuses on the implementation of Data Engineering for AI/ML across on-premise, cloud, and hybrid environments, working closely with AI Engineering / Data Science and platform teams.
Responsibilities
* Design and implement batch, real-time (RT) and near-real-time (NRT) data engineering pipelines between Mastercard systems to support AI deployments. Engineer scalable and efficient ETL/ELT processes to transform raw data into actionable insights and AI-driven velocities.
* Enhance data engineering solutions for reliability, maintainability, and scalability with improved deployment, monitoring, fault tolerance, features and processes.
* Develop monitoring solutions to assess the health and performance of systems.
* Participate in technical discussions, ensuring alignment with business goals and data engineering objectives.
* Reduce technical debt and pursue continuous improvement processes. KPI-first approaches.
* Ensure data security, integrity, privacy, and compliance. Implement data validation frameworks and robust security measures including anonymization and encryption for retrieval systems. Collaborate with compliance teams to align retrieval infrastructure with Mastercard’s data governance policies. Ensure the quality, accuracy, and integrity of datasets and implement data versioning, lineage tracking, and auditability systems.
* Innovate in data solution technologies, balancing performance with business and technology trade-offs. Integrate AI infrastructure for scalability and distributed systems to improve AI application performance. Implement solutions using platform-as-a-service (PaaS), containerization, and cloud-native architectures under Architect’s direction. Design and develop high-transaction-volume financial systems with global scalability and extreme uptime requirements. Build global-scale back-end microservices using Java, Kafka, RabbitMQ, and related technologies. Develop scalable data pipelines with Hadoop, Apache Spark, Spark SQL, Kafka, NiFi for batch and incremental loads. Deploy AI/ML pipelines in production with MLOps best practices. Continuously improve data systems to meet evolving AI needs and business objectives.
* Collaborate with AI and Product Engineering Teams to translate business requirements into technical solutions and integrate data platforms with Mastercard’s enterprise systems.
Corporate Security Responsibility
All activities involving access to Mastercard assets, information, and networks come with inherent risk. Every person working for or on behalf of Mastercard is responsible for information security and must:
* Abide by Mastercard’s security policies and practices.
* Ensure the confidentiality and integrity of information accessed.
* Report any suspected information security violation or breach.
* Complete periodic mandatory security trainings in accordance with Mastercard’s guidelines.
Required Skills & Qualifications
The number of years of experience is flexible if all other requirements are met.
* Demonstrated experience dealing with data engineering complexities of the 5 Vs of Big Data.
* Proven expertise in leading and driving real-time data pipelines and large-scale distributed systems.
* Hands-on experience with Hadoop, Apache Spark, Kafka, NiFi, and Big Data ecosystems.
* Strong knowledge of data security, compliance, and data governance standards.
* Proficiency in programming languages such as Java, Python, or Scala.
* Experience with microservices architecture and messaging frameworks (Kafka, RabbitMQ).
* Familiarity with cloud platforms (AWS) and container orchestration tools (Kubernetes, Docker).
* Background in designing and managing high-performance, scalable, secure financial systems with global-scale architecture.
* Ability to see the big picture with strong design skills as well as attention to implementation details.
* Aligned to Mastercard values: value creation, growth, speed for customers and colleagues.
* Good communication and presentation skills.
Nice to Have
* Experience in MLOps workflows and deploying machine learning models in production.
* Experience in AI projects with understanding of various ML algorithms and their impact on infrastructure.
#J-18808-Ljbffr