Data Architect plays a pivotal role in designing and enhancing data architecture to support real-time and batch processing with a focus on scalability and fault tolerance using Kafka, Flink, and lakehouse principles.
">
* Architectural Leadership: Design and enhance our data architecture to support real-time and batch processing with a focus on scalability and fault tolerance using Kafka, Flink, and lakehouse principles.
* Data Semantics and Discovery: Implement systems and procedures for effective data semantics management, ensuring data is accurately categorized and easily discoverable.
* Pipeline Automation: Develop and maintain automated data pipelines that ensure efficient data flow and processing from multiple sources to our lakehouse architecture.
* Data Consumption Optimization: Create strategies and systems for optimal generation of materialized views and data subsumption to ensure that our data architecture remains cutting-edge, minimizes redundancy, and achieves required level of performance.
Responsibilities
* Work closely with data scientists, business analysts, and other engineering teams to define and refine requirements that drive data solutions.
* Stay abreast of the latest industry developments in data engineering and propose adoption of new technologies or methodologies to keep our data infrastructure ahead of the curve.