Position Responsibilities * Work in a dedicated data reporting and analytics team building a world-class data platform to produce data-driven insights for Transact Campus and our clients * Analyze, interpret and orchestrate complex data across disparate sources comprising unstructured, semi-structured and structured datasets in streaming and batch modes * Design and develop real-time data pipelines using the latest Databricks and Delta Lake Azure cloud technologies * Collaborate with data consumers (reporting, analysis, or data science) to provide metrics that meet their needs * Contribute to standards for data producers streaming data into the Lakehouse * Test commercial software products using both manual and automated testing processes * Support the application lifecycle during QA, UAT, and post-release phases * Comply with and contribute to consistent development guidelines (coding, change control, build, versioning) * Participate in peer code reviews Required Skills * Bachelors degree in Computer Science, IT or related field OR equivalent related work experience preferably with a focus on Data Analytics * 5+ years of experience in software enterprise-level Data Engineering * Expertise in big data workloads * Experience with data lakes and scale-out processing * Relational database design and best practices * Hands-on experience designing and developing Spark data pipelines * Strong SQL and Python skills * Understanding of cloud-based big data workloads * Experience with ETL/ELT patterns, preferably using Databricks jobs * Excellent technical documentation skills * Experience with data lakes (HDFS, Azure Data Lake or AWS S3) * Experience with source code management systems such as Git/TFS/SVN * Experience working in Agile teams (Scrum, XP, Kanban) * Ability to present ideas and insights to business stakeholders * Fluency in written and spoken English Preferred Skills * Good understanding of Azure Data Services (Azure Databricks, Azure Data Factory, Azure Data Lake Gen 2) * Experience with Databricks Delta Lake, Delta Sharing, and Delta Live Tables * Experience with Spark Structured Streaming * Experience with NoSQL databases * Experience with Infrastructure as Code technologies such as Terraform or ARM * Experience in Data Science and ML methodologies * Experience with Azure services for streaming data (EventHub, EventGrid) * Understanding of Data strategy including Data Governance and Data managemen Skills: data engineer Software IT