Principal Executive Consultant @ Cpl | Recruitment Expert IT/ Science & Engineering
Position Responsibilities
* Work in a dedicated data reporting and analytics team building a world-class data platform to produce data-driven insights for Transact Campus and our clients
* Analyze, interpret and orchestrate complex data across disparate sources comprising unstructured, semi-structured and structured datasets in streaming and batch modes
* Design and develop real-time data pipelines using the latest Databricks and Delta Lake Azure cloud technologies
* Collaborate with data consumers (reporting, analysis, or data science) to provide metrics that meet their needs
* Contribute to standards for data producers streaming data into the Lakehouse
* Test commercial software products using both manual and automated testing processes
* Support the application lifecycle during QA, UAT, and post-release phases
* Comply with and contribute to consistent development guidelines (coding, change control, build, versioning)
* Participate in peer code reviews
Required Skills
* Bachelor’s degree in Computer Science, IT or related field OR equivalent related work experience – preferably with a focus on Data Analytics
* 5+ years of experience in software enterprise-level Data Engineering
* Expertise in big data workloads
* Experience with data lakes and scale-out processing
* Relational database design and best practices
* Hands-on experience designing and developing Spark data pipelines
* Strong SQL and Python skills
* Understanding of cloud-based big data workloads
* Experience with ETL/ELT patterns, preferably using Databricks jobs
* Excellent technical documentation skills
* Experience with data lakes (HDFS, Azure Data Lake or AWS S3)
* Experience with source code management systems such as Git/TFS/SVN
* Experience working in Agile teams (Scrum, XP, Kanban)
* Ability to present ideas and insights to business stakeholders
* Fluency in written and spoken English
Preferred Skills
* Good understanding of Azure Data Services (Azure Databricks, Azure Data Factory, Azure Data Lake Gen 2)
* Experience with Databricks Delta Lake, Delta Sharing, and Delta Live Tables
* Experience with Spark Structured Streaming
* Experience with NoSQL databases
* Experience with Infrastructure as Code technologies such as Terraform or ARM
* Experience in Data Science and ML methodologies
* Experience with Azure services for streaming data (EventHub, EventGrid)
* Understanding of Data strategy including Data Governance and Data management
Seniority level: Mid-Senior level
Employment type: Full-time
Job function: Information Technology
Industry: Staffing and Recruiting
#J-18808-Ljbffr