Establish a robust data infrastructure to support business growth.
The ideal candidate will design, implement, and maintain scalable data platforms using SQL, Spark, and other technologies. They will collaborate with stakeholders to source and manage data from across the organization and ensure proper integration with other systems.
This role involves developing data pipelines, automating processes, and supporting analytics through visualization and self-service tools.
Data Engineer Responsibilities
* Design and develop scalable data pipelines using SQL, Spark, and other technologies to support the growing data Lakehouse
* Ensure that data platforms are properly architected for scalability, performance, and data integrity
* Work cross-functionally to integrate data platforms with other systems
* Develop ETL/ELT processes to ingest data from various sources and integrate them into existing data platforms
* Surface high-quality data for analytics, data visualization, and data science
* Collaborate with internal teams to gather requirements and translate them into technical solutions
* Help other members of the organization develop data skills and an understanding of data quality and governance
* Ensure data quality, consistency, security, and compliance across all projects
* Monitor and tune data platforms as needed to ensure a high level of service for users
* Create, maintain, and enhance documentation on systems and data products
* Ensure that code is properly version controlled and managed as it moves from development to production
Data Engineer Requirements
* Extensive experience with ETL/ELT concepts and tools for data integration and management
* Proficient in SQL Server, Microsoft Azure, Azure Functions, Azure Storage, Azure Data Factory, Azure Synapse, Azure Databricks, Microsoft Fabric
* Familiarity with API integration techniques and data management
* Experience with SQL, Python, Spark (Spark SQL & PySpark), Git
* Familiarity with other Microsoft data products such as Power BI, Power Automate, Power Platform
* Understanding of data modeling concepts, dimensional modeling, normalization, and medallion architecture
* Understanding of data security, data privacy and their related compliance standards
* Proficiency with Git for source control management and an understanding of CI/CD concepts
* Excellent problem-solving skills and the ability to troubleshoot complex data-related issues
* Strong communication skills to collaborate with technical and non-technical stakeholders
* Experience with Machine Learning and Artificial Intelligence a plus
* Bachelor's or Master's degree in Computer Science, Information Technology, or a related field
* Proven experience (5+ years) as a Data Warehouse Engineer or in a similar role
* Proficiency in designing and implementing data solutions using technologies such as SQL, Spark, Azure, or other comparable technologies
* Ability to travel to clients as needed
Data Engineer Benefits
* Health care plan
* Retirement plan
* Life insurance
* Paid time off
* Family leave
* Short term and long term disability
* Training and development opportunities
* Wellness resources
* Four weeks paid time off
* Twelve paid holidays