Lead Data Engineer

Lead Data Engineer - Norwalk, CT - in-office work schedule

Position Overview:

This position encompasses the entire lifecycle of the Data Warehouse environments including the design, development, implementation, operation, and ongoing support of these critical systems. The advancement of our On-Premise and Cloud Data Warehouses to effectively meet crucial business needs will also be a crucial responsibility.

Lead Data Engineer

Job Duties and Responsibilities:

The Principal Data Engineer responsibilities include:

• Design, develop, implement, and support our enterprise Microsoft Azure Data Warehouse.
• Provide support in designing and overseeing enterprise-grade data pipelines and data stores.
• Maintain compliance with SOX and all other pertinent regulatory and compliance policies and requirements.
• Recognize, devise, and execute internal process enhancements, including automation of manual tasks, optimizing data delivery, and redesigning architecture or infrastructure to enhance scalability.
• Collate large, intricate datasets that align with functional and non-functional business demands.
• Develop processes that facilitate data transformation, manage data structures, metadata, dependencies, and workload management.
• Collaborate with business users to understand functional and data requirements.
• Apply expert-level analytical and problem-solving skills to diagnose and resolve intricate technical issues.
• Create, maintain, and continuously enhance scalable data pipelines, while also developing new data source integrations to accommodate the growing volume and complexity of data.
• Designing, implementing, and managing data extraction, transformation, and loading (ETL) processes.
• Create comprehensive technical specification documents and application interface designs.
• Create data processing and integration solutions for both batch and real-time scenarios, proficiently handling structured and unstructured data.
• Leading and participating in design discussions, code reviews, and project-related team meetings.
• Providing mentoring and guidance to junior team members, promoting a culture of knowledge sharing and collaborative problem-solving.
• Ensuring data security and compliance with relevant regulations and best practices in all data operations.
• Aid in troubleshooting and resolving data and system issues, as necessary

Lead Data Engineer

Qualifications/Competencies:

· Bachelor’s degree or higher in Information Technology or related field.

· 10+ years of hands-on Data Warehouse architecture and development experience

· Demonstrated expertise in Microsoft SQL Server and/or Azure development.

· Exhibit a thorough understanding of Data Lake architectures, including raw, enriched, and curated layer concepts, and ETL/ELT operations.

· Understand database design, data warehousing concepts, big data platforms, and ETL operations.

· Experience deploying modern data solutions leveraging components like Azure functions, Azure Data Factory, Data Flows, Azure Data Lake, Azure SQL, Azure Synapse, Streaming Analytics or equivalent on another cloud provider such as AWS or Google Cloud.

· Experience with DevOps tools like Azure DevOps, Jenkins, Maven etc.

· Experience in building/operating/maintaining fault tolerant and scalable data processing integrations.

· Strong level of understanding on Azure Data Factory, SQL/Synapse, ADLS, and Azure DevOps.

· Ability to conduct data profiling, cataloging, and mappings for technical design and construction of data flows.

· Strong problem-solving skills with emphasis on optimization data pipelines.

· Experience working with third party providers and vendors for critical support requirements.