Qa Data Engineer

The QA Data Engineer is a critical team member responsible for ensuring the accuracy, reliability, and performance of data systems within our organization. This role involves designing and executing comprehensive testing strategies for data pipelines, ETL processes, and data transformations. The QA Data Engineer collaborates closely with Data Engineers, data scientists, and cross-functional teams to understand data requirements, contributing valuable insights to improve data quality. Proficiency in SQL, scripting languages, and testing frameworks is essential, as is a solid understanding of Data Engineering principles and technologies. This position requires meticulous attention to detail, strong analytical skills, and the ability to identify and address data quality issues. The QA Data Engineer also plays a vital role in implementing and maintaining automated testing processes, ensuring continuous improvement of our data infrastructure.
Job Duties and Responsibilities:
Project Delivery (Testing) (40%):

• Develop and implement comprehensive test plans and strategies for Data Engineering projects, focusing on ETL jobs, SQL queries, and stored procedures.
• Ensure thorough coverage of data pipelines, transformations, and Snowflake target validation through investigations, pipeline results, feedback logs, etc.
• Design, create, and execute detailed test cases and scripts, emphasizing validation of ETL job outputs, SQL queries, and the correct execution of stored procedures.
• Implement and execute robust data validation checks and quality assurance measures to guarantee the accuracy, completeness, and consistency of data processed through ETL pipelines, SQL queries, and stored procedures.
• Conduct performance testing to assess and optimize the speed, scalability, and efficiency of ETL processes, SQL queries, and Snowflake target loading.

Programming, Scripting, and Automation (30%):

• Develop and maintain automated testing scripts and frameworks, with a focus on ETL job automation, SQL query validation, and Snowflake target validation, using scripting languages (e.g., Python, Ruby) and tools such as Cypress.
• Collaborate closely with Data Engineers, data scientists, and stakeholders to validate the functionality of ETL jobs, SQL queries, and stored procedures.
• Ensure strict adherence to QA standards, best practices, and methodologies, particularly in ETL processes, SQL queries, and Snowflake data loading processes.
• Implement and manage continuous integration and continuous testing practices, emphasizing ETL job automation, SQL query validation, and Snowflake target validation for ongoing data quality and reliability.

Communication (20%):

• Identify, document, and communicate defects, anomalies, and issues within ETL processes, SQL queries, and stored procedures.
• Collaborate with development teams to facilitate swift resolution of identified issues.
• Effectively communicate testing results and insights to cross-functional teams.

Extended Learning and Risk Management (10%):

• Conduct root cause analysis for data quality issues, focusing on ETL job failures, SQL query inaccuracies, and Snowflake target loading discrepancies.
• Stay updated on industry trends, emerging technologies, and best practices in Data Engineering and quality assurance

Qualifications (Education, Experience, Certifications & KSA):

• Bachelor's degree required or 3-5 years of relevant experience in quality assurance, Data Engineering, or a related field. Practical experience in testing data pipelines, ETL processes, and data transformations is essential.
• Moderate (3-5 years) proficiency working with the following technologies, processes, and tools:
  • Tools
    • Azure DevOps (ADO), JIRA / JIRA X-RAY, HPQC
    • SQL / Snowflake
    • ETL - Informatica, Airflow, ODI, SSIS, AWS Glue
    • Cypress (nice to have)
  • Languages
    • Python, Java, SQL
  • Processes
    • Scrum / Agile
    • SCALE / SAFe
• General understanding and participant in DevOps
• Understanding of software development lifecycle (SDLC)
• Ability to write test cases in a manner that can be re-used across regression and for automation candidates.
• Highly self-motivated and self-directed. Experience working in a team-oriented, collaborative environment.
• Excellent written and oral communication skills
• Excellent listening and interpersonal skills
• Ability to communicate ideas in both technical and user-friendly language
• Keen attention to detail
• Able to work in a team-oriented, collaborative environment.
• Ability to work both independently and on a team

The above statements are intended to describe the general nature and level of work being performed by people assigned to this job. They are not intended to be an exhaustive list of all responsibilities, skills, efforts or working conditions associated with a job.
#LI-ID1
We offer our employees a robust compensation package! Our comprehensive benefits include: medical, dental and vision insurance coverage; 100% company-paid life and disability coverage, 401k options with company match, three weeks PTO by the end of the first year and much more. Allied proudly promotes from within as part of a strong commitment to providing career growth opportunities for employees of all levels. Our diverse business portfolio allows employees broad career options with the advantage of staying with the same organization.
All qualified candidates will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability status, protected veteran status, or any other characteristic protected by law.
To view our privacy statement click here
To view our terms and conditions click here