Lead Bigdata Spark Engineer

About Client:
The client provides information technology (IT) services, including business outsourcing, infrastructure technology, and application services. The application service offered by the company includes application development, maintenance, and support. The markets served by the company are financial services and insurance, healthcare, manufacturing, government, transportation, communications, and consumer and retail industries.
Salary Range: $110K-$120K/Annum+ Benefits
Job Description:

• The data engineering role is a team member that will help enhance and maintain the CSS Business Analytics and Instant Ink Business Intelligence system.
• You will drive work you're doing to completion with hands-on development responsibilities, and partner with the Data Engineering leaders to implement data engineering pipelines to build solution to help provide trusted and reliable data to customers.
• Years of experience needed – 8+ years

• Design and implement distributed data processing pipelines using Spark, Python, SQL and other tools and languages prevalent in the Big Data/Lakehouse ecosystem.
• Analyzes design and determines coding, programming, and integration activities required based on general objectives.
• Play the technical lead role representing deliverables from vendor team resources at onsite and offshore locations.
• Lead the technical co-ordination and Business Knowledge transition activities to offshore team.
• Reviews and evaluates designs and project activities for compliance with architecture, security and quality guidelines and standards
• Writes and executes complete testing plans, protocols, and documentation for assigned portion of data system or component; identifies defects and creates solutions for issues with code and integration into data system architecture.
• Collaborates and communicates with project team regarding project progress and issue resolution.
• Works with the data engineering team for all phases of larger and more-complex development projects and engages with external users on business and technical requirements.
• Collaborates with peers, engineers, data scientists and project team.
• Typically interacts with high-level Individual Contributors, Managers and Program Teams on a daily/weekly basis.

• Bachelor's or Master's degree in Computer Science, Information Systems, Engineering or equivalent.
• 6+ years of relevant experience with detailed knowledge of data warehouse technical architectures, infrastructure components, ETL/ ELT and reporting/analytic tools.
• 3+ years of experience with Cloud based DW such as Redshift, Snowflake etc.
• 3+ years’ experience in Big Data Distributed ecosystems (Hadoop, SPARK, Unity Catalog & Delta Lake)
• 3+ years experience in Workflow orchestration tools such as Airflow etc.
• 3+ years’ experience in Big Data Distributed systems such as Databricks, AWS EMR, AWS Glue etc.
• Leverage monitoring tools/frameworks, like Splunk, Grafana, CloudWatch etc.
• Experience with container management frameworks such as Docker, Kubernetes, ECR etc.
• 3+ year’s working with multiple Big Data file formats (Parquet, Delta Lake)
• Experience working on CI/CD processes such as Jenkins, Codeway etc. and source control tools such as GitHub, etc.
• Strong experience in coding languages like Python, Scala & Java

• Fluent in relational based systems and writing complex SQL.
• Fluent in complex, distributed and massively parallel systems.
• Strong analytical and problem-solving skills with ability to represent complex algorithms in software.
• Strong understanding of database technologies and management systems.
• Strong understanding of data structures and algorithms
• Database architecture testing methodology, including execution of test plans, debugging, and testing scripts and tools.

• Experience with visualization tools such as PowerBI, Tableau etc.
• Experience with transformation tools such as dbt.
• Have experience in building realtime streaming data pipelines
• Experience in pub/sub streaming technologies like Kafka, Kinesis, Spark Streaming etc

About ApTask:
Join ApTask, a global leader in workforce solutions and talent acquisition services, as we shape the future of work. We offer a comprehensive suite of offerings, including staffing and recruitment services, managed services, IT consulting, and project management, providing unparalleled opportunities for professional growth and development. As a member of our dynamic team, you'll have the chance to connect businesses with top-tier professionals, optimize workforce performance, and drive success for our clients across diverse industries. If you are passionate about excellence, collaboration, and innovation, and aspire to make a meaningful impact in the world of work, come join us at ApTask and be a part of our mission to empower organizations to thrive.
Applicants may be required to attend interviews in person or by video conference. In addition, candidates may be required to present their current state or government issued ID during each interview.
 
Candidate Data Collection Disclaimer:
At ApTask, we prioritize safeguarding your privacy. As part of our recruitment process, certain Personally Identifiable Information (PII) may be requested by our clients for verification and application purposes. Rest assured, we strictly adhere to confidentiality standards and comply with all relevant data protection laws. Please note that we only collect the necessary information as specified by each client and do not request sensitive details during the initial stages of recruitment.
If you have any concerns or queries about your personal information, please feel free to contact our compliance team at businessexcellence@aptask.com