Address
Form of workDescription:
Skiltrek is seeking a Lead Software Engineer. Driving strategy for the team for entire platform and help with migration to cloud. Big Data Hub, the tools are used purely for ETL functions, moving and conforming data, not for building web applications. The use of Python and Java is for Data Engineering, we are not seeking Web Developers, back and front end etc. Job Expectations: Design and implement automated spark-based framework to facilitate data ingestion, transformation and consumption. Implement security protocols such as Kerberos Authentication, Encryption of data at rest, data authorization mechanism such as role-based access control using Apache ranger. Design and develop automated testing framework to perform data validation. Enhance existing spark-based frameworks to overcome tool limitations, and/or to add more features based on consumer expectations. Design and build high performing and scalable data pipeline platform using Hadoop, Apache Spark, MongoDB, Kafka and object storage architecture. Work with Infrastructure Engineers and System Administrators as appropriate in designing the big-data infrastructure Collaborate with application partners, Architects, Data Analysts and Modelers to build scalable and performant data solutions. Effectively work in a hybrid environment where legacy ETL and Data Warehouse applications and new big-data applications co-exist Work with Infrastructure Engineers and System Administrators as appropriate in designing the big-data infrastructure. Support ongoing data management efforts for Development, QA and Production environments Provide tool support, help consumers troubleshooting pipeline issues. Utilizes a thorough understanding of available technology, tools, and existing designs. Leverage knowledge of industry trends to build best in class technology to provide competitive advantage.
Skills:
Hadoop, Scala, Spark, Python, Java
Top Skills Details:
Hadoop,Scala,Spark
Additional Skills & Qualifications:
Desired Qualification 3+ years of Agile experience 2+ years of reporting experience, analytics experience or a combination of both 2+ years of operational risk or credit risk or compliance domain experience 2+ years of experience integrating with RESTful API 2+ years of experience with CICD tools.
Experience Level:
Expert Level
About Us
Skiltrek is an award-winning IT staffing firm and the staffing partner of choice for many leading companies across the US. At Skiltrek, we promise you the perfect opportunity of building technical excellence, understand business performance and nuances, be abreast with the latest happenings in technology world and enjoy a satisfying work life balance.
Skiltrek is committed to equal treatment and opportunity in all aspects of recruitment, selection, and employment without regard to gender, race, religion, national origin, ethnicity, disability, gender identity/expression, sexual orientation, veteran or military status, or any other category protected under the law. Skiltrek is an equal opportunity employer; committed to a community of inclusion, and an environment free from discrimination, harassment, and retaliation.
