Address
Form of work
SalaryJob Description:
About this role:
As a Data Engineer on the ML Platform Team, you will be the backbone of our feature store. You will play a crucial role in ensuring timely and high-quality feature availability for our ML models, working closely with data scientists, ML engineers, and other stakeholders. You will report to the Machine Learning Platform Engineering Manager.
Key Responsibilities:
Lead the onboarding process of feature stores, ensuring efficient integration with our ML platforms.
Design, develop, and maintain scalable and reliable data pipelines for feature extraction, transformation, and loading (ETL) processes.
Design, develop, and maintain scalable and reliable data pipelines for feature extraction, transformation, and loading (ETL) processes.
Monitor feature quality and identify potential issues through data analysis and visualization.
Troubleshoot and resolve feature-related issues impacting ML models or downstream applications.
Work closely with data scientists and analysts to understand their data requirements and build or support features that power machine learning models.
Debug and optimize data pipelines to ensure high data quality and availability.
Contribute to data modeling and architecture to improve the performance of our data lakehouse and data warehouse environments.
Provide technical guidance and support to stakeholders regarding data querying, access, and usage.
Strong ability to debug and optimize Spark application running in EMR.
Troubleshoot and resolve feature-related issues impacting ML models or downstream applications.
Work closely with data scientists and analysts to understand their data requirements and build or support features that power machine learning models.
Debug and optimize data pipelines to ensure high data quality and availability.
Contribute to data modeling and architecture to improve the performance of our data lakehouse and data warehouse environments.
Provide technical guidance and support to stakeholders regarding data querying, access, and usage.
Strong ability to debug and optimize Spark application running in EMR.
Qualifications:
Minimum 5 years of experience in Data Engineering with a focus on machine learning platforms.
In-depth knowledge of data warehouse and data lake architectures, including storage and processing engines.
Proficiency in data modeling and experience with large-scale data processing systems like Apache Spark, Flink.
In-depth knowledge of data warehouse and data lake architectures, including storage and processing engines.
Proficiency in data modeling and experience with large-scale data processing systems like Apache Spark, Flink.
Experience with AWS, git, CI/CD and DataOps best practices
Strong experience with AWS cloud services, particularly in data-related technologies (e.g., EMR, S3, Redshift).
Familiarity with feature store concepts and experience in building and maintaining feature stores.
Good understanding of real-time data processing is a plus.
Excellent problem-solving skills and the ability to debug complex data issues.
Strong communication skills to effectively collaborate with cross-functional teams and support stakeholders' data needs.
Strong experience with AWS cloud services, particularly in data-related technologies (e.g., EMR, S3, Redshift).
Familiarity with feature store concepts and experience in building and maintaining feature stores.
Good understanding of real-time data processing is a plus.
Excellent problem-solving skills and the ability to debug complex data issues.
Strong communication skills to effectively collaborate with cross-functional teams and support stakeholders' data needs.
Remitly is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees.
