Address
SalaryDescription
The DevOps Engineer will establish the organization's data infrastructure, promoting engineering best practices on data management, and keeping the high bar for data quality. Responsibilities include but not limited to:
Work with our data customers to provide reliable and timely data
Be able to choose the appropriate level of data extraction when designing and implementing tables in the data lake
Write code that reliably extracts data from REST service calls and then loads into the data lake
Contribute to data operations with process documentation, service levels, data quality, performance dashboards, and continuous improvement efforts
Implement operational improvements through automation, requiring ingenuity and creativity, within data pipelines
Implement standards and procedures, work with the team to ensure conformity to standards and best practices
Incident management, triage the activities and work with stakeholders and communicate with timely updates
Prepare for and respond to audits for compliance with established standards, policies, guidelines and procedures
Collaborate with data engineering and data governance teams in release planning, training, and handoffs
Manage processes at various levels of maturity, while maintaining consistent quality and throughput expectations
Ensure visibility of all operational metrics, and related issues, to peers and leadership and proactively identify opportunities for operational efficiencies
Be part of our support rotation
Top Performers will be able to demonstrate:
Experience enriching data using third-party data sources
Working knowledge of the latest big data technologies and systems
Experience building out the tooling and infrastructure to make systems more self-service
Knowledge of tools like Databricks and Astronomer/Airflow to run workloads
Demonstrable experience of building internal tooling using Scala, ZIO, and Cats
Requirements
Bachelor's degree in a technical field
5 years of overall relevant experience
2 years of experience in the following:
Developing extract-transform-load (ETL)
Experience in working in both SQL and Python
Hands-on experience in data lakes
Hands-on experience in Airflow
Hands-on experience using Spark in Scala
Hands-on experience using Git performing merges
Experience using asynchronous REST calls to request data
Experience using NoSQL databases like Bigtable, Cassandra, HBase, etc.
Experience in functional programming
Experience using an effects library like Cats IO, ScalaZ, or Zio
Notes:
Position is remote.
Candidates should have experience with AWS, CI/CD, Scala (Apache Spark). Java, Python, C++ Java Data Mining.
