Middle Data Engineer (Apache Spark)

AgileEngine is a top-ranking provider of software solutions to Fortune 500, Global 500, and Future 50 companies. Listed on Inc. 5000 among the fastest-growing US companies, we are always open to talented software, UX, and data experts in the Americas, Europe, and Asia.

If you like a challenging environment where you’re working with the best and are encouraged to learn and experiment daily, there’s no better place — guaranteed! :)

What you will do

• Maintain an evergrowing Datalake with a diverse set of data sources and data consumers;

• Help implement data governance initiative;

• Maintenance, deprecation, and eventual migration of legacy ETL jobs into a LakeHouse solution.

Must haves

• Airflow experience;

• Worked with Spark/Glue;

• IaC experience; CDK or Terraform;

• Familiar with Datalake/Datawarehouse/Lakehouse;

• Worked with Iceberg/Delta;

• 3+ years of experience with application development;

• Experience with AWS tools;

• Experience with data visualization tools;

• Experience designing, developing, operationalizing, and maintaining complex data applications at an enterprise scale;

• Experience with creating software for retrieving, parsing, and processing structured and unstructured data;

• Experience with building scalable ETL and ELT workflows for reporting and analytics;

• Experience with creating solutions in a collaborative, cross-functional team environment;

• Ability to develop scripts and programs for converting various data types into usable formats, support project team to scale, and monitor and operate data platforms;

• Ability to obtain a security clearance;

• Bachelor’s degree.

Nice to haves

• Airflow experience;

• Worked with Spark/Glue;

• IaC experience; CDK or Terraform;

• Familiar with Datalake/Datawarehouse/Lakehouse;

• Worked with Iceberg/Delta;

• 3+ years of experience with application development;

• Experience with AWS tools;

• Experience with data visualization tools;

• Experience designing, developing, operationalizing, and maintaining complex data applications at an enterprise scale;

• Experience with creating software for retrieving, parsing, and processing structured and unstructured data;

• Experience with building scalable ETL and ELT workflows for reporting and analytics;

• Experience with creating solutions in a collaborative, cross-functional team environment;

• Ability to develop scripts and programs for converting various data types into usable formats, support project team to scale, and monitor and operate data platforms;

• Ability to obtain a security clearance;

• Bachelor’s degree.

• Professional growth

Accelerate your professional journey with mentorship, TechTalks, and personalized growth roadmaps.

• Competitive compensation

We match your ever-growing skills, talent, and contributions with competitive USD-based compensation and budgets for education, fitness, and team activities.

• A selection of exciting projects

Join projects with modern solutions development and top-tier clients that include Fortune 500 enterprises and leading product brands.

• Flextime

Tailor your schedule for an optimal work-life balance, by having the options of working from home and going to the office – whatever makes you the happiest and most productive.