Apache Spark Platform Engineer

Apache Spark Platform Engineer-Only W2

REMOTE
Long Term

Must have skills:-
Data Mesh
Virtualization
Metadata

Responsibilities :
Apache Spark:
Strong understanding and hands-on experience with Apache Spark, including Spark SQL, Spark Streaming.

Databricks:
Proficiency in using Azure Databricks, including setting up and managing clusters, notebooks, and jobs.

Python and SQL:
Strong programming skills in either Python and SQL as these are the primary languages used for writing Spark applications.

Data Processing and Analysis:
Experience in designing and implementing data processing and analysis pipelines using Spark and Databricks.

Distributed Computing:
Knowledge of distributed computing concepts and experience with distributed computing frameworks like Spark.

Cloud Platform:
Familiarity with Azure cloud platform and its services, including Azure Data Factory, Azure access, Azure Storage, Azure Data Lake, and Azure SQL Database.
Data Engineering:

Understanding of data engineering concepts and experience with ETL processes, data modeling, and data warehousing.

Performance Optimization:
Ability to optimize Spark applications for performance and scalability, including tuning Spark configurations and leveraging Spark optimizations.

Data Security and Governance:
Understanding of data security and governance practices, including data encryption, access controls, and compliance regulations.

Monitoring and Troubleshooting:
Proficiency in monitoring and troubleshooting Spark applications, identifying and resolving performance bottlenecks or issues.

Collaboration and Communication:
Excellent communication skills and the ability to collaborate effectively with cross-functional teams, including data scientists, data engineers, and business stakeholders. Desired skills: •Must know Spark, Databricks, Azure, and Linux OS experience.
•Experience in Spark platform performance improvements and Capacity planning. •Knowledge of cloud platform-based database troubleshooting.
•Experience with CI/CD pipelines, version control systems (e.g., Git), and automated deployment of Spark applications.

Job Type: Contract

Pay: $43.00 - $45.00 per hour

Schedule:

• 8 hour shift

Experience:

• Data Mesh: 5 years (Required)
• Virtualization: 5 years (Required)
• Metadata: 9 years (Required)
• Apache Spark:: 8 years (Required)
• Databricks:: 6 years (Required)
• Python and SQL:: 8 years (Required)
• Data Processing and Analysis:: 1 year (Required)
• Azure Data Factory, Azure access, Azure Storage: 6 years (Required)
• Azure Data Lake, and Azure SQL Database.: 6 years (Required)

Work Location: Remote