Databricks Architect

Title: Databricks Architect

Location: New York NY (Hybrid Model)

Duration: Full-time / Permanent

In this role, the Databricks Architect is responsible for providing technical direction and leading a group of one or more developers to address a goal.

Responsibilities

• Architect and design solutions to meet functional and non-functional requirements.
• Create and review architecture and solution design artifacts.
• Evangelize re-use through the implementation of shared assets.
• Enforce adherence to architectural standards/principles, global product-specific guidelines, usability design standards, etc.
• Proactively guide engineering methodologies, standards, and leading practices.
• Guidance of engineering staff and reviews of as-built configurations during the construction phase.
• Provide insight and direction on roles and responsibilities required for solution operations.
• Identify, communicate, and mitigate Risks, Assumptions, Issues, and Decisions throughout the full lifecycle.
• Considers the art of the possible, compares various architectural options based on feasibility and impact, and proposes actionable plans.
• Demonstrate strong analytical and technical problem-solving skills.
• Ability to analyze and operate at various levels of abstraction.
• Ability to balance what is strategically right with what is practically realistic.

Qualifications we seek in you!

Minimum Qualifications

• Excellent technical architecture skills, enabling the creation of future-proof, complex global solutions.
• Excellent interpersonal communication and organizational skills are required to operate as a leading member of global, distributed teams that deliver quality services and solutions.
• Ability to rapidly gain knowledge of the organizational structure of the firm to facilitate work with groups outside of the immediate technical team.
• Knowledge and experience in IT methodologies and life cycles that will be used.
• Familiar with solution implementation/management, service/operations management, etc.
• Leadership skills can inspire others and persuade.
• Maintains close awareness of new and emerging technologies and their potential application for service offerings and products.
• Bachelor's Degree or equivalency (CS, CE, CIS, IS, MIS, or engineering discipline) or equivalent work experience
• Experience in a solution architecture role using service and hosting solutions such as private/public cloud IaaS, PaaS, and SaaS platforms.
• Experience in architecting and designing technical solutions for cloud-centric solutions based on industry standards using IaaS, PaaS, and SaaS capabilities.
• Must have strong hands-on experience on various cloud services like ADF/Lambda, ADLS/S3, Security, Monitoring, Governance
• Must have experience in designing platforms on Databricks.
• Hands-on Experience to design and build Databricks-based solutions on any cloud platform.
• Hands-on experience to design and build solutions powered by DBT models and integrate with Databricks.
• Must be very good at designing End-to-End solutions on cloud platforms.
• Must have good knowledge of Data Engineering concepts and related services of the cloud.
• Must have good experience in Python and Spark.
• Must have good experience in setting up development best practices.
• Intermediate level knowledge is required for Data Modelling.
• Good to know about docker and Kubernetes.
• Experience with claims-based authentication (SAML/OAuth/OIDC), MFA, RBAC, SSO, etc.
• Knowledge of cloud security controls including tenant isolation, encryption at rest, encryption in transit, key management, vulnerability assessments, application firewalls, SIEM, etc.
• Experience building and supporting mission-critical technology components with DR capabilities.
• Experience with multi-tier system and service design and development for large enterprises
• Extensive, real-world experience designing technology components for enterprise solutions and defining solution architectures and reference architectures with a focus on cloud technologies.
• Exposure to infrastructure and application security technologies and approaches
• Familiarity with requirements-gathering techniques.

Preferred Qualifications/Skills

• Must have designed the E2E architecture of a unified data platform covering all the aspects of the data lifecycle starting from Data Ingestion, Transformation, Service, and consumption.
• Must have excellent coding skills either Python or Scala, preferably Python.
• Must have a good amount of experience in the Data Engineering domain.
• Must have designed and implemented at least 2-3 projects end-to-end in Databricks.
• Must have at least some experience on DataBricks which consists of various components as below
  • Delta Lake
  • DbConnect
  • DB API 2.0
  • SQL Endpoint Photon engine
  • Unity Catalog
  • Databricks workflows orchestration
  • Security management
  • Platform governance
  • Data Security
• Must have followed various architectural principles to design best suited per problem.
• Must be well versed with Databricks Lakehouse concept and its implementation in enterprise environments.
• Must have a strong understanding of Data warehousing and various governance and security standards around Databricks.
• Must know about cluster optimization and its integration with various cloud services.
• Must have a good understanding of creating complex data pipelines.
• Must be strong in SQL and sprak-sql.
• Must have strong performance optimization skills to improve efficiency and reduce cost.
• Must have worked on designing both Batch and streaming data pipelines.
• Must have extensive knowledge of Spark and Hive data processing frameworks.
• Must have worked on any cloud (Azure, AWS, GCP) and most common services like ADLS/S3, ADF/Lambda, CosmosDB/DynamoDB, ASB/SQS, Cloud databases.
• Must be strong in writing unit test cases and integration tests.
• Must have strong communication skills and have worked with cross-platform teams.
• Must have a great attitude towards learning new skills and upskilling the existing skills.
• Responsible for setting best practices around Databricks CI/CD.
• Must understand composable architecture to take fullest advantage of Databricks capabilities.
• Good to have Rest API knowledge.
• Good to have an understanding of cost distribution.
• Good to have if worked on a migration project to build a Unified data platform.
• Good to know about DBT.
• Experience around DevSecOps including docker and Kubernetes.
• Software development full lifecycle methodologies, patterns, frameworks, libraries, and tools
• Knowledge of programming and scripting languages such as JavaScript, PowerShell, Bash, SQL, Java, Python, etc.
• Experience with data ingestion technologies such as Azure Data Factory, SSIS, Pentaho, Alteryx
• Experience with visualization tools such as Tableau, Power BI
• Experience with machine learning tools such as mellow, Databricks AI/ML, Azure ML, AWS PageMaker, etc.
• Experience in distilling complex technical challenges to actionable decisions for stakeholders and guiding project teams by building consensus and mediating compromises when necessary.
• Experience coordinating the intersection of complex system dependencies and interactions
• Experience in solution delivery using common methodologies especially SAFe Agile but also Waterfall, Iterative, etc.
• Demonstrated knowledge of relevant industry trends and standards