Address
Form of workJob Description
Overview
The Data Engineer is responsible for developing, maintaining, and evaluating big data solutions across functional / process content areas. The profile uses data management best practices and innovative, fit-for- purpose technologies to build stable, flexible, high-performing data & analytics products that help reduce the lead time from data collection to insight generation. It assembles large, complex data sets (e.g. text, time series, imagery) into a useful format for analysis using fit-for- purpose database technologies and designs, builds and orchestrates ETL pipelines.
Required:
Strategy
Develops strategies to identify, acquire and use appropriate data sets to develop practical solutions and support decision making
Defines the strategy and engineering guidelines for major data platforms jointly with the data architect.
Governance
Ensures to implement efficient data access control mechanisms to different raw data sources both on-premises and cloud environments -Identifies, designs, and implements internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability.
Execution
Creates and maintains optimal data pipeline architecture
Builds an infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using various state of the art and big data technologies
Architects, builds and manages a robust, scalable, data pipeline in a hybrid-cloud environment that provides data processing capabilities to meet functional / non-functional business requirements
Responsible for establishing and continuously improving sustainable, efficient, scalable, adaptable data processes and data processing applications
Supports the development of continuous integration processes and infrastructure
Analyzes large and complex data sets and models to gain insights and build suitable data models
Assembles large, complex data sets (e.g. text, time series, imagery) into a useful format for analysis using fit-for-purpose database technologies
Builds services and tools to make data more accessible to all data consumers (tools such as searchable metadata, API gateway)
Ensures end-to-end pipeline orchestration
Supports the further development of existing enterprise data analytics solutions and be available as a contact person for data support -Responsible for participating in the data warehouse by continuously improving ELT and ETL processes for the data lakes and data warehouses
Digital change
Supports software architecture decisions by writing, maintaining, and reviewing software code and ensures the code quality by automated testing
Builds semantic data layers and knowledge graphs using fit-for-purpose methodology and technologies e.g. linked data concept, RDF, triple stores, graph databases to make core data structures that have already been discovered findable, accessible and reusable in an efficient way to for data knowledge workers
Works with the platform development team to design & use tools for data logging and repeatable data tasks to accelerate and automate data scientist duties
Responsible for processing, cleaning, structuring, and improving data model and building processes to support company-wide analytics
Education
Bachelors degree in computer science, management information systems, or a related discipline
Working Experience
Experience in data architecture development, data asset management, data modelling, and linked data concepts as well as data taxonomy creation
Experience with distributed databases (e.g. Cassandra) and NoSQL databases like HBase, MongoDB, Snowflake or Object stores like Blob or S3 and GraphDBs etc.
-Experience with object-oriented, object function, scripting languages (e.g., Go, Python, Scala, Java, C++, etc.)
-Profound experience in information architecture, data science, business intelligence or equivalent areas
-Experience in user experience modelling, information design, and concept generation
-Strong capability to manage implementation and maintenance of data architecture concepts and related platforms and data pipelines
-Able to draw insights from structured and unstructured data
-Relational databases, knows SQL dialects like e.g. T-SQL and PSQL
-Big data technologies (Hadoop, Hive, HBase, Spark, etc.)
-Competent in data systems design and data visualization techniques
-Ability to perform business domain analysis and business process modelling
-Knowledge of data pipeline and workflow management tools (e.g., Airflow, DataFactory, Azkaban, Luigi, etc.)
-Knowledge of cloud computing (e.g., AWS, Azure)
-Knowledge of version control systems (e.g., GitHub, GitLab, etc.)
-Basic knowledge of network security, enterprise security, cloud security, database security
-Knowledge of usability design and data warehousing techniques
-Understanding of data quality standards and data cleaning processes
Personal
-Verbal and written communication skills
-Ability to challenge and convince the various stakeholders involved in any project
The Data Engineer is responsible for developing, maintaining, and evaluating big data solutions across functional / process content areas. The profile uses data management best practices and innovative, fit-for- purpose technologies to build stable, flexible, high-performing data & analytics products that help reduce the lead time from data collection to insight generation. It assembles large, complex data sets (e.g. text, time series, imagery) into a useful format for analysis using fit-for- purpose database technologies and designs, builds and orchestrates ETL pipelines.
Required:
- GO, Kubernates, Geospatial, GCP
- looking for super senior developer
- Data engineer - responsible for developing, maintaining, and evaluating big data solutions
- Creates and maintains optimal data pipeline
- ETL for data warehouse, data lake
- Big data and NoSQL Databases
- Machine Learning
Strategy
Develops strategies to identify, acquire and use appropriate data sets to develop practical solutions and support decision making
Defines the strategy and engineering guidelines for major data platforms jointly with the data architect.
Governance
Ensures to implement efficient data access control mechanisms to different raw data sources both on-premises and cloud environments -Identifies, designs, and implements internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability.
Execution
Creates and maintains optimal data pipeline architecture
Builds an infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using various state of the art and big data technologies
Architects, builds and manages a robust, scalable, data pipeline in a hybrid-cloud environment that provides data processing capabilities to meet functional / non-functional business requirements
Responsible for establishing and continuously improving sustainable, efficient, scalable, adaptable data processes and data processing applications
Supports the development of continuous integration processes and infrastructure
Analyzes large and complex data sets and models to gain insights and build suitable data models
Assembles large, complex data sets (e.g. text, time series, imagery) into a useful format for analysis using fit-for-purpose database technologies
Builds services and tools to make data more accessible to all data consumers (tools such as searchable metadata, API gateway)
Ensures end-to-end pipeline orchestration
Supports the further development of existing enterprise data analytics solutions and be available as a contact person for data support -Responsible for participating in the data warehouse by continuously improving ELT and ETL processes for the data lakes and data warehouses
Digital change
Supports software architecture decisions by writing, maintaining, and reviewing software code and ensures the code quality by automated testing
Builds semantic data layers and knowledge graphs using fit-for-purpose methodology and technologies e.g. linked data concept, RDF, triple stores, graph databases to make core data structures that have already been discovered findable, accessible and reusable in an efficient way to for data knowledge workers
Works with the platform development team to design & use tools for data logging and repeatable data tasks to accelerate and automate data scientist duties
Responsible for processing, cleaning, structuring, and improving data model and building processes to support company-wide analytics
Education
Bachelors degree in computer science, management information systems, or a related discipline
Working Experience
Experience in data architecture development, data asset management, data modelling, and linked data concepts as well as data taxonomy creation
Experience with distributed databases (e.g. Cassandra) and NoSQL databases like HBase, MongoDB, Snowflake or Object stores like Blob or S3 and GraphDBs etc.
-Experience with object-oriented, object function, scripting languages (e.g., Go, Python, Scala, Java, C++, etc.)
-Profound experience in information architecture, data science, business intelligence or equivalent areas
-Experience in user experience modelling, information design, and concept generation
-Strong capability to manage implementation and maintenance of data architecture concepts and related platforms and data pipelines
-Able to draw insights from structured and unstructured data
-Relational databases, knows SQL dialects like e.g. T-SQL and PSQL
-Big data technologies (Hadoop, Hive, HBase, Spark, etc.)
-Competent in data systems design and data visualization techniques
-Ability to perform business domain analysis and business process modelling
-Knowledge of data pipeline and workflow management tools (e.g., Airflow, DataFactory, Azkaban, Luigi, etc.)
-Knowledge of cloud computing (e.g., AWS, Azure)
-Knowledge of version control systems (e.g., GitHub, GitLab, etc.)
-Basic knowledge of network security, enterprise security, cloud security, database security
-Knowledge of usability design and data warehousing techniques
-Understanding of data quality standards and data cleaning processes
Personal
-Verbal and written communication skills
-Ability to challenge and convince the various stakeholders involved in any project
Powered by JazzHR
kkF8JEOBhc
