What is a Data Engineer?
The data engineer is in charge of changing the data format for easy analysis. They carry out their duties by developing and testing the data creation infrastructure. Data engineers assist data scientists in creating a data architecture concept that enables them to complete their work. Data engineers generally have a bachelor's degree in informatics engineering and applied mathematics. Some companies also require the possession of a data engineering certification from a reputable institution, such as an IBM Certified Data Engineer or Google Certified Professionals.
Job Overview
[Company name] is a company engaged in [company field] looking for a data engineer. The data engineer sought is tasked with expanding and optimizing the data architecture and data collection flow to support the performance of software developers, database architects, data analysts, data scientists, and other related teams. The ideal candidates for our company are those who are able to work independently in providing data for system and product development purposes by various teams within the company. In addition, they must also have an interest in developing and redesigning the company's data architecture on an ongoing basis.
Responsibilities and Duties
-
Create and maintain an optimal data architecture.
-
Collect complex data for the functional and non-functional needs of the company.
-
Identify, design, and implement the development process which includes process automation, optimization of data provision, redesign of scalable infrastructure, and so on.
-
Develop optimal infrastructure for the needs of extracting, transforming, and loading data from various sources such as SQL and AWS big data technologies.
-
Build a data pipeline analysis tool that can provide insights into the customer acquisition process, operational efficiency, and other metrics used to measure the company's business performance.
-
Work closely with company executives, product teams, and design teams to address data-related issues and support data infrastructure needs.
-
Perform data separation and ensure data security in various data centers.
-
Contribute to product development and optimization by data analysts and data scientists by creating data analysis tools.
-
Collaborate with data and analytics experts to create more functional data systems.
Minimum Qualifications
-
Bachelor's degree in informatics engineering, statistics, applied mathematics, and other related quantitative fields.
-
Have good project management and organizational skills.
-
Able to work with various teams within the company in a dynamic work environment.
-
Experienced in building and optimizing big data, data architecture and data sets.
-
Have strong analytical skills to work on unstructured data sets.
-
Have a strong understanding of using SQL and processing relational databases, query authoring, and various other database variations.
-
Experienced in carrying out root cause analysis on internal and external data to answer the company's business needs and identify potential developments.
-
Build processes to support data transformation, data structuring, metadata, and workload management.
-
Experienced in manipulating, processing and extracting data from a number of unconnected data sets.
-
Have an understanding of message queuing, stream processing, and scalable big data storage models.
-
Experienced using various software / tools such as:
-
Big data tools: Hadoop, Spark, Kafka, and so on.
-
SQL and NoSQL relational databases: PostGre and Cassandra.
-
Data pipeline and workflow management tools: Azkaban, Luigi, Airflow, and so on.
-
AWS cloud services: EC2, EMR, RDS, and Redshift.
-
Stream processing systems: Strom, Spark-Streaming, and so on.
-
Object-oriented programming languages: Python, Java, C++, Scala, and so on.
Find a Job as a Data Engineer
Oops!
We don't have any jobs matching that search right now.