Data Engineer

Job Title

Data Engineer

Job Scope

The Data Engineer is accountable for supporting projects and leading data related initiatives that meet customer requirements while participating in the business requirement gathering, development, usage and integration of innovative systems. S/He resolves technical issues through effective data utilization and provides training to users.https://th.joblum.com/job/data-management-analytics-consultant-manager/170927

Main Duties and Responsibilities

• Design and build data pipelines to collect, store and process large and complex data sets from various sources including data ingestion, data cleansing, data validation and data transformation
• Work on Big-Data ecosystem including Hadoop, HDFS, Spark, Spark SQL, Hive, Impala, Talend and CFM
• Implement data storage and processing technologies such as data lakes, data warehouses and NoSQL databases to support data analytics and machine learning
• Optimize data pipeline performance, troubleshoot and debug related issues while monitoring data pipeline health
• Collaborate with Data Scientists and other stakeholders to understand data requirements and design data pipelines accordingly
• Create and maintain documentation for data pipelines, data models and data dictionaries
• Ensure data quality by implementing data validation and data cleansing processes
• Implement security measures to protect data and ensure compliance with regulations such as GDPR, HIPAA and PCI-DSS
• Research and evaluate new technologies and tools to improve data pipeline performance and scalability on a regular basis
• Apply performance optimization techniques and technical guidance to other Application Developers
• Provide advisory services for designing and implementing large, scalable distributed systems
• Use programming languages and technologies like Python, SQL, Java, Scala, Informatica, Hadoop, Spark and Hive
• Coordinate with multiple stakeholders on a regular basis.

Position Requirements

Education

Bachelor’s degree in Computer Science, Computer Engineering or any other related field

Experience

At least 1 to 3 years of relevant experience

#LI-AA4