Big Data Consultant

Job Title

Big Data Consultant

Job Scope

The Big Data Consultant is accountable for providing thought leadership by maintaining knowledge of data quality, master data management tools and technology trends, standards, and methodologies.

Main Duties and Responsibilities

• Design, develop, and manage large-scale data pipelines using Cloudera Data Platform (CDP) and related big data technologies.
• Implement and manage data storage solutions (HDFS, Ozone, Hive, HBase), streaming and ingestion frameworks (NiFi, Kafka), and processing frameworks (Spark, Impala, Flink) on Cloudera clusters.
• Develop ETL processes to ingest, transform, and store data for analytics and reporting purposes.
• Optimize and tune data pipelines for performance and scalability.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and ensure data availability.
• Monitor, troubleshoot, and resolve performance issues in big data environments.
• Ensure data security and compliance with relevant regulations and policies.
• Stay up-to-date with the latest big data technologies and industry best practices.
• Participate in the design and deployment of new features and updates to the Cloudera platform.
• Provide support for the deployment and operationalization of AI/ML models on big data infrastructure.

Position Requirements

• 3+ years of experience in big data engineering with a focus on Cloudera platforms (CDP).
• Proficiency in big data technologies like HDFS, Hive, HBase, Spark, Kafka, Flink, NiFi and Impala.
• Strong experience with data processing, ETL, and workflow orchestration tools (NiFi, Oozie, Airflow).
• Expertise in programming languages such as Python, Java, or Scala.
• Experience with data pipeline optimization, tuning, and troubleshooting.
• Hands-on experience with data governance, data security, and regulatory compliance.
• Familiarity with CI/CD pipelines and DevOps tools for big data environments.
• Strong problem-solving and analytical skills.
• Cloudera Certified Big Data Engineer or other relevant certifications.
• Experience working with cloud platforms (AWS, GCP, or Azure) in hybrid or multi-cloud environments.
• Knowledge of containerization and orchestration technologies (Docker, Kubernetes).
• Experience with AI/ML integration on big data platforms.

Education

Bachelor’s degree in Computer Science, Information Technology, or related field.

Experience

At least 7 years of relevant experience

#LI-AA3