Big Data Engineer

Cloudera Big Data Engineer

Summary

The position that we are hiring will form part of a core Product Engineering team that is building a big next generation data analytics platform for the healthcare space. Due to the strategic nature and long-term vision of the product, candidates are expected to demonstrate steadfast commitment and dedication over and above proving themselves to excel in the challenging technical skillset.

Job Description

• Fluent in big data engineering development using the Hadoop/Spark ecosystem
• Hands-on project experience with Cloudera Data Platform and cloud-based data lake architectures (Azure, AWS, or GCP).
• Data ingestion and integration into the Data Lake using the Hadoop ecosystem tools such as Sqoop, PySpark, Impala, Hive, Oozie, Airflow etc.
• Proficiency in Python for data engineering tasks and pipeline automation
• Experience designing and implementing scalable data pipelines for structured and unstructured data.
• Strong knowledge of Hive data structures, metadata management, and data lake/warehouse loading strategies.
• Developing the Data ingestion and integration flows in Hive, Spark and Impala
• Building the data pipeline to migrate and load the data into the Hadoop distributed file system either on-prem or in the cloud
• Experience in building real-time data ingestion pipelines using Apache Kafka, Apache NiFi and integration with healthcare data sources (HL7, FHIR, etc.).
• Hands-on experience with Kestra for orchestrating, automating, and monitoring complex data workflows.
• Developing applications with Apache Kudu and experience in Kudu integration with Spark
• Understanding the requirements from the Functional Team
• Developing the code that aligns to the technical design and coding standards
• Ownership of the code and deployment into test, UAT and production
• Conduct Peer-Code Reviews for early detection of defects and code quality
• Troubleshooting and follow escalation procedures to resolve issues

Qualifications and Experience:

• Overall, 4-6 years of Data Engineering experience in Big Data
• 2-3 years of hands-on experience in Hadoop ecosystem tools such as Sqoop, Hive, Hbase
• Hands-on development experience in Spark framework– PySpark/Spark-Scala/Java
• Bachelor of Engineering or Bachelor of Technology
• Good English communication skills
• Self-driven and self-initiated
• Team player
• Candidates with experience in healthcare big data projects will be preferred

IQVIA is a leading global provider of clinical research services, commercial insights and healthcare intelligence to the life sciences and healthcare industries. We create intelligent connections to accelerate the development and commercialization of innovative medical treatments to help improve patient outcomes and population health worldwide. Learn more at https://jobs.iqvia.com

Experience Level

Senior Level