IN_Manager_ Data Engineer GCP_Data and Analytics_Advisory_Bangalore

Job Description & Summary:  

We are seeking an experienced Data Engineer – GCP to design, build, and operate scalable, reliable, and cost-efficient data pipelines on Google Cloud Platform. The role involves hands-on development using BigQuery, Dataflow, Dataproc, Cloud Composer, Pub/Sub, Cloud Storage, Python, SQL, and related GCP-native services, along with close collaboration with analytics, AI/ML, architecture, and business teams. 

Responsibilities: 

• Design, build, and maintain end-to-end batch and streaming data pipelines on Google Cloud Platform 

• Develop scalable data processing solutions using BigQuery, Dataflow, Dataproc, Pub/Sub, and Cloud Storage 

• Build and optimize Cloud Composer / Airflow DAGs for orchestration, scheduling, retries, and dependency management 

• Ingest, transform, and curate large-scale structured, semi-structured, and streaming datasets 

• Develop reusable ETL/ELT frameworks using Python, SQL, Apache Beam, and Spark / PySpark where applicable 

• Implement BigQuery performance and cost optimization through partitioning, clustering, query tuning, and data model improvements 

• Ensure data quality, reliability, lineage, observability, governance, and operational readiness 

• Collaborate with data scientists, analysts, architects, and business teams to support analytics, reporting, and AI/ML workloads 

• Implement secure access patterns using IAM, service accounts, encryption, and enterprise compliance standards 

• Provide production support, troubleshoot pipeline failures, and resolve performance bottlenecks 

• Document technical designs, data flows, operational procedures, and runbooks 

• Mentor junior engineers and contribute to engineering best practices for GCP data platforms 

Mandatory skill sets: 

4+ years of experience as a Data Engineer with strong hands-on Google Cloud Platform expertise 

• Strong experience with BigQuery, GoogleSQL / SQL, and data warehousing concepts 

• Hands-on experience with Dataflow and Apache Beam for batch and streaming pipelines 

• Experience with Dataproc, Spark, or PySpark for distributed data processing 

• Experience with Cloud Composer / Airflow for orchestration and workflow management 

• Experience with Pub/Sub and Cloud Storage for ingestion and data lake patterns 

• Strong Python skills for data engineering, automation, and reusable pipeline development 

• Understanding of IAM, service accounts, networking basics, and secure data access on GCP 

• Experience with ETL/ELT patterns, data modeling, data quality, and production support 

• Experience working in Agile teams 

Preferred skill sets: 

Experience with Dataplex, Data Catalog, and data governance capabilities 

• Exposure to Dataform, dbt, or similar analytics engineering frameworks 

• CI/CD experience for data pipelines using Cloud Build, GitHub Actions, Azure DevOps, or similar tools 

• Experience with BigQuery ML 

• Knowledge of Cloud Functions, Cloud Run, Workflows, or Cloud Scheduler 

• Google Cloud Professional Data Engineer certification 

Years of experience required: 

5 to 10 years 

Education qualification: 

Bachelor’s or Master’s degree in Computer Science, Engineering, or related field