Junior Data Engineer - GCP

Data Engineer

Responsibilities 
•    Design and build GCP data driven solutions for enterprise data warehouse and data lakes     
•    Design and implement scalable data architectures on GCP, including data lakes, warehouses, and real-time processing systems.
•    Utilize ML Services like Vertex AI.
•    Develop and optimize ETL/ELT pipelines using tools like Python, SQL, and streaming technologies (e.g., Kafka, Apache Airflow).
•    Architect Historical and Incremental Loads and Refine Architecture on an ongoing basis
•    Manage and optimize data storage, partitioning, and clustering strategies for high performance and reliability, utilizing services such as BigQuery, Spark, Pub/Sub, and Object Storage.
•    Collaborate with data scientists, Data engineers, and other stakeholders to understand data needs and deliver solutions aligned with business objectives, security, and data governance.
•    Automate infrastructure and deployments using Infrastructure as Code (IaC) using tools like Terraform and CI/CD practices (e.g., Tekton) to ensure reliability and scalability.
•    Operationalize machine learning models by building data infrastructure and managing structured and unstructured data, supporting AI/ML/LLM workflows, including data labeling, classification, and document parsing.
•    Monitor and troubleshoot data pipelines and systems to identify and resolve issues related to performance, reliability, and cost-effectiveness.
•    Document data processes, pipeline designs, and architecture, contributing to knowledge transfer and system maintenance.

Qualifications and Skills 

• Must have - 
• Professional GCP - Data Engineer Certification. 
• 2 + years coding skills in Java/Python and Terraform.
• 2+ years’ experience 
• Experience in working with Agile and Lean methodologies.
• GCP Expertise: Strong proficiency in GCP services, including BigQuery, Dataflow, Dataproc, Data Fusion, Air Flow, Pub/Sub, Cloud Storage, Vertex AI, Cloud Functions, and Cloud Composer, GCP based Big Data deployments (Batch/Real-Time) leveraging Big Query, Big Table
• Programming & Scripting: Expert-level skills in Python and SQL are essential. Familiarity with languages like Scala or Java can also be beneficial, especially for working with tools like Apache Spark.
• Data Engineering Fundamentals: Solid understanding of data modeling, data warehousing concepts, ETL/ELT processes, and big data architecture, Designing pipelines and architectures for data processing.
• Big Data Technologies: Experience with technologies like Apache Spark, Apache Beam, and Kafka is often required.
• DevOps & MLOps: Knowledge of DevOps methodologies, CI/CD pipelines, and MLOps practices, including integrating data pipelines with ML workflows.
• Security & Compliance: Expertise in implementing Identity and Access Management (IAM) policies, ensuring data encryption, and adhering to data privacy regulations.
• Analytical & Problem-Solving Skills: Demonstrated ability to analyze complex datasets, identify trends, debug issues, and optimize systems for performance and cost efficiency.
• Communication & Collaboration: Excellent communication and teamwork skills, with the ability to collaborate effectively with technical and non-technical stakeholders in agile environments. 
• Experienced in Visualization Tool – Qlik, Looker Studio, Power BI
• Knowledge of VBA, will be a plus

Experience Level

Mid Level