IN_Senior Associate_AWS Data Engineer_D&A_Advisory_Bangalore

 Role Overview 

We are looking for an experienced AWS Data Engineer with 4–8 years of hands-on experience in designing, developing, and maintaining scalable data pipelines and cloud-based data platforms. 

The ideal candidate will have strong expertise in AWS, Snowflake, Apache Airflow, Python, PySpark, and SQL, with a solid understanding of data warehousing, ETL/ELT, data modeling, and performance optimization. 

The candidate will work closely with data architects, analysts, application teams, and business stakeholders to build reliable and scalable data solutions.  

Responsibilities 

 Design, develop, and maintain scalable data pipelines and ETL/ELT workflows using AWS services. 

Build and orchestrate data pipelines using Apache Airflow, including DAG development, scheduling, monitoring, retries, dependencies, and error handling. 

Develop data processing and transformation solutions using Python and PySpark. 

Design and implement data warehouse solutions using Snowflake. 

Develop complex SQL queries, stored procedures, views, CTEs, and data transformations. 

Work with AWS services such as S3, Glue, Lambda, EMR, Athena, Redshift, and IAM. 

Build batch and, where required, near-real-time data ingestion pipelines. 

Implement data ingestion from APIs, databases, files, and other source systems into AWS/Snowflake. 

Perform Snowflake performance and cost optimization, including warehouse sizing, query optimization, clustering, partitioning, and efficient data loading. 

Implement Snowflake features such as Snowpipe, Streams, Tasks, stages, file formats, and secure data sharing. 

Develop scalable Spark/PySpark jobs and optimize transformations, joins, partitioning, caching, and resource utilization. 

Implement data quality checks, validation, reconciliation, and monitoring mechanisms. 

Troubleshoot pipeline failures, data issues, performance bottlenecks, and production incidents. 

Follow best practices for data security, governance, access control, and PII-sensitive data handling. 

Use Git and CI/CD practices for source control, automated testing, and deployment of data pipelines. 

Collaborate with cross-functional teams in an Agile/Scrum environment. 

Create technical documentation for data pipelines, workflows, data models, and operational procedures.  

Mandatory Skill sets:

 4–8 years of experience in Data Engineering. 

Strong hands-on experience with AWS Data Engineering. 

Strong experience with Snowflake. 

Hands-on experience with Apache Airflow and DAG development. 

Strong programming experience in Python. 

Strong hands-on experience with PySpark / Apache Spark. 

Advanced SQL skills. 

Strong understanding of ETL/ELT and data pipeline development. 

Experience working with AWS S3 and AWS Glue. 

Good understanding of data warehousing and dimensional data modeling. 

Experience with data pipeline monitoring, debugging, and performance optimization. 

Good understanding of Git and CI/CD.

Preferred Skill sets:

 AWS Lambda, EMR, Athena, Redshift, Kinesis, Step Functions, IAM. 

Snowflake Snowpipe, Streams, Tasks, Dynamic Tables, Time Travel and performance tuning. 

Experience with dbt. 

Experience with Kafka or other streaming technologies. 

Experience with Terraform / Infrastructure as Code. 

Experience with data quality tools such as Great Expectations. 

Knowledge of Lakehouse / Medallion Architecture. 

Experience with Databricks. 

Snowflake certification such as SnowPro Core. 

Exposure to Docker/Kubernetes is a plus.