Data Engineer Consultant

Consultant - Data Engineer

Job Title: Consultant - Data Engineer

Career Level: C3

Introduction to role:

Are you ready to run high-impact data platforms at scale and see your decisions translate into faster, smarter outcomes for patients and the business? Join a team where the reliability, security, and efficiency of our data stack directly accelerates how insights flow from idea to action.

Based in Chennai, you will administer and evolve Databricks and Redshift Serverless on AWS, keeping production stable while unlocking performance and cost efficiency for analytics and machine learning. You will coordinate ETL/ELT end to end, maintain strong governance, and enable colleagues to move from proof of concept to enterprise-grade solutions with confidence.

Do you thrive where platform administration meets SRE and data engineering? This role places you at the nexus of operations and delivery, partnering closely with engineers, analysts, and vendors to set standards, solve complex issues, and improve benchmarks for how we use data to drive impact.

Accountabilities:

Platform Operations: Provision and lead Databricks workspaces, cluster policies, and Workflows; administer Redshift Serverless workgroups and name spaces; lead all aspects of Linux runtimes. Your ownership ensures availability, reliability, and a consistent operating model from day one.

User and Access Governance: Implement IAM/RBAC and Unity Catalog permissions, handle Redshift roles and data-level controls, and implement least-privilege. You will protect critical data while enabling teams to move quickly.

Performance and Cost Optimization: Monitor and tune PySpark/Spark SQL and Redshift Serverless; apply right-sizing and cost controls to deliver faster pipelines and predictable spend that scales with demand.

Security and Compliance: Implement encryption, key and secret management, patching, and hardening; maintain audit and lineage logging. Your efforts underpin trust in our data and alignment to regulatory expectations.

Automation and SRE: Use Bash/Python and CI/CD to codify infrastructure and deployments; set up monitoring, alerting, SLIs/SLOs, and runbooks. You will reduce toil, increase repeatability, and improve time to recovery.

Data and ETL Operations: Orchestrate and support ingestion and transformation pipelines across S3, Databricks, and Redshift; implement data quality checks and resilient recovery. Your work keeps insights flowing to the people who need them.

Incident and Change Leadership: Lead incident response and root-cause analysis; plan and implement changes, releases, and maintenance windows; communicate clearly with customers to minimize risk and downtime.

Release Management: Own GitHub strategy, tagging, and protection rules; run pull requests and code reviews; maintain CI/CD workflows for Databricks jobs, notebooks, and IaC with robust versioning and rollback plans.

Communication and Collaboration: Produce clear documentation and handovers; partner with engineers, DBAs, and vendors. You will align teams with varied strengths around shared standards and outcomes.

Essential Skills/Experience:

Databricks: PySpark and Spark SQL, Delta Lake guidelines, Workflows/Jobs, cluster policies, Unity Catalog governance, and cost controls.

Redshift Serverless: Workgroup administration, RPU management, query monitoring rules, performance tuning, Spectrum/external tables, COPY/UNLOAD to S3, snapshots/restore, security and permissions.

Unix/Linux: System administration (users/groups, SSH, systems, networking, filesystems), Bash scripting, log management, and OS hardening.

AWS: S3, IAM, CloudWatch/Cloud Trail, basic VPC concepts, budgeting and tagging.

DevOps/IaC: Git and GitHub (branching, PRs, Actions), CI/CD pipelines and deployments.

Support: Strong troubleshooting experience across Spark/SQL/OS; on-call readiness and clear incident communications.

Educational Background: B.E/B.Tech/MCA/MSc (or equivalent experience)

Overall Experience: 5 - 6 years

Shifts and On-Call: Rotational Shift: 06:00-15:00 (First) or 09:00–18:00 (General) or 14:00–23:00 (Second); Weekend: On-call support once per month (Saturday and Sunday)

Desirable Skills/Experience:

Cloud Infrastructure: Amazon EC2, S3, and exposure to PostgreSQL.

Tooling and Process: Jira and ServiceNow for ITSM.

Orchestration and Workflow: Airflow (or similar).

Fast learner, proactive problem solver, able to grasp new technologies and concepts.

Preferable Industry: IT, Manufacturing, Pharma, or similar enterprise environments.

When we put unexpected teams in the same room, we ignite bold thinking with the power to inspire life-changing medicines. In-person working gives us the platform we need to connect, work at pace and challenge perceptions. That's why we work, on average, a minimum of three days per week from the office. But that doesn't mean we're not flexible. We balance the expectation of being in the office while respecting individual flexibility. Join us in our unique and ambitious world.

Why AstraZeneca:

Join a global, mission-led company where data and digital capabilities power every function—from freeing scientists to do more discovery to reimagining enterprise processes. You will work with curious, kind, and ambitious colleagues who put unexpected teams in the same room to spark bold thinking, using predictive analytics, smart automation, and modern data platforms to solve real problems that matter to patients. Here, you will be trusted to act, supported to learn, and encouraged to challenge how things have always been done—so your governance expertise translates directly into safer, faster impact at scale.

If you are ready to turn thorough governance into an impetus for responsible innovation, take the lead and help us set the standard for credible data and AI today!

Date Posted

11-Jun-2026

Closing Date

13-Jun-2026

AstraZeneca embraces diversity and equality of opportunity.  We are committed to building an inclusive and diverse team representing all backgrounds, with as wide a range of perspectives as possible, and harnessing industry-leading skills.  We believe that the more inclusive we are, the better our work will be.  We welcome and consider applications to join our team from all qualified candidates, regardless of their characteristics.  We comply with all applicable laws and regulations on non-discrimination in employment (and recruitment), as well as work authorization and employment eligibility verification requirements.