Senior Data Engineer (P50) — Reliability Process Group (RPG)

Senior Data Engineer (P50) — Reliability Process Group (RPG)

Overview

Data and observability are central to engineering reliability at Atlassian. The Reliability Process Group (RPG) builds and manages the foundational data infrastructure, ingestion pipelines, and analytics surfaces that track incident posture, Post-Incident Reviews (PIRs), service level objectives (SLOs), and reliability metrics across the company (e.g., Reliability Watch, MSR reporting, AutoHOT).

We are looking for a Senior Data Engineer (P50) to take end-to-end ownership of our data pipelines and analytical data products. In this role, you will design robust batch and incremental ETL/ELT pipelines, implement automated Data Quality (DQ) frameworks, optimize query performance at scale, and deliver trusted datasets powering executive and team-level reliability reporting.

Responsibilities

• End-to-End Pipeline & System Ownership: Design, build, optimize, and maintain scalable data pipelines (batch & incremental) ingesting data from Jira/JSM, OpsGenie, internal incident services, and telemetry stores.

• Data Modeling & Architecture: Design and evolve reliable data models and marts (e.g., Incident Context Enriched, PIR SLAs, Hot Reviews) using modern table formats, dbt, and Databricks.

• Data Quality & Pipeline Hygiene: Establish automated Data Quality (DQ) checks, anomaly detection, threshold monitoring, and P1/P2 alerting to prevent data drift and pipeline failures.

• Analytics & Visualization Enablement: Partner with cross-functional teams and leadership to deliver performant, aggregation-ready datasets for dashboards (Tableau, internal tools) and executive reliability reporting.

• Performance & Cost Optimization: Profile and optimize SQL/Spark/dbt queries, leverage incremental loading, and improve partitioning strategies across Databricks and Socrates datasets.

• Engineering Standards & Mentorship: Champion best practices in code reviews, CI/CD pipelines (Bitbucket Pipelines), testing, runbooks, and mentor peers across data engineering practices.

Qualifications

Minimum Requirements:

• 5+ years of experience in a Data Engineering role with proven success building and operating production data platforms at scale.

• Strong programming & scripting skills in Python, with a solid grasp of software engineering fundamentals (OOP, testing, modular code, CI/CD).

• Expert SQL & data modeling skills, including dimensional modeling, query tuning, partition management, and schema design.

• Hands-on experience with modern Big Data ecosystems & orchestration: Databricks / Apache Spark, dbt, and workflow schedulers (e.g., Airflow / Bitbucket Pipelines).

• Cloud Infrastructure experience: Familiarity with AWS or GCP data services and storage primitives.

• Data Quality & Observability mindset: Experience writing automated data validation checks, monitoring pipeline health, and configuring alert thresholds.

• Strong communication & stakeholder skills: Ability to navigate ambiguity, translate business/reliability requirements into technical specifications, and document system architecture.

Preferred / Nice-to-Have Qualifications:

• Experience with internal Atlassian data infrastructure (Socrates pipelines, Compass, Teamwork Graph, Jira/OpsGenie APIs).

• Experience with BI and visualization tools (e.g., Tableau workbook optimization, custom SQL extracts, incremental refreshes).

• Experience with table formats such as Delta Lake or Apache Iceberg.

• Background in Site Reliability Engineering (SRE), incident management data, or telemetry/observability domains.

Experience Level

Senior Level