Data Engineer - Artificial Intelligence

The Data Engineer (India Offshore) is responsible for building, maintaining, and optimising data  pipelines and transformation processes that deliver trusted, analytics-ready datasets for Kaplan  Australia. This role will be the primary contributor driving data transformation into the Lakehouse  (Databricks), ensuring data is modelled, curated, and made available for reporting, analytics, and AI/ML  use cases. 

Reporting to the Senior Solutions Architect, the role contributes hands-on engineering expertise across  ingestion, transformation, orchestration, and monitoring, and supports the enablement of AI/ML by  providing well-governed features and training datasets.

Key Responsibilities 

Data Engineering Delivery (Pipelines, Transformations, and Data Modelling) • Design, build, and operate reliable ELT/ETL pipelines to ingest, transform, and curate data into  analytics-ready datasets. 

• Implement and maintain data models (dimensional and/or lakehouse curated layers) with clear  semantic definitions to support reporting, analytics, and AI use cases. 

• Build data quality controls (validation, reconciliation, and automated checks) and implement  observability (monitoring, alerting, and SLAs) for pipelines and jobs. 

• Optimise performance and cost through efficient compute usage, incremental processing,  partitioning, and tuning of Spark/SQL workloads. 

Lakehouse / Databricks Transformation Ownership 

• Be the primary engineer responsible for implementing and evolving the Lakehouse  transformation layer in Databricks. 

• Build and maintain Databricks workflows, notebooks, and jobs using Spark/SQL/Python, applying reusable patterns and standards. 

• Contribute to Lakehouse design decisions (data layout, medallion architecture, Delta patterns,  and access strategies) in collaboration with the Head of AI and platform stakeholders. 3. AI / ML Engineering Enablement 

• Work within the AI team to support delivery of AI/ML initiatives through hands-on engineering  (data preparation, pipeline build-out, and integration into production patterns). • Collaborate on GenAI projects, including prompt engineering iteration support (test cases,  evaluation datasets, and telemetry/metrics to assess prompt performance).

Minimum Qualifications

• Demonstrated 5+ years’ experience in data engineering, building production-grade data  pipelines and transformation layers. 

• Strong hands-on experience with Databricks and Lakehouse patterns (e.g., medallion  architecture) including Spark and SQL.  Proficiency in Python and SQL for data engineering (transformations, testing, and automation).

• Experience working with cloud‑based data platforms (AWS) and integrated data storage,  orchestration, and analytics services. 

• Strong understanding of data modelling (dimensional modelling, curated layers, semantic  definitions) and data quality management. 

• Working knowledge of data governance, privacy, and information security practices (access  controls, PII handling, lineage, retention). 

• Strong problem-solving skills, attention to detail, and ability to work autonomously with  distributed stakeholders. 

• Experience with Jira, Confluence, and Microsoft 365 tools; comfortable working across Agile  delivery practices. 

Highly Desirable 

• Experience enabling AI/ML use cases (feature engineering datasets, training/serving data  considerations, reproducibility). 

• Experience supporting GenAI / LLM solutions, including RAG-style data preparation (document  ingestion, chunking, metadata enrichment) and evaluation dataset creation. 

• Experience collaborating on prompt engineering and prompt evaluation cycles, including  collecting ground truth, defining test cases, and analysing prompt performance telemetry. • Understanding of vector search / embeddings concepts and data requirements for semantic  retrieval. 

• Experience delivering data solutions within higher education or regulated environments.

Compliance & Governance Obligations

• Demonstrate company values and contribute to a kind, safe, supportive and collaborative  workplace.  

• Adherence to all Kaplan policies and procedures. 

• Compliance with Workplace Health & Safety legislation and requirements. 

• Completion of all mandatory training as required by Kaplan. 

• Compliance with information security, privacy, data governance and risk management  frameworks.

Kaplan is a drug-free workplace and complies with applicable laws.