Data Engineering Specialist

About the role

• The Data Engineer supports the capture, management, storage and utilisation of structured and unstructured data from internal and external sources, translating business needs into robust data solutions that underpin BT Group’s data strategies and strategic decision making.
• Delivers the coding, testing and deployment of data processes for the ingestion, transformation and curation of data, ensuring compliance with data security and privacy standards.
• Contributes to the development and refinement of data architecture and design patterns, supporting the processing and storage of high volume, structured and unstructured data sets.
• Contributes to the delivery of key engineering projects, helping shape technical approaches and driving new solutions to complex or open ended problems.
• Delivers business relevant and actionable data outputs, enabling the creation of reliable data assets and strengthening data engineering capabilities.
• Applies engineering best practices and Agile delivery approaches to ensure consistent, high quality and iterative delivery of data engineering solutions.
• Executes initiatives to develop and enhance data and analytics infrastructure, supporting scalable and reusable data engineering capabilities.
• Supports the identification and implementation of improvements to data engineering processes, using data analysis to assess issues, optimise workflows and drive efficiency.
• Conducts quality assurance and testing of data engineering routines, models and frameworks, ensuring accuracy, robustness and suitability for business use.

What you’ll be doing

• Data Engineer Lead within their Squad, including coaching and mentoring other engineers within the Alliance ensuring all adhere to Engineering standards and best practice, facilitating workshops and running training sessions where needed
• Ensure both self and others take responsibility for achieving brilliant customer and business outcomes through high-quality execution, collaboration, and a focus on continuous improvement.
• Design and build data products and data pipelines that are cost effective, performant and resilient.
• Build both batch and NRT data pipelines.
• Ensure all data products delivered and maintained to a high quality, accurate, secure, and always adhere to data protection policies.
• Support Data Product documentation, including clear and comprehensive user guides, and runbooks to aid control and knowledge transfer.
• Operational maintenance of all data products

Essential Skills / Experience

• Experience with CI/CD setups and infrastructure as code frameworks (ideally Terraform or OpenTofu)
• Experience with open-source big data frameworks (ideally Spark, Apache beam, or Apache Flink)
• Experience with NRT data pipelines is highly desirable
• High proficiency with SQL, Python and one of Scala/Java
• Strong experience of cloud environments (ideally GCP) in the following areas.
• SQL and noSQL databases
• Orchestration tools
• Big data and ETL services
• Serverless compute services
• Experience with K8 and cloud networking is desirable but not required
• Strong data management and data architecture experience
• Strong understanding of data security and compliance processes
• Strong communication skills and ability to articulate complex technical concepts to both technical and non-technical stakeholders at all levels, written and verbal
• Excellent stakeholder management skills, planning skills and the ability to prioritise demand in line with business and commercial requirements.
• Strong experience leading / coaching and mentoring
• Experience working in Agile
• Experience working on AI Applications Amazon Q, Kiro or Gemini.