Data Engineer

About the role

Data is a vital part of our business and with your help, we can improve the services that we provide to our customers through engineering excellence of our data products. You’ll be working on exciting data products which support TV, Broadband, Digital Voice and market leading technologies like Unbreakable Wi-Fi.  We deliver these critical services using the latest cloud technologies on AWS, GCP and Cloudera.
When you join us as a lead data engineer, you’ll have the opportunity to be part of our Data & AI Enablement Team where you will help to design, build, and support high quality engineering products from our extensive data sources and multi-cloud environment.
As a team we believe in the agile principles of openness, transparency and continuous improvement, which underpin strong relationships with our business stakeholders.
There will be lots of opportunities to explore new technologies, develop new skills, innovate and grow as an engineer. 

•  Design, develop, and maintain end-to-end data pipelines using Ab Initio (GDE, PDL, Conduct>IT, Express>IT).
•  Build scalable batch and streaming workloads using Apache Spark (PySpark/Scala).
•  Develop and orchestrate data ingestion flows using Apache NiFi for real-time and near-real-time use cases.
•  Optimize ETL/ELT jobs for scalability, throughput, and resource efficiency.
•  Perform bottleneck analysis and tuning for Spark jobs, NiFi processors, and Ab Initio graphs.
•  Demos and knowledge transfers to stakeholders around product developments.
•  Spread your experience and knowledge to junior engineers.
•  Improve automated testing and deployments using CI/CD pipelines.
•  Expectation to line manage small team of junior data engineers.
•  Continual learning through internal and external training.

What you’ll be doing

•  Use of Hadoop, GCP or AWS to build Big Data solutions using some of the following technologies: Scala, Java, Python, Spark,
• Terraform, Kafka, Kinesis, Kinesis Analytics, BigQuery, Dataflow, BigTable, SQL.
•  Knowledge of Infrastructure as Code Toolsets. 
•  Strong expertise in Ab Initio: 
  GDE, PDL, Conduct>IT, Express>IT, EME
•  Parallel processing frameworks and component optimization
•  Strong expertise in Apache Spark: 
  PySpark or Scala
  Spark SQL, Spark Streaming, performance tuning
•  Hands-on experience with Apache NiFi: 
  Flow design, processor tuning, error handling, security configs
•  Hands-on experience with Continuous Integration and Deployment Strategies
  Prior experience using Confluence, JIRA, or other Atlassian tools.
•  Software version control using git or SVN.
•  Presentation of technical issues and successes to other members of the team and to Product Owners.
•  Keen interest in learning new technologies and putting these into practice.
•  Has experience of people management or desire to manage individuals on the tea