Senior Data Engineer

Senior Data Engineer
  • £80,000 per Annum
    ANNUM
  • Permanent
  • England, UK GBR SE12 8RR
Job Ref: 50214629
Sector: Other
Date Added: 11 November 2021

Senior Data Engineer

Role Purpose:

Our client are looking for a Senior Data Engineer to join them on a permanent basis.

This role will be 40% on site 60% remote.

Working closely with the Senior Engineer the Senior Data Engineer is responsible for continuously

developing new and improved data engineering capabilities. Taking a lead in developing and

maintaining data pipelines between systems of record, the data integration platform and the data

lake using best practises and within a dataops methodology.

The Senior Data Engineer will ensure a consistency in the approach to maintaining and building all

data flows across the organisations data management platform, and that the data is well curated for

varying levels of analytics use cases.

Outputs & Accountabilities:

  • Work with the Lead and Senior Engineers to leverage data for use in a cross section of

analytics use cases, from descriptive to prescriptive analytics.

  • Sole Senior Data Engineer for the engineering team and responsible for operating inside the

agreed dataops process.

  • Own and extend the data pipelines through the collection, storage, processing, and

transformation of large data-sets.

  • Assemble large, complex data sets that meet functional / non-functional business

requirements.

  • Identify, design, and implement internal process improvements: automating manual

processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.

  • Ensure that the infrastructure required for optimal integration, extraction, transformation,

and loading of data from a wide variety of data sources is built and maintained.

  • Collaborate and support the Senior Engineer to define the business's data requirements,

which will ensure that the collected data is of a high quality and optimal for use across the

business.

  • Design, develop and maintain solutions to keep data separated and secure across on

premise and cloud based infrastructure and software.

  • Collaborate with the senior analytics and data team colleagues through the Senior Engineer

to analyse and understand data sources, participate in design, and provide insights and

guidance on database technology and data modelling best practices.

  • Design, and develop algorithms for real-time data processing within the business and create

the frameworks that enable quick and efficient data acquisition.

  • Keep track of industry best practices and trends and apply knowledge in order to take

advantage of process and system improvement opportunities.

  • Ensure that Data Governance procedure are well defined and applied through the data

engineering team.

  • Ensure that all development occurs within a CICD approach.

Skills / Background:

  • Strong communication skills, both written and verbal.
  • Strong capability to convey complex messages in a clear, simplified, and understandable

manner.

  • Advanced working SQL knowledge and experience working with relational databases, query

authoring (SQL) as well as working familiarity with a variety of databases.

  • Experience building and optimizing data pipelines, architectures and data sets.
  • Experience performing root cause analysis on internal and external data and processes to

answer specific business questions and identify opportunities for improvement.

  • Strong analytic skills related to working with unstructured datasets.
  • Build processes supporting data transformation, data structures, metadata, dependency and

workload management.

  • A history of manipulating, processing and extracting value from large disconnected datasets.
  • Working knowledge of message queuing, stream processing, and highly scalable data stores.
  • Project management and organizational skills.
  • Experience supporting and working with cross-functional teams in a dynamic environment.
  • Experience using the following software/tools:
    • Experience with big data tools: Hadoop, Spark, Kafka, etc.
    • Experience with relational SQL and NoSQL databases, including Postgres and

Cassandra.

  • Experience with data pipeline and workflow management tools: Azkaban, Luigi,

Airflow, etc.

  • Experience with AWS cloud services: EC2, EMR, RDS, Redshift
  • Experience with stream-processing systems: Kafka, Storm, Spark-Streaming, etc.
  • Experience with object-oriented/object function scripting languages: Clojure, Java,

Python, etc

We are an equal opportunities employer and welcome applications from all suitably qualified persons regardless of their race, sex, disability, religion/belief, sexual orientation, gender reassignment, marriage and civil partnerships, pregnancy or maternity or age

Lorien
11/12/2021 17:41:08
GBP 80000 80000 Annum
Contact Consultant:
Joseph Bilbe-Chrisfield

Submitting application, please wait..