Senior Data Engineer
Our client are looking for a Senior Data Engineer to join them on a permanent basis.
This role will be 40% on site 60% remote.
Working closely with the Senior Engineer the Senior Data Engineer is responsible for continuously
developing new and improved data engineering capabilities. Taking a lead in developing and
maintaining data pipelines between systems of record, the data integration platform and the data
lake using best practises and within a dataops methodology.
The Senior Data Engineer will ensure a consistency in the approach to maintaining and building all
data flows across the organisations data management platform, and that the data is well curated for
varying levels of analytics use cases.
Outputs & Accountabilities:
- Work with the Lead and Senior Engineers to leverage data for use in a cross section of
analytics use cases, from descriptive to prescriptive analytics.
- Sole Senior Data Engineer for the engineering team and responsible for operating inside the
agreed dataops process.
- Own and extend the data pipelines through the collection, storage, processing, and
transformation of large data-sets.
- Assemble large, complex data sets that meet functional / non-functional business
- Identify, design, and implement internal process improvements: automating manual
processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
- Ensure that the infrastructure required for optimal integration, extraction, transformation,
and loading of data from a wide variety of data sources is built and maintained.
- Collaborate and support the Senior Engineer to define the business's data requirements,
which will ensure that the collected data is of a high quality and optimal for use across the
- Design, develop and maintain solutions to keep data separated and secure across on
premise and cloud based infrastructure and software.
- Collaborate with the senior analytics and data team colleagues through the Senior Engineer
to analyse and understand data sources, participate in design, and provide insights and
guidance on database technology and data modelling best practices.
- Design, and develop algorithms for real-time data processing within the business and create
the frameworks that enable quick and efficient data acquisition.
- Keep track of industry best practices and trends and apply knowledge in order to take
advantage of process and system improvement opportunities.
- Ensure that Data Governance procedure are well defined and applied through the data
- Ensure that all development occurs within a CICD approach.
Skills / Background:
- Strong communication skills, both written and verbal.
- Strong capability to convey complex messages in a clear, simplified, and understandable
- Advanced working SQL knowledge and experience working with relational databases, query
authoring (SQL) as well as working familiarity with a variety of databases.
- Experience building and optimizing data pipelines, architectures and data sets.
- Experience performing root cause analysis on internal and external data and processes to
answer specific business questions and identify opportunities for improvement.
- Strong analytic skills related to working with unstructured datasets.
- Build processes supporting data transformation, data structures, metadata, dependency and
- A history of manipulating, processing and extracting value from large disconnected datasets.
- Working knowledge of message queuing, stream processing, and highly scalable data stores.
- Project management and organizational skills.
- Experience supporting and working with cross-functional teams in a dynamic environment.
- Experience using the following software/tools:
- Experience with big data tools: Hadoop, Spark, Kafka, etc.
- Experience with relational SQL and NoSQL databases, including Postgres and
- Experience with data pipeline and workflow management tools: Azkaban, Luigi,
- Experience with AWS cloud services: EC2, EMR, RDS, Redshift
- Experience with stream-processing systems: Kafka, Storm, Spark-Streaming, etc.
- Experience with object-oriented/object function scripting languages: Clojure, Java,
We are an equal opportunities employer and welcome applications from all suitably qualified persons regardless of their race, sex, disability, religion/belief, sexual orientation, gender reassignment, marriage and civil partnerships, pregnancy or maternity or age
your application has been submitted