Senior Data Engineer will be part of a cross-disciplinary team, working closely with other data engineers, software engineers, data scientists, data managers, product owners and Portfolio managers.
- Architects, designs, implements and maintains reliable and scalable data infrastructure.
- Writes, deploys and maintains software to build, integrate, manage, maintain, and quality-assure data at bp.
- Adheres to and advocates for software engineering best practices (e.g. technical design, technical design review, unit testing, monitoring & alerting, checking in code, code review, documentation),
- Responsible for deploying secure and well-tested software that meets privacy and compliance requirements; develops, maintains and improves CI / CD pipeline,
- Responsible for service reliability and following site-reliability engineering best practices: on-call rotations for services they maintain, responsible for defining and maintaining SLAs. Design, build, deploy and maintain infrastructure as code. Containerizes server deployments.
- Actively contributes to improve developer velocity.
- Creating and code management of data pipelines and automation templates
- Testing new / changes to functionalities , creating test cases
- Designing data storage
Years of experience: 8 to 12 years with minimum of 5 to 7 years relevant experience
- Deep and hands-on experience (typically 5+ years) designing, planning, productionizing, maintaining and documenting reliable and scalable data infrastructure and data products in complex environments
- Development experience in one or more object-oriented programming languages (e.g. Py-spark, Python, Java, C++)
- Experience designing and implementing large-scale distributed data pipelines on AWS/Azure cloud (preferably AWS)
- Advanced data modeling and dimensional modeling knowledge
- Advanced SQL knowledge
- Deep knowledge and hands-on experience in technologies across all data lifecycle stages
- Experience of working in relational and non-relational data bases
- Experience using Azure DevOps , GitHub
- Continuous learning and improvement mindset