Data Engineer

Job summary

Role Synopsis:-
As part of bp “reinvent”, we have created a major new business line called “Innovation & Engineering” (I&E). One key remit of this group is to drive the transformation of the company through its use of digital and data. A major digital sub-team within I&E is Digital Production & Business Services (DP&BS). DP&BS are responsible for all digital and data initiatives and operations across the following areas of the bp business:

• Production & Projects including Health, Safety, Environment & Carbon
• Refining & Operations
• Wells & Subsurface
• Business Services including Finance, Procurement, People & Culture, Performance Management
• Strategy & Sustainability

“DataWorx” is the name for the data team that is responsible for all data within these area and we are developing deep data capability to transform the access, supply, control and quality to our vast and ever growing reserves of data. The DataWorx team covers many data sub-disciplines, including data science, analytics, engineering and management as well as specialist areas such as geospatial, remote sensing, knowledge management and digital twin.

Key Accountabilities

• Architects, designs, implements and maintains reliable and scalable data infrastructure
• Writes, deploys and maintains software to build, integrate, manage, maintain, and quality-assure data
• Design, develop, and deliver large-scale data ingestion, data processing, and data transformation projects on Azure
• Mentor and share knowledge with customers as well as provide architecture reviews, discussions, and prototypes
• Work with customers to deploy, manage, and audit best practices for cloud products
• Adheres to and advocates for software engineering best practices (e.g. technical design and review, unit testing, monitoring, alerting, checking in code, code review, documentation)
• Deploy secure and well-tested software that meets privacy and compliance requirements; develops, maintains and improves CI / CD pipeline
• Service reliability and following site-reliability engineering best practices: on-call rotations for services they maintain, responsible for defining and maintaining SLAs. Design, build, deploy and maintain infrastructure as code. Containerizes server deployments.
• Work closely with other data engineers, software engineers, data scientists, data managers and business partners

Essential Education :-
Bachelor or master’s degree in computer science, Engineering, Information Systems or in another quantitative fields

Job Requirements:-

• Years of experience: 4 to 8 years with minimum of 3 to 5 years relevant experience
• Deep and hands-on experience (typically 3+ years) designing, planning, productionizing, maintaining and documenting reliable and scalable data infrastructure and data products in complex environments
• Hands on experience with:
  • Databricks and using Spark for data processing
  • Configuring Delta Lake on Azure Databricks
  • Python, Scala, SQL
  • C#, ASP.NET, MVC, .NET Core, .Net Framework (4.6) JSON, and API development (optional)
  • Azure Data Factory
  • Azure Data Lake, Azure SQL DB, Synapse, and Cosmos DB
  • Data Management Gateway, Azure Storage Options, Stream Analytics and Event Hubs
  • Designing data solutions in Azure incl. data distributions and partitions, scalability, disaster recovery and high availability
• Advanced hand-on experience with different query languages
• Experience designing and implementing large-scale distributed systems
• Deep knowledge and hands-on experience in technologies across all data lifecycle stages
• Stakeholder management and ability to lead large organizations through influence

Desirable Criteria:-

• Strong stakeholder management
• Continuous learning and improvement mindset