Migrating multiple legacy systems into a cloud data warehouse at Essent

Essent is among the biggest energy companies in the Netherlands. The energy industry has been evolving rapidly for years now. So, to keep up with all the changes they are innovating their data landscape by migrating data from older, outdated systems to a modern cloud platform to make data more accessible and efficient for the business. This comes with significant challenges, both technical and organizational.

Challenge 

At Essent NL, the goal to migrate multiple legacy systems into a cloud data warehouse is driven by the organization's strategic move towards cloud technology adoption, compounded by end-of-life considerations for aging technologies. An additional driver is the commitment to implementing a data mesh approach, where teams take ownership of their own data, to create a more flexible and decentralized way of working. Which brings along a challenge of reorganizing and change management on its own. 

Solution 

We worked together with Essent to tackle their challenges with a focus on improving the Snowflake platform and standardizing the data transformation practices. 

1. Streamlining data ingestion 

   To enable the decommissioning of the outdated systems, we were responsible for the data ingestion of multiple sources into Snowflake. With a focus on efficiency, we improved the ingestion process and developed a Python tool for code generation to speed up dbt development. The Snowflake ingestion team can now make new data available in less than a day. 

2. Standardizing data transformation 

   To improve the data landscape, Essent has now adopted dbt as the default data transformation tool. This way the transformation is standardized, and the ownership and development of all transformations are decentralized with each team being responsible for their data. In order to get this implemented, we helped organize workshops and training sessions to get more teams on board with dbt. Thereby playing our role in the organizational challenge of moving to a data mesh approach. 

3. Implementing the logical data model 

   Next to ingesting new data and maintaining the Snowflake environment, our team also implemented the logical data model. The DMO (Data Management Office) discusses all entities in this model with the right stakeholders, ensuring the presentation of the data follows the company definitions and standards, ready to gather new insights through dashboarding and analysis. 

"Digital Hive helped Essent with a quality injection within the Essential Insights team. They brought on a Senior Data Engineer that seamlessly integrated within the team. Additionally, it was proposed to bring in a junior counterpart which expanded the capabilities of the team even more. Their collaborative efforts have added significant value to our data operations."

Anthony Roes

Product Owner Essential Insights

Results 

Looking back at our project at Essent, we have empowered the organization to make better data-driven decisions. Our role in the ingestion process has significantly improved the time to make data available. The integration of dbt as the default transformation tool has standardized the way-of-working and decentralized the data transformation efforts, creating a strong culture of ownership and accountability across teams. These changes have successfully helped Essent forward in current challenges, ready to adapt for future innovations in the energy industry.