Problem statement
At Essent NL, the imperative to migrate multiple legacy systems into a cloud data warehouse is driven by the organization's strategic move towards cloud technology adoption, compounded by end-of-life considerations for aging technologies. An additional driver is the commitment to implementing a data mesh organizational approach, necessitating the utilization of cloud technologies to support decentralized data consumption across teams.
This project aims to seamlessly transition to the cloud, address technology obsolescence, and embrace a data mesh framework, ensuring enhanced data accessibility, collaboration, and long-term adaptability across the organization.
Contributions
Management, monitoring and administration of both the Snowflake and dbt platforms.
Knowledge transfer and coaching on the use of new technologies in the established team.
Improve on existing and implement new automations to reduce time-to-delivery for data products.
Establishing new and improving on existing ways-of-working for data testing, ingestion and transformation.
Ingestion, transformation and data modelling to support reporting and data science needs as well as the creation of data products.
Support and participation in pilots and workshops for new technologies and technology advances (Gen AI).
Impact
Increased end-user trust in the platform and solution by vastly increasing robustness of the nightly load, as well as raising maturity of the implementation processes.
Decreased ingestion time-to-delivery for new sources by 5x.
Supported data object growth within the project to 1800, of which 670 objects are user-facing.
CI/CD pipeline throughput time reduced by 6x and pruned approx. 25% redundant steps.
"Digital Hive helped Essent with a quality injection within the Essential Insights team. They brought on a Senior Data Engineer that seamlessly integrated within the team. Additionally, it was proposed to bring in a junior counterpart which expanded the capabilities of the team even more. Their collaborative efforts have added significant value to our data operations."
Anthony Roes
Product Owner Essential Insights
Outcome
The benefits of faster time-to-delivery for new data objects compound into both faster delivery of business value, as well as more rapid growth of the platform. Both are big drivers in the adoption of the platform and the happiness and perception of the end-users.
The streamlining of the implementation cycle results in delivery targets being met more comfortably. This allows for innovation time, which further increases the delivery of value and the speed with which this value gets delivered.