Digital Hive blog

Headless 360 could change how we think about CRM

Thu, 27 Aug 2026 00:00:00 GMT

Salesforce and Anthropic announced Claudeforce this week, bringing Claude much deeper into the Salesforce ecosystem. Claude will be able to work with Salesforce data and actions, while Salesforce is also making Claude available within Agentforce and other parts of its AI stack.

There is plenty to unpack in the partnership itself, but one part of the announcement caught my attention more than anything else: Headless 360.

I think this is potentially much more important than the introduction of yet another AI assistant. It points towards a fundamental change in how we interact with CRM systems, and I expect that change to become very visible over the next two years.

From working in CRM to working with CRM

CRM has traditionally been an application that users have to actively work in. A salesperson opens Salesforce, navigates to an account or opportunity, looks at activities and dashboards, interprets that information and decides what to do next. Afterwards, they update the opportunity, create a task or move to another application to send an email or discuss something with a colleague.

We have spent a lot of time over the years trying to make that experience better. We simplify page layouts, remove unnecessary fields, build better dashboards and automate administrative tasks. A significant part of CRM adoption is ultimately about making it as easy as possible for people to correctly use the system.

Headless 360 starts to challenge that model.

Salesforce is exposing its applications and business capabilities so that authorized AI agents can discover and invoke them through technologies such as MCP and APIs. Those agents don't necessarily have to live inside Salesforce. Salesforce explicitly mentions Agentforce, Claude, ChatGPT, Cursor and other AI platforms.

That means the user no longer necessarily needs to go through the Salesforce interface to use Salesforce.

Imagine a salesperson starting the morning in Slack and asking what deserves their attention today. An AI agent could analyze the pipeline in Salesforce, combine it with recent customer communication, documents and internal conversations, and identify a few opportunities that require action. Instead of simply presenting a dashboard, it can explain why those opportunities matter and suggest what should happen next.

The user can then continue the conversation and ask the agent to execute those actions. Salesforce is updated, tasks are created and the appropriate workflows are triggered in the background.

The important thing is that Salesforce hasn't disappeared from this scenario. In fact, almost everything still depends on Salesforce. What has disappeared is the requirement for the user to interact directly with the Salesforce application.

That is a very different definition of CRM.

Salesforce as a business platform rather than a user interface

The move towards headless architecture isn't unique to CRM, but applying it at this scale to enterprise applications has interesting consequences.

Salesforce can increasingly become the underlying business platform that provides customer data, permissions, metadata, workflows and business logic, while another layer determines how users interact with those capabilities.

Today, we still tend to think of a Sales Cloud implementation as an application. In a more headless future, Sales Cloud could increasingly be considered a collection of trusted business capabilities that can be consumed by humans, agents and other applications.

An AI agent doesn't need to understand where a particular button sits on an Opportunity page. It needs to understand what an Opportunity is, what information is available, which business rules apply and which actions it is allowed to perform.

That distinction matters.

It also explains why I find Headless 360 strategically more interesting than Claudeforce itself. Salesforce is effectively accepting that it doesn't necessarily need to own the interface through which users consume Salesforce.

If Salesforce can remain the trusted layer underneath those interfaces, it can benefit regardless of which AI assistant ultimately becomes dominant.

Slack could become a much more logical place to work

This also makes Slack particularly interesting within the Salesforce ecosystem.

CRM has never been the place where all work happens. Teams discuss opportunities, customers and projects elsewhere. Documents live in different systems. Decisions are made in conversations and meetings. CRM is often the place where the result of that work is eventually recorded.

An AI agent changes that relationship because it can connect the conversation with the underlying business systems.

A discussion about a customer in Slack could immediately include the relevant Salesforce context without someone having to open the account. The agent could identify missing information, retrieve a document, explain the status of an opportunity and eventually execute an action in Salesforce.

That feels like a more natural working model to me than trying to move every activity into CRM.

It also makes Salesforce's ownership of Slack strategically important. Slack can become the collaboration environment where humans, documents and AI agents come together, while Salesforce provides much of the structured business context and execution behind it.

But where does this leave Agentforce?

This is the part of the announcement where I still have more questions than answers.

Claude is becoming an increasingly capable general-purpose agent. It can reason across documents, conversations and multiple enterprise systems. With Headless 360, it can now also discover and invoke Salesforce capabilities.

At the same time, Salesforce continues to invest heavily in Agentforce and is integrating Claude as a reasoning model within it.

That creates an interesting overlap.

If Agentforce is primarily positioned as an intelligent assistant that understands Salesforce and performs Salesforce actions, I think it risks being surpassed by general-purpose agents such as Claude. Those agents aren't limited to the Salesforce ecosystem and can reason across a much broader working context.

However, I don't think that necessarily makes Agentforce obsolete.

Its role may simply have to evolve.

Instead of competing with Claude on intelligence, Agentforce could become the Salesforce-native orchestration and governance layer for agents. It could determine which capabilities an agent can use, which actions it can execute, when human approval is required, which model is appropriate for a particular task and how all of those actions are monitored.

In that model, Claude and Agentforce aren't necessarily competitors.

Claude can provide much of the reasoning, while Agentforce provides the controlled environment in which that reasoning can safely interact with Salesforce.

I think this distinction will become important over the coming years. Salesforce will need to make the boundaries between Claude, Agentforce and Headless 360 increasingly clear. If all three are presented as different ways of creating intelligent Salesforce assistants, the proposition becomes confusing. If they become distinct layers in an enterprise AI architecture, the combination becomes much more compelling.

Governance becomes more important as the interface disappears

There is another reason why I don't think Salesforce becomes less relevant in a headless world: governance.

Reading Salesforce data with an AI assistant is relatively straightforward. Allowing an agent to actually perform business actions is a completely different matter.

Once an agent can update forecasts, change customer information, generate quotations, trigger approvals or communicate with customers, organizations need very clear answers to some basic questions.

What is the agent allowed to see? Which user is it acting on behalf of? Which actions can it execute autonomously? When does a human need to approve something? Which existing business rules still apply? And can the organization afterwards understand what the agent did and why?

This is where Salesforce has a significant advantage.

Organizations have spent years building their identity models, permissions, workflows, validation rules, approval processes and business logic into Salesforce. Headless doesn't mean bypassing that layer. The value is precisely that external agents can use Salesforce capabilities while those controls remain in place.

In my view, this will become one of the most important architectural principles for enterprise AI.

The smartest model won't automatically win in an enterprise context. Organizations also need a reliable execution layer around that intelligence.

This will also change what a good CRM implementation looks like

The consequences go beyond Salesforce's product strategy.

If users increasingly interact with CRM through AI agents, some of the things we currently spend significant implementation effort on will become less important, while others become considerably more important.

We will probably spend less time debating exactly where a field should appear on a page or how many clicks a salesperson needs to perform a certain action.

Instead, the quality of the underlying CRM model becomes critical. Data needs to be structured correctly. Business processes need to be explicit. Permissions need to make sense. Automations need to be predictable. The meaning of data needs to be clear enough for an agent to correctly interpret it.

Data quality becomes even more important as well. Poor data is annoying when a human is looking at a dashboard. Poor data becomes dangerous when an autonomous agent is making decisions and executing actions based on it.

The same applies to badly designed processes. AI can make a good process dramatically more efficient, but it can also execute a bad process at unprecedented speed.

For CRM consultants, I think this accelerates a shift that was already happening. Deep product knowledge will remain valuable, but knowing which Salesforce button to configure becomes less differentiating. Understanding the business, designing good processes, structuring data and translating business objectives into governed technology becomes much more important.

The next two years could fundamentally change CRM

I don't expect traditional CRM interfaces to disappear in two years. There will still be users working directly in Salesforce, and there will still be plenty of use cases where a structured application interface is the best way to work.

But I do think the balance will change quickly.

The question we have asked for years has been: how do we get users to work better in CRM?

Headless 360 introduces a different question: how can CRM support users wherever they are already working?

That sounds like a subtle distinction, but architecturally it is enormous.

Claudeforce is an interesting announcement and the Anthropic partnership will undoubtedly get most of the attention. I'm particularly curious to see how Claude and Agentforce will coexist as both products mature.

But for me, Headless 360 is the part to watch.

If Salesforce succeeds in turning CRM from an application people have to visit into a trusted business platform that agents can securely use from anywhere, we aren't just looking at another generation of Salesforce.

We're looking at a different way of working with CRM altogether.

Explain Snowflake EXPLAIN: Catching Join Explosions Before They Cost You

Wed, 03 Jun 2026 00:00:00 GMT

Every data engineer has been there: you write a dbt model, hit dbt run, and watch in horror as the warehouse spins for forty minutes. When you check the query profile, you find a massive join explosion that caused data to spill to remote disk storage, burning through credits.

When joining massive datasets, forgetting a secondary join key or making a typo in your ON clause can cause a row-count fan-out.

You do not have to run the actual query to know it is going to explode. By using Snowflake's EXPLAIN command, you can predict join explosions before the actual run. EXPLAIN calculates execution plans using Snowflake's global services layer. This is why it runs instantly and costs zero warehouse credits.

Join Fan-Out and Memory Spills

When Snowflake runs a join, it loads the data into the fast, local memory of your virtual warehouse. If your join logic is correct, the data processes smoothly.

If you accidentally create a partial Cartesian product, a table with ten million rows joined to a table with fifty thousand rows can suddenly explode into five hundred million rows.

When that happens, the data overflows the warehouse's local memory. Snowflake is forced to write those temporary rows to slow, remote disk storage. This is called a remote disk spill, and it is the primary reason queries stall and compute bills spike.

The EXPLAIN Plan Matrix

When you prepend EXPLAIN USING JSON to your query, Snowflake maps out exactly what it intends to do. It analyses the metadata of your tables and estimates the row counts for every single step of the operation.

An automated parser looking at an EXPLAIN plan can view an execution graph that looks like this:

Notice the Join node. The optimizer predicts that joining these two sets will produce five hundred million rows, which is exponentially larger than the cumulative inputs. That is your red flag.

Catching Many-to-Many Bridge Table Explosions

Another classic trap is joining two tables through an intermediate "bridge" table (like matching users to multiple assigned account teams or multi-tenant groups) where duplicate keys exist on both sides of the bridge.

You want to map web_traffic_events (50,000,000 rows) to user marketing_segments (100,000 rows) using a user_mapping_bridge table. You run your pre-flight check:

EXPLAIN SELECT * FROM web_traffic_events e

JOIN user_mapping_bridge b ON e.cookie_id = b.cookie_id

JOIN marketing_segments s ON b.segment_id = s.segment_id;

Reading the Native UI Output

You look at the estimated row metrics for the execution nodes:

The first join step outputs 250,000,000 rows from your initial 50 million events.
The final Join node predicts a massive output of 1.2 Billion rows!

The Fix

The EXPLAIN data shows you that the bridge table contains historical, overlapping duplicate mappings for the same cookie_id.

Instead of waiting an hour for a query that will fail due to remote disk spilling, you immediately rewrite the query to use a QUALIFY statement or an aggregated subquery on the bridge table to isolate the single, active mapping per cookie. Re-running EXPLAIN confirms the output rows now match your baseline 50 million events perfectly.

Summary

As a growing data engineer, shifting from reactive debugging to proactive engineering is a massive milestone. By utilizing EXPLAIN to flag join explosions during code review, you save your team thousands of dollars in wasted compute and keep your production pipelines running fast.

The Dashboard Migration Button Won't Save Your Snowflake Credits

Wed, 06 May 2026 00:00:00 GMT

Snowflake has made it official. On April 20, 2026, the creation of new Snowsight dashboards was permanently disabled across all accounts. On June 22, 2026, weeks from now, Legacy Dashboards will be fully removed from Snowsight. No access. No fallback. Gone.

If you are still running legacy dashboards, you are not in a grey area. You are on a hard deadline with no exceptions.

The obvious escape hatch is Snowflake's built-in "Generate Streamlit app" button: open your dashboard, click once, done. But at best it produces equal compute to what you had. At worst, with multiple concurrent users, it can cost you more. The default warehouse runtime spins up a separate app instance per viewer, where a legacy dashboard would have served them from a shared result. This post looks at what that means in practice and what to write instead.

What the Button Produces

The conversion tool does a 1-to-1 translation. Every dashboard tile becomes its own session.sql().to_pandas() call. The SQL is preserved verbatim. There is no caching, no in-memory filtering. A five-tile dashboard becomes five separate warehouse queries on every single page load and on every filter change.

What the button generates

session = get_active_session()

df1 = session.sql("SELECT region, SUM(sales_amount) ... GROUP BY region").to_pandas()

st.bar_chart(df1) # warehouse hit #1

df2 = session.sql("SELECT COUNT(DISTINCT customer_id) ...").to_pandas()

st.metric("Customers", df2['UNIQUE_CUSTOMERS'][0]) # warehouse hit #2

df3 = session.sql("SELECT DATE_TRUNC('month', order_date), SUM(sales_amount) ... GROUP BY 1").to_pandas()

st.line_chart(df3) # warehouse hit #3

Three tiles. Three warehouse spin-ups on load. Three more on every filter interaction. You have migrated the format, not the problem.

The Right Approach: One Query, Everything in Memory

Write one broader query, cache it with @st.cache_data, and build all your charts from that single DataFrame. Filter interactions operate on the cached data in memory with zero additional warehouse invocations.

The efficient approach

session = get_active_session()

@st.cache_data(ttl=3600) # warehouse hit once per hour

def load_data():

return session.sql("""

SELECT order_date, region, sales_amount, customer_id

FROM production.sales.fct_orders

WHERE order_date >= DATEADD(year, -1, CURRENT_DATE())

""").to_pandas()

df = load_data()

In-memory filter — no warehouse cost

selected_region = st.sidebar.selectbox("Region", ["All"] + df['REGION'].unique().tolist())

filtered = df[df['REGION'] == selected_region] if selected_region != "All" else df

All visuals from the same cached DataFrame

col1, col2 = st.columns(2)

col1.metric("Total Sales", f"${filtered['SALES_AMOUNT'].sum():,.2f}")

col2.metric("Active Customers", f"{filtered['CUSTOMER_ID'].nunique():,}")

trend = filtered.groupby(filtered['ORDER_DATE'].dt.to_period('M'))['SALES_AMOUNT'].sum().reset_index()

trend['ORDER_DATE'] = trend['ORDER_DATE'].dt.to_timestamp()

st.line_chart(trend, x='ORDER_DATE', y='SALES_AMOUNT')

Same three visuals. One warehouse query per hour. Every filter is instant and free.

The ttl=3600 parameter is the key control lever. It determines how long the DataFrame lives in memory before the warehouse is queried again. Match it to your pipeline's update frequency: overnight batch jobs can tolerate a longer TTL, operational dashboards may need a shorter one.

Button vs. Manual


	Generate Streamlit App	Manual Approach
Warehouse hits on load	One per tile	One total
Warehouse hits on filter	One per tile, every time	Zero
Caching	None	@st.cache_data
Time to migrate	Minutes	Hours
Credit efficiency	Same as legacy	Significantly reduced

Which Path Should You Take?

The two approaches are not mutually exclusive. If June 22 is bearing down on you, we suggest you approach the transition to Streamlit in to phases:

Phase 1: Use the button to meet the deadline. Convert everything before June 22. The generated apps will work and your data stays accessible.
Phase 2: Refactor what matters. After the deadline, audit your converted apps. Dashboards with the most tiles, the highest traffic, or the heaviest SQL are your refactoring priorities. For each one, consolidate the tile queries into a single cached base query. Not every dashboard is worth the effort. A low-traffic internal report with two simple tiles is not. A customer-facing dashboard with daily active users is.

Conclusion

Snowflake's migration button solves one problem: keeping you off a deprecated feature before June 22. It does not fix the compute inefficiency that will make your legacy dashboards expensive.

For the full technical reference, see the official Snowflake Streamlit documentation.

Managing Snowflake Infrastructure-as-Code

Fri, 10 Apr 2026 00:00:00 GMT

Modern data engineering teams typically rely on tools like dbt to manage data transformations through version-controlled, CI/CD-integrated pipelines. However, the underlying Snowflake infrastructure databases, schemas, virtual warehouses, and role-based access control (RBAC) is frequently provisioned using imperative SQL scripts or manual UI configurations.

While many engineering teams utilize third-party Infrastructure-as-Code (IaC) solutions like Terraform, Pulumi, ect... to manage these objects, these tools often introduce their own operational overhead, requiring external state file management, cross-platform authentication, and proficiency in domain-specific languages.

Snowflake Declarative Configuration Management (DCM) addresses this gap by providing a native, integrated IaC framework. This guide details how to implement DCM to manage a Medallion Architecture and outlines the architectural boundary between DCM and dbt.

Declarative State Management vs. Imperative Scripting

The traditional approach to Snowflake management relies on imperative execution (CREATE OR REPLACE, ALTER). Imperative scripts are inherently brittle. They require the developer to explicitly define the sequence of state changes.

DCM shifts this to a declarative model. You define the desired end-state of the Snowflake environment in local project files. During execution, the Snowflake CLI evaluates the delta between your local definitions and the current state of the target Snowflake account. It then dynamically generates and executes the necessary Data Definition Language (DDL) operations—CREATE, ALTER, or DROP—to achieve the target state.

This ensures idempotency. Repeated deployments of the same DCM project will yield no changes if the target environment is already synchronized with the codebase.

Architectural Boundary: DCM and dbt

Effective data platform management requires a strict separation between infrastructure provisioning and data modelling:

Infrastructure (DCM): DCM provisions the databases, schema structures (L0_RAW through L3_GOLD), virtual warehouses, roles and grants.

Logic (dbt): Executing DML to materialize business logic into the tables and views within the DCM-established schemas.

This separation ensures your transformation layer operates within a secure, audited, and version-controlled environment without infrastructure and logic competing for the same state.

Implementation Guide of a Medallion Architecture via DCM

The following steps demonstrate how to initialize a DCM project and provision a multi-environment Medallion Architecture.

Prerequisites

Python environment.
Snowflake CLI installed (pip install snowflake-cli-labs).
CLI connection configured in ~/.snowflake/config.toml (e.g., utilizing externalbrowser for SSO).

Step 1: Workspace Initialization

DCM enforces a standardized directory structure. The CLI specifically targets the sources/definitions/ directory for SQL definitions.

Initialize project directories

mkdir snowflake_cdm && cd snowflake_cdm mkdir -p dcm/sources/definitions

Initialize virtual environment and install CLI

python -m venv .venv source .venv/bin/activate # Windows: .venv\Scripts\activate pip install snowflake-cli-labs

Enable the DCM preview feature flag

export SNOWFLAKE_CLI_FEATURES_ENABLE_SNOWFLAKE_PROJECTS=true

Step 2: Manifest Configuration

The manifest.yml file dictates project metadata, deployment targets, and templating variables. It maps local execution contexts to persistent DCM Project objects within Snowflake.

File: dcm/manifest.yml

manifest_version: 2 type: DCM_PROJECT default_target: dev

targets: dev: project_name: DCM_ADMIN.PROJECTS.DCM_DEV templating_config: dev_config prod: project_name: DCM_ADMIN.PROJECTS.DCM_PROD templating_config: prod_config

templating: configurations: dev_config: db_name: ANALYTICS_DEV prod_config: db_name: ANALYTICS_PROD

Note: when using VSCode, the integrated YAML server may incorrectly flag the manifest file against the Snowflake Native App schema. To resolve this, change the file's Language Mode in the status bar from 'Snowflake Application Package Manifest' to 'YAML'.

Step 3: Declarative Object Definition

Infrastructure is defined using the DEFINE keyword rather than CREATE. Jinja2 templating is utilized to inject environment-specific variables defined in the manifest configuration.

File: dcm/sources/definitions/medallion.sql

-- Provision target database DEFINE DATABASE {{ db_name }};

-- Provision standard Medallion schemas DEFINE SCHEMA {{ db_name }}.L0_RAW; DEFINE SCHEMA {{ db_name }}.L1_BRONZE; DEFINE SCHEMA {{ db_name }}.L2_SILVER; DEFINE SCHEMA {{ db_name }}.L3_GOLD;

-- Provision base ingestion table DEFINE TABLE {{ db_name }}.L0_RAW.HEARTBEAT ( device_id VARCHAR, payload VARIANT, ingested_at TIMESTAMP_NTZ DEFAULT CURRENT_TIMESTAMP() );

Step 4: Plan and Deploy Lifecycle

The DCM deployment lifecycle consists of two distinct operations: validation and execution.

State Diff Generation (Plan)

The plan command performs a dry run, outputting the computed DDL required to align the target Snowflake environment with your local definitions.

cd dcm snow dcm plan --target dev

Execution (Deploy)

Once the deployment plan is verified, the deploy command applies the DDL transactions to the Snowflake account.

snow dcm deploy --target dev

Environment Promotion

To promote the infrastructure architecture to production, execute the deployment utilizing the prod target. The Snowflake CLI will automatically resolve the prod_config templating variables (e.g., ANALYTICS_PROD) and provision the isolated environment.

snow dcm deploy --target prod

Strict State Enforcement and Idempotency

It is vital to understand the operational impact of declarative IaC. The local repository serves as the absolute source of truth.

If an engineer manually executes a CREATE TABLE statement within the targeted Snowflake database (out-of-band change), that table is fundamentally orphaned from the IaC state. Upon the next execution of snow dcm deploy, the DCM engine will identify the unmanaged object and automatically issue a DROP statement to enforce the exact state defined in the codebase.

This behavior eliminates environment drift but strictly mandates that all infrastructure changes proceed through the established GitOps workflow.

Future DataOps

Looking forward, DCM is the foundation for a unified Snowflake DataOps platform, potentially integrating with Native Git for automated deployments and Dynamic Tables for declarative orchestration. By adopting DCM, you future-proof your stack for a code-centric Snowflake ecosystem.

From NiagaraFiles to Snowflake OpenFlow

Fri, 06 Mar 2026 00:00:00 GMT

The trajectory of data engineering is shifting from batch-oriented Extract, Transform, Load (ETL) processes toward continuous data logistics. Within this evolution is Apache NiFi, a technology originally developed within the National Security Agency (NSA) and recently integrated into the Snowflake Data Cloud as OpenFlow. This transition represents a significant change in how enterprises handle high-velocity, multimodal data ingestion and orchestration.

NSA Origins and Apache NiFi

Apache NiFi began as a project called NiagaraFiles in the early 2000s. The technical requirements at the NSA necessitated a system capable of moving massive volumes of data across geographically distributed and often unreliable networks while maintaining a strict chain of custody.

In 2014, the NSA released the source code to the Apache Software Foundation as part of its Technology Transfer Program. The software introduced a design paradigm known as Flow-Based Programming (FBP). Unlike traditional ETL tools that execute discrete jobs or scripts, NiFi treats data as a continuous stream of "FlowFiles."

In 2024, Snowflake acquired Datavolo, a company founded by the creators of Apache NiFi. Now Snowflake has integrated the core engine into its platform under the name OpenFlow.

Principles of NiFi:

FlowFiles: Each piece of data is encapsulated as an object containing both the content (the raw data) and attributes (metadata).
Processors: These are the functional units that perform operations such as fetching, filtering, transforming, or routing data.
Data Provenance: A native repository that records every event in the life of a FlowFile, providing an audit trail for compliance and debugging.
Back Pressure: A mechanism that allows the system to automatically throttle data producers when downstream consumers reach capacity, preventing buffer overflows and system crashes.

Defining Snowflake OpenFlow

Snowflake OpenFlow is a managed data integration service built on Apache NiFi 2.0. It serves as a visual orchestration layer that connects external data sources to Snowflake and, to other destinations.

The architecture is divided into two primary segments:

The Control Plane: Managed by Snowflake, this provides the browser-based visual canvas (accessible via Snowsight) and the APIs for managing flow definitions and monitoring.
The Data Plane: The execution environment where the actual data processing occurs. This can be deployed in two modes:

Snowpark Container Services (SPCS): A fully managed deployment where Snowflake provisions the underlying compute resources.
Bring Your Own Cloud (BYOC): A deployment where the OpenFlow runtimes run within the customer's own Virtual Private Cloud (VPC), typically on AWS or Azure, while still being managed by the Snowflake control plane.

The image illustrates the functional architecture of Snowflake OpenFlow, depicting how it streamlines the movement of diverse data.

Benefits of OpenFlow

Multimodal Data Support: Unlike many ingestion tools that focus on tabular data, OpenFlow inherits NiFi's ability to handle unstructured data such as images, audio, and PDFs. It integrates natively with Snowflake Cortex AI processors to perform operations like OCR or text summarization during the ingestion flow.
Reduced Operational Friction: In SPCS deployments, the infrastructure is managed by Snowflake, eliminating the need for data engineers to maintain separate virtual machines, manage Java Virtual Machine (JVM) tuning, or configure complex SSL/TLS certificates for the NiFi cluster.
Real-time Observability: The visual nature of OpenFlow allows engineers to inspect live data as it moves through the pipeline. Data provenance provides a granular view of every transformation, which is often difficult to achieve in code-based solutions like Airflow or custom scripts.

Trade-offs of OpenFlow

Learning Curve: Flow-Based Programming is a distinct discipline from traditional SQL or Python-based engineering. Teams without NiFi experience may find the "box-and-line" configuration more complex than writing standard scripts for edge cases.
Ecosystem Lock-in: While NiFi is open source, OpenFlow is deeply integrated into the Snowflake ecosystem. Utilizing specific Snowflake processors (e.g., Snowpipe Streaming) makes it difficult to migrate those pipelines to other data warehouses or lakes.
Connector Maturity: Compared to Fivetran’s 700+ pre-built connectors, OpenFlow’s library of "Turnkey Connectors" is still evolving. While the 300+ generic NiFi processors provide extensive connectivity, they often require more manual configuration for specific SaaS APIs.
Resource Consumption: NiFi is a resource-intensive application. In a BYOC model, the cost and maintenance of the underlying Kubernetes (EKS) or EC2 instances remain the responsibility of the customer’s DevOps team.

The Integration of AI and Governance

The main distinguishment for OpenFlow in is its role in AI Data Logistics. Because it can process data before it is loaded in the table, it functions as a preprocessing engine for Retrieval-Augmented Generation (RAG) workflows. For example, a flow can ingest raw documents from SharePoint, use a Cortex processor to generate vector embeddings, and write those embeddings directly into a Snowflake vector data type.

Furthermore, because OpenFlow uses Snowflake Role-Based Access Control (RBAC) and managed tokens, it inherits the security posture of the warehouse. This solves a common security gap where third-party ingestion tools require high-level administrative credentials to be stored outside the primary data environment.

Conclusion

The integration of Apache NiFi into Snowflake as OpenFlow marks a maturation of the data engineering stack. It acknowledges that ingestion is no longer just about moving rows from a database to a warehouse, but about managing complex, real-time data flows that involve AI processing. While tools like Fivetran will continue to dominate for standard SaaS-to-Warehouse replication due to their ease of use, OpenFlow provides the extensibility required for complex, low-latency, and high-governance environments.

Data engineers must evaluate whether the flexibility and native Snowflake integration outweigh the specialized skills required to master the NiFi engine.

References

How I Fast-Tracked My Snowflake SnowPro Certification using Generative AI

Tue, 03 Feb 2026 00:00:00 GMT

Introduction: The "Simple" Task

When I joined Digital Hive as their newest Data Engineer in November 2025, I was ready to hit the ground running. I brought with me a background in data engineering, specifically experience working with Google BigQuery and Databricks. I felt confident in my SQL skills, understood cloud data warehousing, and was eager to grow my knowledge.

Then came my first assignment.

My manager welcomed me to the team and presented me with a "simple" task to wrap up my onboarding: Get the Snowflake SnowPro Core certification before the end of the year.

I looked at the calendar. It was November. I had roughly six weeks.

In the past, a task like that in less than two months would have been a panic-inducing sprint. But let’s be honest; this is 2025. So, of course, I accelerated my learning using AI, like we all are doing.

I didn't just want to pass; I needed to actually understand the tools I'd be using daily. Here is how I combined structured learning with a personalized AI tutor to bridge the gap and pass the exam on my first try.

The Challenge: Translating Concepts

The biggest hurdle wasn’t learning how to write SQL: SELECT * FROM is fairly universal. The challenge was understanding the architecture and concepts under the hood.

Coming from BigQuery, I was used to a serverless world where compute management is largely abstracted away. Like Databricks, Snowflake requires a different way of thinking regarding Virtual Warehouses, credit usage, and specific caching layers. My previous experience with Databricks was always in a pre-configured environment, so I never needed to worry about the underlying resource management. Now, I had to master it.

Step 1: The Foundation (Udemy)

I knew that jumping into random documentation pages wouldn't be efficient. I needed a structured path to ensure I covered every domain of the exam. Since I learn better from video than text, I looked for a course that fit my style.

I enrolled in the Ultimate Snowflake SnowPro Core Certification Course & Exam on Udemy using their free trial. This gave me the necessary syllabus and video lectures to understand the breadth of the platform. It was excellent for getting the vocabulary right and understanding the basics of the ecosystem, from Snowpipe to Data Sharing.

However, watching videos is passive. To pass a certification like SnowPro Core, you need to understand the nuance. I found some practice exams online, and after failing one of them, I found myself with questions that the pre-recorded videos just couldn't answer.

Step 2: The Accelerator (Google Gemini)

This is where my study strategy shifted gears. I realized that memorizing answers from static practice exams wasn't enough; I needed to understand the logic behind them.

Instead of endlessly Googling confusing concepts or wading through technical forums, I started using Google Gemini as my personalized "Data Engineering Tutor."

My workflow was simple: I would take a practice exam, and whenever I hit a question I was unsure about (or got wrong), I wouldn't just look up the answer. I would paste the question into Gemini with the guided learning mode.

The difference was night and day. Instead of acting like a search engine, Gemini acted like a senior engineer sitting next to me. It didn't just give me the answer; it used the Socratic method to test my reasoning.

Here is a real example from one of my study sessions where I was confused about Snowflake workloads:

*Me: [Pastes question about whether Snowflake is OLAP or OLTP] "I think Snowflake is more suited for OLTP." *

*Gemini: "Actually, it is the other way around! Let's break this down. Think back to the Micro-partitions (our 'Books'). These files are immutable. If Snowflake has to rewrite an entire 50MB 'Book' every time you want to update one single row (like in a bank transaction), would that be efficient?" *

*Me: "No, you are right. The micro-partitions make it more suited for large datasets... so Analytics (OLAP)." *

*Gemini: "That is exactly right! 🏆 You reasoned it out perfectly. Because micro-partitions are immutable and optimized for reading huge chunks of data, Snowflake shines at Analytics." *

This guided learning mode was a game-changer. It helped me visualize abstract concepts like comparing Multi-Cluster Warehouses to opening more lanes on a highway during a traffic jam, or viewing Resource Monitors as "circuit breakers" for cost control.

By the time I finished my prep, I wasn't just memorizing "A, B, or C." I was confident I could explain why an answer was correct.

Step 3: The Deep Dive

Once I grasped the comparisons, I used Gemini to drill down into my weak spots. If I got a practice question wrong regarding Snowflake’s governance features or zero-copy cloning, I engaged in a dialogue with the AI.

I asked it to:

Generate scenario-based quiz questions regarding Time Travel and Fail-safe.
Summarize the differences between Standard, Enterprise, and Business Critical editions (a popular exam topic!).

This transformed my learning from passive reading to active interrogation. It allowed me to simulate months of "experience" by asking about edge cases I hadn't seen yet in real life.

The Result

On December 26th at midday, I sat down for the exam. I won't lie I was nervous. The SnowPro Core is known for asking tricky questions that test if you truly understand the architecture, not just the syntax.

The pressure was on: if I failed, I would be in a terrible mood, not exactly ideal for sitting down to Christmas dinner with my in-laws later that evening!

During the exam, I felt uncertain. The questions were designed to be tricky, with answer choices that often looked painfully similar. I finished the exam unsure if my strategy had actually worked.

To my surprise, I passed! I secured my certification just before the New Year's Eve deadline (and saved the Christmas dinner).

Takeaways: Why AI was the Key

If you are looking to get certified, especially on a tight timeline, traditional studying might not be enough. Here is my advice on how to use AI to fast-track your success:

Turn AI into a Study Partner: Tools like Gemini are incredible for filling in knowledge gaps. Don't just ask for the answer; ask it to explain the concept using comparisons to tools you already know (like BigQuery).
Pro Tip for Long Sessions: If you have a very long conversation, the Gemini thread can eventually become slow. If that happens, give it the instruction to "Summarize this conversation so far for the next tutor," then copy that summary into a fresh chat window to keep the momentum going.
Don't rely solely on Practice Exams: You can find several example exams online, and they serve as a good knowledge check. But in my experience, if you only memorize those without using AI to understand the "Why," you will struggle on the real exam.

I am thrilled to start 2026 fully certified and ready to contribute to the data team here at Digital Hive. Now, it’s time to put this certification to work!

Pipeline visibility from Airflow to dbt

Mon, 08 Dec 2025 00:00:00 GMT

When using Airflow to run your dbt models, some use cases may require having the exact DAG run timestamp stamped directly into your data. This allows you to trace the specific orchestration run that generated a row or table. This becomes a necessity when you have multiple DAGs with processes that generate data. Or if the DAGs overlap in the models they trigger or if you are running backfills and need to distinguish between data related time and processing related time.

Pipeline visibility from Airflow to dbt

Why not just current_timestamp( )?

The first instinct for many engineers is to simply include a current_timestamp() column in their SQL to capture when the model ran. While useful for simple auditing, this approach has a critical flaw: it captures Wall Clock Time, not Logical Time.

Airflow template references

To solve this, we need to leverage Airflow’s internal Jinja templating engine. Airflow exposes several variables that contain run metadata. Here is the cheat sheet for the most relevant ones:

Here, we will use {{ ts }} because it provides the full timestamp in a standard format that is easy to cast in SQL. However there move variables available than the ones mentioned in the table above on the following web page https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html.

Passing variables via CLI

The easiest way to bridge the gap between Airflow and dbt is to pass the variable directly into the dbt run command using the --vars flag. This ensures that the context is bound to the specific task execution.

Step 1: configure the Airflow DAG

In your Airflow DAG, you likely use the BashOperator to execute dbt. You need to construct your bash command to accept a JSON string of variables.

***Quoting ***

dbt expects the --vars argument to be a valid JSON dictionary. This means the keys and values inside the JSON must use double quotes ("). To prevent the bash shell from interpreting these quotes, we must wrap the entire JSON string in single quotes (').

Here is how to set it up in your Python DAG file:

from airflow import DAG 2. from airflow.operators.bash import BashOperator 3. from datetime import datetime 4.   5. with DAG("dbt_metadata_dag", start_date=datetime(2023, 1, 1), schedule="@daily") as dag: 6.   7. run_models = BashOperator( 8. task_id="dbt_run", 9. # We pass the Airflow {{ ts }} macro into the dbt command 10. bash_command="dbt run --select my_model --vars '{"dag_run_ts": "{{ ts }}"}'" 11. ) 12.

Step 2: Update the dbt model

Now that the variable is in the context, you can access it in your SQL model using the var() function. But this will cause issues if you run this model locally on your laptop, Airflow isn't there to provide the dag_run_ts variable, and your run will fail. To prevent this, provide a default value (like run_started_at, which is a built-in dbt variable).

Here is the SQL for an example model:

{{ config(materialized='table') }} 2.   3. SELECT 4. id, 5. order_amount, 6. status, 7. -- Get the variable 8. -- Use a default if missing (for local runs) 9. -- Cast the string to a timestamp 10. '{{ var("dag_run_ts", run_started_at) }}'::timestamp as dag_run_id 11. FROM {{ ref('stg_orders') }} 12.

Now, every time this model runs via Airflow, the dag_run_id column will contain the exact logical timestamp of the DAG.

Observability with Elementary

While passing the DAG timestamp helps with data lineage (tracing rows), it doesn't solve operational lineage (monitoring the health and stats of your runs).

For this, many teams turn to Elementary. Elementary is an open source dbt package that automatically collects run results and uploads them to a schema in your data warehouse.

What data does Elementary capture?

When you install the Elementary dbt package, it adds an on-run-end hook to your dbt project. This hook scans the artifacts dbt produces (run_results.json and manifest.json) and uploads them to your warehouse.

Without any custom Airflow configuration, Elementary captures the following data about your "Airflow-triggered" dbt runs:

Run status and timing:

Success/Failure status of every model.
Execution time (duration) per model.
Exact start and end time of the job (dbt command/invocation).

Data quality and volume:

Rows Affected: How many rows were inserted or updated in that specific run.
Freshness: When the source data was last loaded (if you use dbt source freshness).
Test Results: Pass/Fail status of all data tests associated with the run.

Job identity:

The specific dbt_cloud_run_id (dbt Cloud).
The invocation_id (a unique GUID generated by dbt for every command).

Bridging Elementary and Airflow

By default, Elementary knows everything about dbt, but nothing about Airflow. It sees a run, but it doesn't know "DAG A" caused it.

However, since Elementary simply reads the dbt artifacts, you can use the same logic we used above to enrich Elementary's data. By passing Airflow metadata like {{ dag.dag_id }} and {{ ts }} into dbt environment variables.

dbt_run = BashOperator( 2. task_id="dbt_run", 3. bash_command="dbt run", 4. # Pass Airflow macros as OS Environment Variables 5. env={ 6. "DBT_ORCHESTRATOR": "Airflow", 7. "DBT_AIRFLOW_DAG_ID": "{{ dag.dag_id }}", 8. "DBT_AIRFLOW_RUN_TS": "{{ ts }}", 9. # Required for dbt to find your project/profiles 10. "DBT_PROFILES_DIR": "/path/to/profiles", 11. **os.environ # Pass existing env vars if needed 12. } 13. )

Elementary can pick up custom_env_variables that are defined under the elementary variable block in the dbt_project.yml file.

vars: 2. elementary: 3. # Tell Elementary to look for these specific env vars 4. # and save them in the 'dbt_invocations' table 5. custom_env_vars: 6. - DBT_ORCHESTRATOR 7. - DBT_AIRFLOW_DAG_ID 8. - DBT_AIRFLOW_RUN_TS 9.

This creates a complete observability loop:

Airflow triggers the job and passes the ID.
dbt runs the logic and stamps the data with the ID and ts (using the approach described above).
Elementary observes the run and logs the metadata and row counts for alerts and dashboards.

Achieving total pipeline visibility

By using dbt run --vars, every data row gets the exact Airflow logical timestamp ({{ ts }}) for perfect lineage. Integrating Elementary closes, the loop by monitoring run health and collecting metadata (like DAG ID) through environment variables. This ensures data is traceable, and your entire pipeline is observable. This solves both data quality and operational challenges.

References

https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html

https://hub.getdbt.com/elementary-data/elementary/latest/

SnowPro Associate: Platform Certification

Thu, 30 Oct 2025 00:00:00 GMT

Recently I passed the Snowflake Platform certification exam. Here I want to quickly go over my Snowflake journey, what led me to take the exam and some tips regarding preparation for the exam.

SnowPro Associate: Platform Certification

My journey

I first started using the Snowflake platform about three years ago when I began working as a data engineer for Digital Hive. As part of my learning path, I got to know the main platforms and tools used within the company and at major clients. Besides the default Snowflake hands-on essentials track¹ I had the opportunity to explore more novel features such as Cortex² via Digital Hive’s partner account. After using snowflake in production for 6 months, i decided to try and get the SnowPro core certificate. At the time this was the only certification snowflake provided.

Shortly after obtaining the certification, I was fortunate enough to start working on a project where Snowflake was the main tool. I became more familiar with the practicalities of data ingestion and transformation in Snowflake. After more than a year and a half of using Snowflake daily, several more certificates became available. I found that the Associate: Platform Certification was closest in content to what I have been using on a day-to-day basis with respect to Snowflake features.

About the certification

The SnowPro Associate: Platform Certification guide¹ states that a minimum of three months of Snowflake usage is recommended before attempting the exam. I agree with this, as it is roughly half of what is required for the SnowPro Core certification. Unlike the Core exam, the Associate: Platform Certification focuses more on modern Cortex functionalities and general platform operations rather than broad architectural theory.

Here are the main topic domains and their weightings:


Domain	Weighting
Interacting with Snowflake and the Architecture	35%
Identity and Data Access Management	15%
Data Loading and Virtual Warehouses	40%
Data Protection and Data Sharing	10%

The certification has no prerequisites, which might make it attractive for new starters as an introductory exam. However, in my opinion, the SnowPro Core Certification should be everyone’s first introduction to Snowflake exams, since it has a broad coverage of all the Snowflake features, both on a theoretical and implementational level. So: no Core = no Platform!

Some additional details from the official documentation that are worth knowing:

The Associate: Platform Certification was announced as of February 3, 2025.⁵
The exam cost for the Associate level is USD 100.²
The exam is unproctored, meaning you take it online in your own environment and have 24 hours after scheduling to launch it. ⁵
Once you have the credential, it remains valid for two years. ⁵
The exam includes multiple choice, multiple select, and interactive question types such as matching and drag and drop. ⁵

The preparation

For this certification Snowflake has made live training sessions available via the Snowflake Education platform⁴. However, I do not think this is a must if you already obtained the SnowPro Core Certification and have used Snowflake long enough to be comfortable with the following activities:

Create databases and stages, and use compute resources
Load and leverage structured, semi-structured, and unstructured data
Understand Snowflake roles, and data access management
Understand and manage the Snowflake Account structure
Set up and navigate the Snowflake user interface

Aditionally what is important is a good understanding of the Snowflake Cortex AI functionalities and being up to date with the latest developments regarding:

Cortex LLM functions
Snowflake Notebooks

Memorise the different LLM functions available their capabilities and limitations. The same applies to Notebooks.

I was surprised when I planned the exam and redeemed my voucher that I received a notification that I should complete the exam within the next 24 hours, or the voucher would become invalid. My guess is that since there is no proctoring, there is also less scheduling overhead. So, make sure you do all your preparation first and only redeem your voucher on the day you want to take the exam.

Final thoughts

I see this certification more as an approachable refresher of Snowflake’s Cortex and novel platform capabilities. It is something that complements a practitioner’s day-to-day use of Snowflake rather than a deep architectural or migration credential.

I would recommend this exam to anyone who has been using Snowflake for a few months and wants to formalise their knowledge. It is not about deep theoretical mastery but about confident, effective use of the platform. Prepare properly, cover the features, and walk in with practical experience. This combination will give you the best shot at success.

References

SnowPro® Associate: Platform Certification - Snowflake University guide page
SnowPro® Certifications overview - Snowflake University
SnowPro® Core Certification guide - Snowflake University
SnowPro Registration and Training Resources - Snowflake Education
Announcement: SnowPro Associate: Platform Certification - Snowflake University
SnowPro Practice Exams - Snowflake University

Getting Databricks certified leaning on my Snowflake experience

Mon, 06 Oct 2025 00:00:00 GMT

In this blog, I want to have a look at my experience towards my newest certificate: Databricks Data Engineer Associate. I’ll cover why I pursued the certificate, how I prepared, and my thoughts on the resources and the exam itself.

DataBricks Certified Data Engineer Associate Exam

Why Databricks?

For the past two and a half years, I have been working with Snowflake as my primary data platform. With Databricks offering both similarities and differences, I was curious to dive into it and broaden my knowledge.

I am certainly not the first to compare the two. I found a great comparison in Graphable’s blog (Databricks vs Snowflake Wars: 7 Critical Differences). It highlights Snowflake’s origin in cloud data warehousing versus Databricks’ roots in data engineering and data science and it also states how they are growing towards each other with growing competitiveness.

This difference in origin is what interests me most. Since Databricks is typically more technical, it also has a higher learning curve than Snowflake. But with the right experience this does give more in-depth control. My combined background in Python and data engineering gives me that technical experience, which allowed me to really enjoy that learning experience towards the certificate.

Study resources

Before I started preparing, we (Digital Hive) became a Databricks Partner. One of the benefits of that is the Databricks Partner Academy. A platform where they offer many courses and learning paths. In preparation of different certificates, they offer role-based learning paths. In my case this is obviously the Data Engineering Path.

This path aligns closely with the exam topics, with the first half focusing on the Associate exam and the second half focusing on the Professional exam. A combination of video lessons and slides creates a good balance between theoretical content and a practical approach in demo videos where they are showing the topics discussed in a lab environment.

For the theory, I believe it covers everything you need to know to prepare for the exam. However, for some hands-on experience I felt like it is lacking. The demo videos, while having a more practical approach, had more focus on SQL. As Databricks is often associated with Spark and therefore PySpark, it is no surprise that the exam questions are more focused on Python.

One drawback I would like to mention is that the lab environment used in the videos is not available to practice with yourself. This would have been a valuable addition. You can still set up your own trial account and practice the topics on your own, but that would take a lot more time. Another alternative is instructor-led courses where I believe the lab environment is included. These are also available in the Partner Academy.

Besides the study material, I was also looking for a way to test my knowledge before taking the exam. This search led me to LeetQuiz, definitely one to remember! They offer free practice questions for many certificates. Besides Databricks they also offer questions and practice exams for AWS, Google Cloud and Microsoft Azure certifications. With a mix of real exam questions, exam-like questions and AI-generated questions, they have more than enough to test yourself. For my certificate for example, they (currently) have 943 questions available. Note: the AI-generated questions felt a bit off at times but overall, it was a very useful resource.

The exam

After studying with the mentioned resources, I planned my exam and got started. I have to admit my focus while preparing was mostly on theoretical content, since I did not create a trial account. This meant that, even though the practice exams went fine, I was a bit nervous to start the exam with minimal actual Databricks expertise.

In the end, this confirmed once again that learning new topics and broadening your horizons makes you more agile. I passed my exam by combining the theory I studied with my knowledge of general data engineering best practices I have learned throughout the years.

Looking back, I’m glad I took the leap. The certificate not only gave me more confidence in Databricks but also made me more versatile as a data engineer. If you’re already comfortable in one data platform and want to broaden your horizons, I can highly recommend the Databricks Associate exam.

My first year at Digital Hive: Growing as a CRM consultant

Thu, 18 Sep 2025 00:00:00 GMT

CRM consultant: Remco Daniels

In February, I began my journey at Digital Hive as a Junior CRM Consultant. Over the past seven months, I’ve been immersed in the world of consultancy, gaining valuable experience and learning key lessons along the way.

My first client: Solar Assistance

Just a few weeks into my role, I was assigned my first client, Solar Assistance, a company specializing in repairing and maintaining solar panels. They were already using Salesforce but needed more efficient processes within their CRM.

Initially, I assumed the most challenging part of consultancy would be the technical aspects, mastering every detail of a CRM system and building what clients need. However, I quickly realized that the real challenge lies in the preparation: learning methodologies like agile and understanding concepts such as story point estimation, which were entirely new to me.

Equally eye-opening was the level of client communication required. Coming from an admin role, I was used to working in the background with limited user interaction. As a consultant, the expectation is different: understanding client needs through discussions, framing the right solution, and clearly explaining why a specific approach was chosen. This was a steep but invaluable learning curve.

Providing IT support at Tosca

A few months later, I started working with Tosca, a global leader in reusable packaging and supply chain solutions. This project felt like a big step forward: not only was I introduced to a relatively new CRM environment, but I was also working with a globally distributed company with different processes and structures.

My role focuses on providing user support and managing system changes, from analyzing requests to testing and deployment. At first, the biggest challenge was understanding the new environment. I learned quickly that the best way to familiarize yourself is by diving in and solving tickets.

Although it was intimidating to realize some users knew the system better than I did, I adapted by asking the right questions and gradually became more confident. Gradually I was resolving issues more efficiently and even suggesting system improvements independently.

Building knowledge through certifications

In consultancy, credibility is essential. Certifications in platforms like Salesforce and Dynamics 365 not only demonstrate expertise but also enable us to work across multiple client environments.

Before joining Digital Hive, I had mainly focused on the skills needed for my day-to-day work. Now, I recognize the importance of broadening my knowledge base. Studying for certifications can be time-intensive, but I found ways to make the process more efficient. AI tools, for example, have been invaluable in helping me create summaries to structure my learning. By reviewing these first, I can process information more effectively and save significant time.

So far, I’ve achieved three Salesforce certifications and one Dynamics 365 certification. I’m proud of this progress and look forward to continuing to expand my expertise.

Working at Digital Hive

These past seven months have been an incredible growth experience. Digital Hive has played a key role in that journey. The company has a clear vision of what it stands for, and I’ve appreciated both the structured feedback and the supportive culture that drives personal development.

A special thanks goes to Mathieu, who has been an outstanding mentor. From sparring sessions that challenged my thinking to regular feedback and support, I’ve learned a great deal from him and hope to continue doing so.

Looking ahead

I’m grateful for the colleagues and clients I’ve met so far and for the opportunities I’ve had to grow as a consultant. I look forward to building on this foundation, continuing to learn, and supporting even more clients in the future.

DBT Certificate reflections and lessons learned

Mon, 08 Sep 2025 00:00:00 GMT

After my first year of working with dbt at Digital Hive I was encouraged as part of my growth path to take the dbt Analytics Engineering Certification Exam dbt Analytics engineer certification exam | dbt Labs and passed. As with most exams, things are never as they seem. With this blog I want to provide future exam participants with some pointers that could help them improve their preparation process for the exam.

Background

My background in biomedical sciences and bioinformatics has led me to start a professional career as a data engineer in the pharma and life sciences sector. Although it has been 3 years since I first came across dbt, I have only been intensively using it for the past year. While working on different projects, I managed to stay up to date with the latest dbt developments. This helped me a lot when I started my current project. Here it is the core tool. Hence, my first advice to anyone that is currently not working with dbt but wants to use it, is to stay up to date with the high-level developments and changes. This will keep the familiarity with the tool alive and prevent concepts and terms from fading from your memory even if they are not used on a daily basis.

Why did I take the exam

Besides the fact that certificates serve as confirmation of a person's competence in using a tool/platform, they can also be a great motivator to dive into the less used and more detailed features of that tool. After all the thrill of passing an IT exam comes quite close to that of passing your average college or university exams. By preparing and studying for the dbt exam you get insights into the structure of the documentation, and even if not all the topics are relevant, you will become familiar with navigating through it, which on its own is a great skill to acquire.

Learning the exact commands and object names may be boring, but it helps you describe your methods and solutions accurately. As a result, you won't always need to look up tool-specific terms online. These aspects come back during the exam when you have to recall terms under stress and limited time.

Setting up the study schedule

This part of preparing for the exam is crucial and very personal. For me, even though I had already worked with dbt for a year I planned to study for at least an hour a day three weeks in advance. As the saying goes “repetition is the mother of learning”.

A large part of the exam comes down to memorizable theory that is packaged in not only multiple-choice question but also Fill-in-the-blank, Matching, Hotspot, Build list and in my opinion the most challenging Discrete Option Multiple Choice (DOMC).

More practical knowledge is also tested on the exam. Therefore, one should make sure to set up the features that will be part of the examination (see study guide) at least once on a local learning project.

Make a schedule that covers all topics and combine specific topics to memorize documentation with trying out the features. The documents that are suggested as reading material reflect the values and thought process that are important for a dbt Analytical engineer. This is smoothing to always keep in the back of your mind and not to skip!

The study resources

Up to today there is only one major resource besides the documentation and dbt learn paths that are referred to in the study guide namely, the dbt community on qanalabs.com . The qanalabs.com dbt community provides a comprehensive and up to date collection of mock exam questions with detailed explanations.

Since I prefer to write my own summaries, I am not affected by the ongoing issue of limited access to condensed course materials. This will probably be resolved as soon as dbt grows in popularity. But for those who try to refrain as much as possible from taking their own notes qanalabs.com provides excellent and broad explanation on every topic the questions cover. Take into account that these are not ordered according to the study guide so you will need to organize that yourself.

In short preparing for the exam is not self evident and requires great attention to detail if you do not want have a blank sport in your knowledge and leave exam questions unanswered.

*(very important please read this several times in detail before starting to plan your study or even think about purchasing the exam voucher)

Booking the exam voucher

Read the study guide, this may have changed by the time you are reading this 😊.

The exam

Besides the different types of questions that make up the exam it does not differ from any mainstream proctored online exam. Except for two points that I noticed with the platform that maintains the proctoring talview.com (previously examity.com) was used

The first point of attention is that there is no pre-exam test link, so make sure to go through the documentation on what to do before the start of the exam Learner : Proctoring & Certification Support.

The second point of attention is that the exam will start only at the exact moment you scheduled it so opening the link and doing the identity and surroundings check too early will result in an awkward and silent moment with you looking into the camera.

During the exam it’s handy to flag the questions that you can't answer immediately or that are very time consuming (DOMC questions) and keep them for the end. Also make sure to read the questions correctly and review all your answers at least once!

Final notes

I can recommend the following blog for more resource detailed information regarding the exam.

www.digitalhive.be

The path to the dbt Developer Certification

Getting certified with dbt will test your knowledge of the framework. Read this blog for some tips and guidance to make sure you are ready.

If you still feel unsure and need to talk to someone or if you have specific questions. qanalabs.com has a comment section for the study exams where people frequently discuss their ideas and get question answered by the content maintainers.

The Future of AI with Salesforce Agentforce

Fri, 05 Sep 2025 00:00:00 GMT

The first AI, created in 1950, was like a strict recipe follower, executing a fixed set of instructions with no room for creativity or adjustment. Today’s AI is more like a skilled chef who tastes, adapts, and invents recipes dynamically, generating creative and responsive outputs. But what if AI could take it even further? Not just generating a recipe, but actively bringing the meal to life by buying ingredients, scheduling reservations, and managing unexpected challenges?

This is the promise of agentic AI: autonomous, proactive AI agents that use large language models (LLMs) to understand complex contexts, reason through decisions, and take independent actions to complete specialized tasks.

Introducing Salesforce Agentforce: Autonomous AI Agents to Scale Your Workforce

Salesforce Agentforce is a revolutionary new layer on the Salesforce Platform that empowers organizations to build and deploy these autonomous AI agents. Industry leaders like OpenTable, Saks, and Wiley are already using Agentforce to enhance their teams, scale operations, and elevate customer experiences.

While trust in autonomous AI remains cautious today, 77% of workers believe they will trust AI agents in the future, revealing a tremendous opportunity for businesses ready to adopt and integrate AI into their workflows in 2025 and beyond.

How AI Agents Will Change the Way We Work

An estimated 41% of employee time is currently spent on repetitive, low-impact work. Salesforce leaders foresee a future where AI agents handle these routine tasks, freeing humans to focus on strategic, relationship-driven work that actually moves the needle on revenue and innovation.

1. AI Will Become Simpler and More Actionable

“AI will become easier to implement within day-to-day business applications, allowing companies to securely integrate agents that use customer and business data to support processes,” explains Steve Hammond, EVP of Marketing Cloud. “This combination of ease and real-time actionability will boost confidence, utilization, and business value.”

2. Analytics Will Be Ambient and Seamless

By 2025, Nate Nichols, VP at Tableau, predicts that 25% of all analytical insights will be delivered ‘ambiently’, meaning AI will proactively surface insights embedded within daily workflows without users needing to seek them out. Whether it’s a smart suggestion during a meeting or a notification on a wearable device, data-driven decision-making will become effortless.

3. AI Agents Will Become the New Apps

Jayesh Govindarajan, EVP of Salesforce AI, compares AI agents to mobile apps in their transformative potential. These agents will be highly customizable and autonomous, capable of anticipating needs and optimizing tasks across industries. Conversational and embedded in workflows, they will elevate productivity and scale business operations intelligently.

4. New Jobs Will Demand Hybrid Skills

Lori Castillo Martinez, EVP of Talent Growth & Development, highlights the importance of cultivating a new blend of skills: technical, human, and soft. The future workforce will need programming and data analysis know-how, along with creativity, emotional intelligence, and problem-solving abilities to thrive alongside AI agents.

5. Mastering Unstructured Data Will Be Key

With 80% of enterprise data unstructured, Sarah Walker, COO of Slack, notes that the companies that can harness and make AI-ready this unstructured data will outperform competitors. From customer sentiment analysis to generating strategic content, the ability to navigate this data will define success in 2025.

6. Upskilling Will Be a Business Priority

Nathalie Scardino, Chief People Officer, predicts 2025 will be the year of upskilling, as companies race to equip employees with AI tools and knowledge to remain competitive. Employees will increasingly choose employers who invest in their continuous development.

7. AI Will Redefine the Employee Experience

Relina Bulchandani, EVP of Real Estate and Workplace Services, sees AI agents transforming how workers use office spaces, from booking meeting rooms to optimizing real estate portfolios, creating more efficient and collaborative workplaces.

How AI Agents Will Transform Industries

The impact of AI agents will differ across industries, unlocking new possibilities tailored to their unique needs.

1. SMBs Will Leapfrog Larger Competitors

Kris Billmaier, EVP of Sales Cloud, sees small and midsize businesses (SMBs) leveraging AI agents to scale rapidly, streamlining operations, engaging customers, and delivering personalized marketing. AI will empower these companies to compete and even surpass larger rivals.

2. Goodbye to Cumbersome Nonprofit Annual Reports

Nonprofits will evolve beyond slow, bulky reports, says Lori Freeman, VP & GM of Nonprofit. Donors will directly engage with AI agents that show real-time impact, provide proof points, and enable instant giving or volunteering, revolutionizing donor engagement.

3. AI-Powered Retail Experiences

Michael Affronti, SVP of Commerce Cloud, envisions AI agents transforming retail by providing personalized shopper assistance, especially during peak seasons. With AI already influencing 16% of sales in recent months, personal shopper agents will become a must-have for digital retailers.

4. Scaling Nonprofit Impact with AI Agents

Nonprofits are early adopters of AI, with 90% using it to boost engagement, according to Molly Ford, VP of Employer Brand. AI agents will help nonprofits optimize staffing, co-create volunteer roles, and connect employees to local volunteering opportunities.

5. Education Will See Rapid AI Adoption

Margo Martinez, VP of Education, explains how AI agents will help educational institutions understand costs at a granular level and reduce staff burnout by taking on scheduling and enrollment tasks. These agents will be crucial for managing seasonal surges in demand.

6. Government Services Will Become More Accessible

Nasi Jazayeri, EVP of Public Sector, predicts that by the end of 2025, AI agents will be widely deployed across federal and local agencies, helping citizens quickly renew passports, register vehicles, and understand benefits, making government services easier and more efficient.

How Humans Will Interact with Agents in the Future

Salesforce leaders offer deep insights into how AI agents will change our daily work and customer interactions.

Copilots will become commonplace, evolving from simple assistants to business-aware agents that handle strategic tasks (Adam Evans, EVP, Salesforce AI Platform).
We’ll see multi-agent teams tackling complex challenges, simulating product launches or marketing strategies with advanced reasoning (Mick Costigan, VP, Salesforce Futures).
Growing confidence in AI will accelerate adoption, transforming customer engagement and workflows (Rob Seaman, Chief Product Officer, Slack).
AI agents will become the preferred channel for customer interactions, requiring businesses to create cross-functional teams focused on optimizing AI experiences (Jon Belkowitz, Senior Director, Marketing Cloud).
The rise of personal AI agents will prompt companies to integrate “bring your own AI” policies (BYOAI) to keep pace (Mick Costigan).
Agentforce Inspectors will provide continuous monitoring and instant actions to improve business outcomes (Ryan Aytay, CEO, Tableau).
AI agents will collaborate in swarms, working together to solve strategic tasks seamlessly (Silvio Savarese, Chief Scientist, Salesforce AI Research).
Agents will fix fragmented customer journeys by sharing data and orchestrating smooth transitions between teams (Gabrielle Tao, SVP, Product Management).

The Bottom Line

Agentic AI represents a paradigm shift, from AI that assists to AI that acts autonomously and collaboratively. With Salesforce Agentforce, businesses can automate repetitive tasks, unlock strategic human potential, and transform industries through trusted, intelligent AI agents.

The future will be defined by humans and AI agents working side-by-side, combining creativity, empathy, and judgment with scalable autonomous execution to deliver extraordinary outcomes.

Processing both medical and commercial sales data from multiple countries

Tue, 22 Jul 2025 00:00:00 GMT

The project

Our team was tasked with processing both medical and commercial sales data from multiple countries to generate actionable recommendations for sales representatives. The recommendations provided details on which Health Care Professionals (HCPs) to contact, along with their communication preferences and the optimal frequency and topics for engagement. This initiative was designed to enable targeted outreach and improve overall sales effectiveness.

The challenge

This project presented several complex challenges that required innovative solutions and seamless collaboration across teams.

Regional Differences Between Countries

Pharmaceutical data varies widely between countries, with differences in HCP titles, product mappings, and therapeutic area classifications. To accommodate these regional nuances, we developed a flexible system where each country was assigned its own configuration file, layered over a shared global pipeline. This approach ensured that localized differences were captured without compromising the consistency of the data processing workflow.

Separation of Medical and Commercial Data

The project integrated data from two distinct sources: commercial data (such as sales figures and market share data purchased from third parties) and sensitive medical data. Given the stringent regulatory requirements in the pharma industry, we designed separate pipelines for processing each type of data. This separation ensured that medical data was never inadvertently used in commercial applications, while still allowing both data sets to undergo equivalent processing techniques for quality and consistency.

Collaboration Between Data Engineering (DE) and Data Science (DS)

The success of this project hinged on the strong collaboration between the Data Engineering and Data Science teams. We adopted Kedro—an open-source framework for data science code—to implement the pipelines, utilizing Python and Spark. The pipeline architecture was split into two main segments: the initial data ingestion and transformation was managed by the DE team, while the final data modeling and recommendation tasks were handled by the DS team. By maintaining all code in a single repository, we not only streamlined the development process but also fostered mutual understanding between the teams. A key benefit of using Kedro was its built-in lineage tracking, which enhanced transparency, reproducibility, and overall quality assurance.

(Example visualization available at: https://kedro.org)

Our contribution

We began our engagement on the project with a senior consultant integrated into the existing data engineering team. Our early efforts focused on optimizing and maintaining current pipelines, as well as designing new flows to enhance data processing efficiency.

Over time, as trust was established and our technical expertise recognized, we were entrusted with the end-to-end rollout of a new medical recommendation flow across the EMEA region. This innovative flow utilized AI-driven recommendations, drawing on insights from medical publications and expert analyses to guide sales strategies. One of our senior consultants assumed full ownership of this rollout, handling sprint planning, stakeholder communication, and ensuring the alignment of project objectives with business goals. Additionally, we expanded the team by onboarding and managing five junior engineers, including additional consultants from Digital Hive, to support the increased scope of work.

Conclusion

The project was a success, leading to increased sales figures for the affected products.

After an initial ramp-up period, our team assumed full responsibility for the medical recommendation flow, leading to a successful rollout across 20 EMEA countries.

This project is an excellent example of how robust data pipelines and strong DE-DS collaboration are critical to the success of AI-driven initiatives. By ensuring that the AI models were fed with high-quality, reliable data, we not only achieved operational excellence but also directly supported strategic decision-making in the pharma sector. Our integrated approach to data engineering and data science not only enhanced HCP engagement but also drove tangible improvements in business performance.

Orchestrating Snowflake queries in Airflow

Fri, 04 Jul 2025 00:00:00 GMT

When building data pipelines with Apache Airflow and Snowflake, executing SQL queries is a core operation. Whether you're managing test data, orchestrating data transformations, or cleaning up unused artifacts, it's important to understand the options available for running queries efficiently and scalable. This blog explores three ways to execute SQL queries on Snowflake from Airflow:

Using the SnowflakeOperator
Using the SnowflakeHook
Using a Python function wrapped SnowflakeOperator.

Network policy update

If you're using managed Airflow (e.g., AWS MWAA), you often need to whitelist the egress IP of your Airflow environment in Snowflake’s network policy. Without this, Snowflake may block the connection. You can add the IP using:

ALTER NETWORK POLICY my_policy SET ALLOWED_IP_LIST=('your_airflow_egress_ip');

Creating a Snowflake connection in Airflow

The connection to Snowflake in Airflow can be created and maintained using the Airflow UI connections. Follow the steps below to create the connection.

In the Airflow UI, navigate to Admin > Connections
Click + Add
Fill out:

Conn Id: SNOWFLAKE_CONN_ID
Conn Type: Snowflake
Provide credentials and connection details: account, user*, password, role, warehouse, database, schema.

*Preferably you would have a dedicated Service Account User with a private key. There are dedicated fields for this available.

Define Airflow Variables

In many cases, you may want to maintain reusable variables that are specific to a DAG or environment. This can also be managed directly from the Airflow UI.

In the Airflow UI, navigate to Admin > Variables
Click + Add
Fill out:

Key: MY_DAG_VARIABLE_ID
Val: Json dictionary with the variables that you want to pass to the DAG. This should contain the name of the connection made in the previouse step (SNOWFLAKE_CONN_ID).
Description:

You can access the connection variable from within the dag with the following code:

From airflow.models import Variable
snowflake_connection = Variable.get("DAG_VARIABLE_ID", deserialize_json=True)["SNOWFLAKE_CONN_ID"]

Approach 1: Static SQL with the SnowflakeOperator

The SnowflakeOperator offers a straightforward approach for executing SQL on Snowflake directly from Airflow. This approach is best for executing static queries as templated queries would need to be passed from another task that would generate the query.

from airflow.providers.snowflake.operators.snowflake import SnowflakeOperator
snowflake_direct_operator_task = SnowflakeOperator(
task_id="delete_test_data",
snowflake_conn_id=snowflake_connection,
sql="DELETE FROM dev_staging.orders WHERE is_test_data = TRUE;",
autocommit=True,
dag=dag,
)

Pros

SQL is executed on Snowflake compute, not Airflow
Tasks are fully visible and managed in the Airflow UI
Retries, logging, and failures are handled automatically

Cons

The SQL is static templated query usage is not straight foreword.
Looping or parameterization is difficult without rewriting the DAG

Approach 2: SQL execution using SnowflakeHook

Using the SnowflakeHook to connect to Snowflake allows SQL query templating and full control over query execution from within a Python function. This approach involves manually opening a connection and cursor. The function will then be executed in a task that uses the PythonOperator.

from airflow.providers.snowflake.hooks.snowflake import SnowflakeHook
from airflow.operators.python_operator import PythonOperator
def execute_snowflake_cursor(**kwargs):
hook = SnowflakeHook(snowflake_conn_id=snowflake_connection)
conn = hook.get_conn()
cursor = conn.cursor()
try:
cursor.execute("DELETE FROM dev_staging.orders WHERE is_test_data = TRUE;")
results = cursor.fetchall()
print("Results:", results)
finally:
cursor.close()
conn.close()
Define task using the PythonOperator
execute_snowflake_cursor_task = PythonOperator(
task_id="execute_snowflake_indirect_operator_task",
python_callable=execute_snowflake_operator,
op_kwargs={'snowflake_connection': snowflake_connection}, # Pass the connection here
provide_context=True,
dag=dag
)

Pros

Full flexibility in Python
Ideal for logic-heavy tasks, metadata inspection, or dynamic SQL
Can integrate complex control flow

Cons

SQL is executed inside the Airflow worker, increasing load and runtime
Less scalable for large operations
You manage error handling and connection lifecycle manually

This method is suitable for smaller, metadata-driven, or low-latency tasks where query results need to be processed immediately in Python.

Approach 3: Dynamic SnowflakeOperator Inside a Python function

The third approach combines the flexibility of Python with the performance of the SnowflakeOperator. You dynamically create and execute SnowflakeOperator tasks from inside a PythonOperator. This enables looping over dynamically generated SQL while preserving execution on Snowflake.

from airflow.providers.snowflake.operators.snowflake import SnowflakeOperator
def execute_dynamic_snowflake_ops(**kwargs):
table_names = ["orders_2023", "orders_2024"] # Dynamically calculated in real use
for table in table_names:
sql_stmt = f"DELETE FROM {table} WHERE is_test_data = TRUE;"
dynamic_op = SnowflakeOperator(
task_id=f"cleanup_{table}",
snowflake_conn_id=snowflake_connection,
sql=sql_stmt,
autocommit=True,
dag=kwargs["dag"],
)
dynamic_op.execute(context=kwargs)
define task using the PythonOperator
execute_snowflake_cursor_task = PythonOperator(
task_id="execute_snowflake_indirect_operator_task",
python_callable=execute_snowflake_operator,
op_kwargs={'snowflake_connection': snowflake_connection}, # Pass the connection here
provide_context=True,
dag=dag

Pros

Flexibility to calculate query content and targets dynamically
Keeps execution on Snowflake, avoiding Airflow compute usage
Useful for batch-style or templated SQL operations

Cons

All subqueries run within one Python task, reducing visibility in the Airflow UI
Failures may be harder to trace to specific queries
Loses automatic retry behavior for individual queries
More complex to manage and test

This hybrid method shines when you need to loop over dynamic inputs (dates, teams, business units) but still want Snowflake to do the work, not Airflow.

Summary Comparison


Feature	SnowflakeOperator	SnowflakeHook	SnowflakeOperator in Python Function
Compute Location	Snowflake	Airflow Worker	Snowflake
Flexibility	Low	High	High
Airflow Resource Load	Low	High	Low
Airflow UI Task Visibility	High	Medium	Medium
Dynamic Query Generation	Low	High	High
Best Use Case	Static queries	Metadata-based logic	Dynamic batch SQL execution

Conclusion

Use SnowflakeOperator when your SQL is static and doesn't depend on runtime variables. It’s Snowflake’s recommended method, as it keeps compute on Snowflake and lets Airflow orchestration.

Use SnowflakeHook with a cursor only when you need Python-level logic, such as looping over metadata or handling small result sets. Snowflake advises against this for heavy workloads, since execution happens on the Airflow worker.

Use SnowflakeOperator inside a Python function when you want to generate queries dynamically at runtime, for example when looping over tables or dates. This keeps compute on Snowflake, while allowing flexible control through Python.

From Snowflake's perspective, both the first and third approaches are aligned with best practices. Let Snowflake handle compute, and let Airflow orchestrate. This avoids overloading your scheduler and ensures scalable performance.

References

apache-airflow-providers-snowflake: https://airflow.apache.org/docs/apache-airflow-providers-snowflake/stable/index.html

Snowflake Connector for Python: https://docs.snowflake.com/en/developer-guide/python-connector/python-connector

Airflow Variables: https://airflow.apache.org/docs/apache-airflow-providers-snowflake/stable/connections/snowflake.html

Can I still automate ingestion if my source data is inconsistent ? - A solution using Snowflake's infer schema

Mon, 02 Jun 2025 00:00:00 GMT

When dealing with real world data we often face inconsistencies in source data structure, especially when those sources are file based structured or unstructured files. Snowflake provides a solution for tackling these issues. Here we dive into how you can safely handle source data inconsistencies using the Snowflake tool stack.

Originally intended to infer schemas from semi-structured or structured files during prototyping, Snowflakes INFER_SCHEMA can easily be implemented in production settings.

What is the problem

Imagine having a steady snowpipe that has not needed any updating or maintenance since it was created. Structured data files are being dropped on an external stage, and the pipe gets triggered and ingests the data into the predefined columns in your Snowflake table.

Everything is going smoothly until you get a new request from the marketing team for several new fields to be included. Now you need to not only update the pipe but also your destination table.

What is the best way to tackle the issue ?

One approach is to have your file content fully ingested into a file content column. This can be easily implemented. However, in the case of a generic medallion architecture, this approach will defer the issue from the bronze (raw) layer to the silver layer. Happy ingestion team, not so happy transformation team.

A more sustainable approach is to allow your source some flexibility and be able to handle this in the ingestion process. Here Snowflake offers a solution that is based on automatically detecting schemas and column definitions in structured and semi structured data files: INFER_SCHEMA.

Requirements

Stage: Your data file must be in a named stage (internal or external) or user stage.
Target Files: The LOCATION argument must point to one or more files. You can provide the full path to a specific file or point to a directory containing multiple file and use the MAX_FILE_COUNT argument to limit the amount of files that will be used to infer the schema.
File formats: INFER_SCHEMA works with semi-structured formats only which are the following files structures: Parquet, Avro, ORC, JSON, and CSV. This is not extension dependent. .txt files containing csv or json data can also be handled. The same goes for zipped files (.gz). You must specify a pre-defined FILE_FORMAT object, even for common types like JSON or CSV. This is how Snowflake knows how to interpret the raw files.

How does it work ?

Here we go over a typical case where we want to have the latest file dropped in an external stage to be ingested into a table in Snowflake. Historical data is not kept in this approach. This case is a good representation of ingesting reference data. We will work out the approach for ingesting csv files.

Create a file format

This file format will be used to parse the files staged at a given path.

CREATE FILE FORMAT my_csv_format 2. TYPE = csv 3. PARSE_HEADER = TRUE;

Note: Use PARSE_HEADER = TRUE in your file format to infer actual column names instead of defaulting to c1, c2, etc.

Create table using template

1. CREATE TABLE mytable 2. USING TEMPLATE ( 3. SELECT ARRAY_AGG(OBJECT_CONSTRUCT(*)) 4. FROM TABLE( 5. INFER_SCHEMA( 6. LOCATION=>'@mystage/csv/', 7. FILE_FORMAT=>'my_csv_format' 8. ) 9. ));

The INFER_SCHEMA function in combination with the USING TEMPLATE clause is used to auto-generate a table schema from file metadata. INFER_SCHEMA scans staged data (e.g., csv files) and returns metadata including column names, data types, nullability, and more*. This schema array is passed into USING TEMPLATE, which allows the construction of a table based on the schema definition.

Copy data into the created table

COPY INTO mytable 2. FROM '@mystage/csv/' 3. FILE_FORMAT = (FORMAT_NAME = 'my_csv_format') 4. MATCH_BY_COLUMN_NAME = CASE_INSENSITIVE;

The COPY INTO command is used to load the data from @mystage/csv/, using the predefined file format, and loads it into mytable.

The clause MATCH_BY_COLUMN_NAME = CASE_INSENSITIVE ensures that the mapping between source and target columns is done by name, ignoring case differences.

PURGE = TRUE automatically deletes the source file after it has been successfully ingested, preventing duplicate loads and saving storage.

Automate the process

You can wrap the ingestion logic inside a SQL stored procedure or script and schedule it to run at regular intervals. Tasks can be time-based using CRON expressions or event based.

CREATE OR REPLACE TASK infer_and_ingest_task 2. WAREHOUSE = my_wh 3. SCHEDULE = 'USING CRON 0 * * * * UTC' -- every hour 4. AS 5. BEGIN 6. CREATE OR REPLACE TABLE mytable 7. USING TEMPLATE ( 8. SELECT ARRAY_AGG(OBJECT_CONSTRUCT(*)) 9. FROM TABLE( 10. INFER_SCHEMA( 11. LOCATION=>'@mystage/csv/', 12. FILE_FORMAT=>'my_csv_format' 13. ) 14. ) 15. ); 16.   17. COPY INTO mytable 18. FROM '@mystage/csv/' 19. FILE_FORMAT = (FORMAT_NAME = 'my_csv_format') 20. MATCH_BY_COLUMN_NAME = CASE_INSENSITIVE 21. PURGE = TRUE; 22. END;

This makes schema-flexible ingestion pipelines fully automatable with little maintenance overhead. For large or enterprise workflows, you can integrate these tasks into orchestration platforms like Apache Airflow or dbt Cloud.

Pro tip

Instead of using the inferred data types, hardcode column definitions using OBJECT_CONSTRUCT and manually assign all columns to a VARCHAR datatype. See the example below.

CREATE TABLE mytable 2. USING TEMPLATE ( 3. SELECT ARRAY_AGG( 4. OBJECT_CONSTRUCT( 5. 'COLUMN_NAME', COLUMN_NAME, 6. 'TYPE', 'VARCHAR', 7. 'NULLABLE', NULLABLE 8. ) 9. ) 10. FROM TABLE( 11. INFER_SCHEMA( 12. LOCATION=>'@mystage/csv/', 13. FILE_FORMAT=>'my_csv_format' 14. ) 15. ));

*INFER_SCHEMA metadata

Limitations

All the variations of timestamp data types (DATE_FORMAT, TIME_FORMAT, and TIMESTAMP_FORMAT) are retrieved as TIMESTAMP_NTZ without any time zone information.
For CSV files, SKIP_HEADER does not work in schema inference mode. Use PARSE_HEADER = TRUE in your file format instead.
Using * for ARRAY_AGG(OBJECT_CONSTRUCT()) can cause errors if the returned result is larger than 16MB. Only use the required columns, COLUMN NAME, TYPE, and NULLABLE, for the query.
For files with nested data, only the first level of nesting is supported.

Conclusion

If your source data structure is inconsistent or evolves frequently, don’t fight it. Adapt your pipeline using INFER_SCHEMA and Snowflake Tasks. This approach removes the friction from handling schema drift, eliminates the need for frequent manual DDL updates, and allows your ingestion pipelines to stay responsive to upstream changes. With automation via Snowflake Tasks, dynamic table creation, and flexible ingestion strategies, your data platform can evolve as fast as your source data does.

References

File formats: https://docs.snowflake.com/en/sql-reference/sql/create-file-format

Infer schema: https://docs.snowflake.com/en/sql-reference/functions/infer_schema

Copy into: https://docs.snowflake.com/en/sql-reference/sql/copy-into-table

Data loading: https://docs.snowflake.com/en/user-guide/data-load-overview#label-detect-column-definitions-in-semi-structured-data-files

dbt build selector: A Practical Guide

Wed, 30 Apr 2025 00:00:00 GMT

When you're working with dbt, especially on larger projects, you don’t always want to build every model every time. That’s where selectors come in. By grouping the models that need to be refreshed in one command time and resources are used more efficiently. With the dbt build command, selectors give you fine-grained control over exactly what gets built. In this post, we’ll break down how to use the --select and --exclude options with dbt build and how combining them can supercharge your workflow.

Before we dive in, let’s clarify a few important flags and commands:

--select and -s are interchangeable. They both define what to build.
--models is an older flag that does the same thing as --select, but it’s now considered deprecated. It still works, but it’s best to switch to --select or -s in modern dbt workflows.
--exclude defines what not to build.
The dbt list or dbt ls command will allow you to list the selected models. It is recommended to use this command to test whether the selection you use will build the models you intend to build.

A selector tells dbt which nodes (models, tests, snapshots, etc.) you want to act on. With dbt build, this means which parts of your DAG should be compiled, run, and tested.

The syntax is:

dbt build --select [selector]

List equivalent:

dbt list --select [selector]

You can also exclude nodes:

dbt build --select [selector] --exclude [other-selector]

List equivalent:

dbt list--select [selector] --exclude [other-selector]

Let’s walk through the common ways to use selectors.

1. Build a Single Model

To build just one model:

dbt build --select my_model

This runs the model my_model, along with any relevant tests and post-hook actions.

2. Build a Model and Its Dependencies

To build a model and all of the upstream models it depends on:

dbt build --select +my_model

The + before the model means “include all parents”.

To build a model and all of its downstream dependents:

dbt build --select my_model+

Want to include both upstream and downstream? Use:

dbt build --select +my_model+

3. Build a Whole Directory or Path

You can select all models in a specific folder:

dbt build --select path:models/staging/

This runs everything in models/staging/, recursively.

4. Use Tags for Logical Grouping

You can assign tags to models in your .yml or model files. Then use:

dbt build --select tag:nightly

This is perfect for grouping models by schedule or theme.

5. Build by Resource Type

Want to build only tests or snapshots? Use:

dbt build --select resource_type:data_test dbt build --select resource_type:snapshot

This is useful if you want to isolate part of the pipeline for debugging or CI.

6. Exclude Specific Models

To build everything except certain models:

dbt build --select tag:nightly --exclude my_model

This is great for skipping a known-broken model or one that’s not relevant to your current work.

7. Combine Multiple Selectors

You can combine selectors using commas (,), which means OR:

dbt build --select model_a,model_b

Use spaces for AND:

dbt build --select tag:nightly path:models/marts/

This will select models that match both the tag and the path.

You can also nest selectors for more control:

dbt build --select +tag:nightly+

8. Special Selector: state:

When using dbt in a CI/CD pipeline, you can build only what has changed:

dbt build --select state:modified --state path/to/artifacts

This builds only models that have changed (and their children) since the last run.

9. Build All Upstream Dependencies with @

The @ selector in dbt is a powerful tool for building a model, all its downstream nodes (dependents), and all dependencies (ancestors) required by those downstream nodes. This ensures that every model needed to build any node downstream of the selected model is also refreshed.

dbt build --select @my_model

Usecase

For example, your dashboard relies on 3 models (report_a, report_b, and report_c), and you've updated report_a. To ensure proper integration with the other two reports and their dependencies:

[stg_orders] → [report_b]

↘

[stg_customers] → [report_a] → [dashboard]

↗

[stg_products] → [report_c]

dbt build --select @report_a

The command above will build:

report_a
Upstream models report_b, report_c, stg_customers
Ancestors of upstream models stg_orders and stg_products.

Whereas the following command:

dbt build --select +report_a

Will build:

report_a
stg_customers

Pro Tip

When testing a new dbt build command, use dbt list or dbt ls instead. It shows which models would be affected, without running them. Example: dbt list --select my_model > build_1.txt This saves the output to a file, so you can easily compare changes across different selector versions.

Final Thoughts

Selectors are essential for managing large dbt projects efficiently. Whether you’re debugging, optimizing CI pipelines, or running targeted builds, understanding how to use --select and --exclude gives you speed and precision.

Start small, test a single model and its dependencies. Then build up with tags, paths, and more. With a little practice, you’ll be navigating complex DAGs like a pro.

Snowflake external stage limit exceeded

Fri, 04 Apr 2025 00:00:00 GMT

When querying an external stage you should be aware of the following limitations. You may encounter an error 001057, which relates to exceeding file descriptor limits. This error can occur during operations that involve listing or processing a large number of files in a stage. Let's dive into the details of this error and explore strategies to address it effectively.

Error Details

001057 (0A000): Total size (>=1,073,741,912 bytes) for the list of file descriptors returned from the stage exceeded limit (1,073,741,824 bytes); Number of file descriptors returned is >=3,490,462. Please use a prefix in the stage location or pattern option to reduce the number of files.

The Total size here refers to the size of the file descriptors. The file descriptors are references to the selected files in the staging area. When querying or loading data, Snowflake accesses files from a staging area, which is a temporary storage location.

If the number of files or their combined metadata exceeds Snowflake’s threshold (in this case, 1GB or ~1,073,741,824 bytes), the database cannot handle the volume efficiently, resulting in this error. This typically occurs with large or unoptimized datasets, especially when files accumulate over time or when a generic query pattern (like a wildcard) retrieves all files in the stage.

The error can be thrown by any of the operations listed in the table below.

Strategies to Avoid the Error

This limitation can be avoided by making sure you do not process more metadata than the limit in file descriptors. There are several approaches that can be implemented.

1. Minimize metadata by cleaning the stage.

Apply a snowflake PURGE or when using S3 buckets as externals stage use a lambda function to remove the data that is not required from the stage. An example of how the PURGE command can be used is shown below.

COPY INTO my_table

FROM @my_stage

PURGE = TRUE;

2. Querry only the required part of the stage.

Use the directory name, or part of the directory name in the stage to narrow down on the data you want to query. @stage_name/directory

For instance if you have the following folder structure on your stage:

my_stage/ ├── a_dir1/ │ ├── sub_dir1/ │ └── sub_dir2/ ├── b_dir2/ ├── sub_dir1/ └── sub_dir2/

Using the query below will only select data from directories that start with 'b_' in my_stage.

SELECT * FROM '@my_stage/b_ '

3. Use Patterns to Filter Files

In Snowflake, stages can contain many files, and sometimes the list of files may also include directories which do not contain data or data that we do not want to query. These directories can cause errors if not handled properly. The pattern option allows you to specify a regular expression pattern (enclosed in single quotes) that matches the file names or file paths. The files whose names or paths match the regular expression are the ones included in the query result. More specifically the pattern is applied before any data processing takes place, and it limits which files are included in the query.

For example, the pattern used in the select statement below will only query the files that have the string “data” in their file name and end with .csv.

SELECT * FROM @my_stage (PATTERN => '.data..csv$');

Quick fix

If you are in a situation where the data that you need to query exceeds the file descriptors size limit a quick solution can be to loop over the query using a list of strings that represent the first several characters of the directory names that you want to query. This way you will query the stage in smaller chunks. You can use that same approach with the PATTERN. Keep in mind that this should not be a permanent solution.

1. Generate a List of Prefixes or Patterns

Create a list of strings representing directory names, prefixes, or file patterns.

2. Loop Through the List

Query the stage multiple times, each time focusing on a specific prefix or pattern. The example below uses letters but you can use this with any prefix of your directory names.

WITH prefixes AS ( SELECT column1 AS prefix FROM VALUES ('a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z') ) SELECT * FROM @my_stage WHERE directory_name ILIKE prefix || '%';

This fix reduces the file descriptors processed per query, mitigating the snowflake error. However, this will increase the runtime due to the multiple queries that are ran. It is important to make sure the correct prefixes and patterns are generated and used.

Conclusion

Error 001057 reflects Snowflake’s limits on file descriptors during stage queries. To resolve this:

Keep stages clean using PURGE or lifecycle policies.
Use directories and patterns to narrow queries.
Split large queries into smaller chunks.

For more information you can consult the snowflake documentation on this topic :

Querying Data in Staged Files | Snowflake Documentation

From Synapse to Fabric: How Glowi Modernized Its Data Architecture

Thu, 20 Mar 2025 00:00:00 GMT

Client Introduction

Glowi is a multi-brand organization, including Het Poetsbureau, which operates 75 offices across Flanders and has over 10K domestic helpers employed. They mostly have data about customers and employees. It is mainly used to create the schedules, matching the right customers to the right employees. It is also used as an input for their recruiting and marketing teams.

Challenge

The current data setup is built using Microsoft synapse on azure infrastructure.

It is focused on being cost effective and fit for purpose. Tailored to the size and requirements of Glowi. To stay up to date with new technologies, they wanted to investigate the possibility of moving to Microsoft Fabric (What is Microsoft Fabric - Microsoft Fabric | Microsoft Learn), the new data platform by Microsoft. This should be able to replace their older synapse system.

There are many reasons to move to Fabric. It is clear that the focus of Microsoft is on developing this platform, shifting away from synapse. This means all the new development and features will be added to Fabric. Furthermore, the platform will be closer integrated with PowerBI and machine learning use cases, ensuring that Glowi is both future-proof and ready for AI.

One of the main challenges here was cost. The current system was very effective, and the team at Glowi managed to align hardware with the requirements of the data, keeping the costs very low. The question was if it would be possible to migrate to Fabric without significantly increasing costs. Both costs related to infrastructure but also costs related to licenses (both Fabric and PowerBI licenses).

The other challenge was the infrastructure behind setting up the new Fabric environment. Since Fabric is a new offering, no one at Glowi had deep knowledge of the platform. This means that they were unsure about how to implement it, and what the best way of doing things on the platform is. This goes from how to run jobs to best practices and limitations of the platform.

Scope

A workshop around efficiently ingesting new data into the Fabric lakehouses
Investigation around capacity and licenses.
A POC of one month to do an initial onboarding and setup of the Fabric platform.

Results

**Cost analysis **

To tackle the concerns around cost we provided Glowi with an overview of the costs related to using Fabric and PowerBI. We also provided them with an overview of the costs, comparing Synapse and Fabric (part of this analysis can be found in A comparison of Spark pools in Synapse and Fabric). The main result from this analysis is that is it highly preferred to use reserved capacity, since this comes at quite a discount. There are however also some limitations, since the reserved capacity is available 24/7, and if you are not using it, you are still paying for it. To identify the best capacity for their jobs, we suggested the use of pay-as-you-go resources to identify the right size. Afterwards this capacity can then be reserved for a year or longer. It is also taken into consideration when designing the infrastructure, with runs over night and analysis during the day, to make sure the reserved capacity is fully used.

POC: setup and infrastructure

During a one month POC, we set up the infrastructure of their new Fabric environment, and migrated a first data source to this new environment. This was done using a medallion architecture, making use of a lakehouse for each layer. Multiple ingestion flows were defined, with data coming in from api’s and azure databases. An example of this medallion architecture in fabric can be seen below:

As a part of the POC, we identified a few key flows in the current synapse workflows, and we reimplemented them in fabric. This allowed us to set up the way of working, using pyspark in notebooks, reusing shared logic, and allow orchestration through notebooks. This provides a clear starting point for future implementation of new data sources.

We will now provide the details of some of the choices we made during this POC:

Implementation using **pyspark **and notebooks: Moving away from the more ui-based workflows in spark, we decided to do all the ETL using pyspark and notebooks. This makes it a lot easier to test and standardize the way the ETL is done. It also allows for easier versioning and sharing of the code. A clear advantage of this was ensuring all data writes were done using a specific helper function, this allowed us to easily add logging to all flows later on, using only a single line of code.
Medallion architecture with each layer in its own lakehouse: Following this standard as advised by Microsoft, we made a lakehouse for the bronze, silver, and gold layer of the medallion architecture. This allows for a clear separation of data. Furthermore it makes it easier to set up different access rules and rights to the data. In each of these lakehouses we store the data in delta parquet, which is perfect for distributed access using pyspark. In the final layer, we suggest a setup of the data in a dimensional model, using facts and dimensions.
Orchestration using notebooks: while it is possible to create a workflow to run notebooks, for now the most efficient way is to use a notebook that references other notebooks. This means that for each layer we created an orchestration notebook, which firstly called the helper notebook with shared functions, and then in turned called the notebooks relevant to that layer, structured based on the data that was being processed. This means that if you need to find the details about some data, you can easily find it by searching for the concept in the right layer. (eg Employee, Customer …). It also makes it easy to find the shared logic, since this is in a separate notebook.
To ingest new data into the system, we defined three different methods. Firstly, for the azure databases that allowed this, we set up mirroring in Fabric. This ensures that the data from this database is automatically copied into the delta lake, without having any extra maintenance or setup. We can then setup a shortcut to allow access to this data from the bronze lakehouse. Secondly, for databases that are too small to allow mirroring, we designed a reusable notebook setup that allows for incremental ingestion of the data into the bronze lakehouse. Thirdly, for data coming from API sources, the same setup can be followed, creating a python notebook that accesses the api and loads it into the bronze lakehouse. This allows for API ingestion that fits perfectly into the tech stack and infrastructure, making it easy to be maintained.

Conclusion

Together with Glowi, we created the basis of their Fabric infrastructure, ensuring that they are future-proof and ready to get started with this new platform. We used our expertise to make sure the setup and infrastructure fits the needs and expectations of Glowi.

In a short period of time, we were able to not only design and document the setup, but to also onboard a first data source into this system, giving them a strong starting point for further development.

When we started this project, Glowi mostly had questions about Fabric, now they have a clear idea of what the future will look like, and even better, they have a working proof of concept that can be used to judge the platform and make it their own.

Accelerating Clinical Studies with AI

Thu, 13 Mar 2025 00:00:00 GMT

Accelerating Clinical Studies with AI by Digital Hive

Challenges

Pharmaceutical companies operate in a high-stakes environment where speed and accuracy are critical. The profitability of new products hinges on the patent-protected period, a narrow window before generic alternatives enter the market. However, clinical studies - essential for regulatory approval - consume valuable time, typically taking 18-24 months. Every delay shortens the exclusive revenue period, impacting profitability.

To stay ahead, pharmaceutical companies are turning to AI to streamline clinical study development. While AI offers immense potential, its implementation comes with key challenges:

Maintaining scientific integrity by ensuring AI-generated outputs are accurate and reliable.
Navigating strict legal frameworks, especially concerning copyright-protected materials.
Providing intuitive review mechanisms to keep human experts in control.
Ensuring airtight security—no proprietary data can be exposed beyond the internal environment.

Solution

Digital Hive responded with Evidence CoPilot (ECP): a cutting-edge AI-powered application designed to transform clinical study development. A dedicated internal team of data scientists, guided by a product owner and project manager of Digital Hive, engineered ECP to accelerate workflows while ensuring compliance and security. The platform consists of four core components:

Literature Search

An AI agent scans multiple platforms for the most relevant publications, providing researchers with instant insights.
Both open-access and licensed, copyright-protected materials are accessible, ensuring a comprehensive research base.
Articles are delivered in an organized format, ready for review and integration into subsequent study phases.

Synopsis Generator

AI rapidly summarizes selected articles, whether sourced through Literature Search or uploaded manually.
The concise, structured synopses enable faster decision-making and can be refined or exported as needed.

Systematic Literature Review (SLR)

Users compile structured reports leveraging insights from previous searches and summaries.
SLR serves as a foundation for study development, ensuring a robust, evidence-based approach.
Built-in review functionality guarantees that human expertise remains at the forefront.

Publication Development

ECP automates the final study document’s creation using predefined templates, expediting the evidence generation process.
AI accelerates screening, analysis, summarization, and structured report generation.
Researchers can efficiently transition from data collection to publication without compromising quality.

Results

After a year of development, ECP is now ready to be used, delivering measurable impact for pharmaceutical companies:

AI-driven automated reviews ensure quality control at every step.
Advanced AI prompts minimize errors, boosting the reliability of insights.
User-centric control allows researchers to review, edit, and export results at any stage.
Enterprise-grade security keeps all generated content private within a protected portal—no data is exposed online.

With ECP, Digital Hive has empowered pharmaceutical companies to fast-track clinical study preparation, maximize patent profitability, and maintain full regulatory compliance. By leveraging AI for efficiency, they can now focus on what truly matters: bringing life-saving innovations to market, faster and smarter.

Interested how Digital Hive can help you implement AI-driven solutions tailored to your business needs? Get in touch today to start accelerate your work and stay ahead in an evolving market!

Transforming Digital Performance with Martech Optimization for Solar Assitance

Thu, 13 Mar 2025 00:00:00 GMT

Transforming Digital Performance with Martech Optimization

Challenges

Solar Assistance is Belgium’s leading expert in technical assistance, maintenance, and troubleshooting for solar systems in the B2C sector. In B2B, the company specializes in Energy Management Systems (EMS), maintenance and monitoring of solar farms, and solar panel installations. The company wanted to take its marketing strategy to the next level. After a scan we saw the following opportunities:

Achieve higher conversion rates
Increase interaction with customers
Optimise Martech stack architecture

Solution

A comprehensive digital marketing and Martech optimization roadmap was drawn up to address these issues:

Website & Conversion rate Optimization

Redesign the website with better UX and a simplified conversion funnel.
Improve CTAs (Call-To-Actions) and implement A/B testing to enhance lead capture.
Integrate dynamic forms and chatbots for improved lead qualification.

Salesforce & Martech Integration

Connect Salesforce, the website, and ad platforms (Google Ads, Meta, LinkedIn) to create a unified data flow.
Implement real-time lead tracking and automated lead nurturing.
Develop custom dashboards for better marketing and sales alignment.

Pay Per Click (PPC) Campaigns

Improve targeting using first-party data from the integrated Martech stack.
Optimize Google Ads and Meta campaigns for better return on investment (ROI) and lower cost per acquisition (CPA).
Use lookalike and retargeting audiences to maximize lead conversion.

Content & SEO Strategy

Develop SEO-driven content to improve organic search ranking.
Create engaging blog posts, video content, and case studies to educate customers.
Improve social media engagement through a structured content calendar.

Data & Performance Framework

Design and implement a structured Measurement Plan based on clear KPIs.
Set up Google Analytics, Google Tag Manager, and Salesforce tracking for end-to-end attribution.
Create automated performance reports to track campaign effectiveness and sales conversion rates.

Conclusion

Digital Hive’s expertise in CRM integration, conversion rate optimization, and performance marketing played a key role in overcoming Solar Assistance’s challenges. With a robust Martech ecosystem in place, Solar Assistance will be ready for sustained growth, improved efficiency, and enhanced customer engagement in a competitive market.

Marketing Consultancy for Glowi

Mon, 10 Mar 2025 00:00:00 GMT

Marketing Consultancy for a Dynamic Business Landscape

Client Introduction

Glowi is a multi-brand organization, including Het Poetsbureau, which operates 75 offices across Flanders. The company sought to enhance its marketing efforts across various brands, requiring versatile and hands-on support to manage numerous tasks efficiently.

Problem Statement

Glowi needed a marketing consultant who could coordinate marketing campaigns and internal projects across external partners, ensure consistent communication, and handle a broad spectrum of marketing tasks. The company required someone adaptable and capable of quickly shifting focus to meet the needs of a fast-paced and diverse business environment. Additionally, Glowi needed a partner capable of delivering both strategic insights and hands-on execution, with the agility to shift gears quickly in a fast-paced environment.

Approach

With a flexible marketing skillset, our consultant took on a variety of roles and responsibilities, tailoring the approach to meet Glowi’s dynamic needs:

Campaign and Event Coordination

Playing a central role in coordinating campaigns, working with external partners to ensure flawless execution across multiple channels. From events to the display stickers at all 75 Het Poetsbureau offices, efforts were coordinated between Glowi and external partners for smooth execution.

Internal Communication

Maintaining a cohesive internal culture, our consultant leveraged platforms like SharePoint to keep employees informed about key projects and events, ensuring all employees were informed.

Gadgets, Print, and Gifts Coordination

Coordinating the production and distribution of promotional materials, including T-shirts, flyers, brochures, pancartes, etc. Additionally, overseeing the selection and delivery of gifts for employees during key occasions such as Easter and year-end celebrations, further solidifying Glowi’s employee engagement initiatives.

Email Marketing and Automation

Utilizing ActiveCampaign and Flexmail to design and execute email marketing campaigns. Also implementing marketing automation flows, streamlining communication with employees and enhancing engagement.

Community and Social Media Management

Managing Glowi’s community presence, ensuring responsive communication and engaging content. Also providing support for social media content creation, developing posts and videos that showcased the company’s culture and diverse services.

Design and Creative Support

Beyond digital efforts, contributing to offline marketing by designing materials such as banners, brochures, and other print materials. The ability to shift between online and offline marketing helped ensure a consistent and cohesive brand presence across all touchpoints.

Collaboration with BluePanda

Coordinating efforts with BluePanda, a sister company of Digital Hive located in Portugal, which provided nearshoring solutions for Glowi. BluePanda worked on an SEO/SEA track for Glowi Facilities, and our consultant played a key role in ensuring smooth communication and alignment between Glowi and BluePanda. This collaboration helped enhance Glowi’s online presence through targeted SEO and SEA campaigns.

Results

Improved Campaign Execution

The seamless coordination of the upcoming campaign ensures high visibility across Glowi’s extensive footprint, particularly for Het Poetsbureau.

Increased Employee Engagement

Internal communications and thoughtfully coordinated gifts boosted employee morale and engagement, reinforcing Glowi’s strong company culture.

Efficient Marketing Operations

By implementing automation flows and effectively managing the marketing inbox, our consultant helped streamline Glowi’s marketing efforts, making communication more efficient and targeted.

Agile and Flexible Support

A hands-on and adaptable approach allowed our consultant to meet evolving business needs quickly, contributing to the smooth execution of both large-scale campaigns and day-to-day marketing activities.

The road to POC app building with Streamlit

Fri, 28 Feb 2025 00:00:00 GMT

**When developing a proof-of-concept (POC) application, the aim is to rapidly validate the core logic, assumptions, and functionalities of an idea before committing to full-scale development. Here, a Python-based tool like Streamlit can be invaluable for quickly building, deploying, and sharing a POC. In this blog, we explore the benefits and limitations of using it to build POC apps. **

Streamlit POC app

What is Streamlit?

Streamlit is an open-source framework designed to simplify the creation of interactive, data-driven web applications. Originally launched in late 2019 by Adrien Treuille, Thiago Teixeira, and Amanda Kelly, it was built to empower data scientists and engineers to create and share applications without needing extensive web development expertise. In March 2022, Snowflake, the data cloud company, acquired Streamlit, enhancing its enterprise capabilities while maintaining its core mission to simplify app development for data professionals.

Why choose Streamlit for POC applications?

As a Python-only solution, Streamlit allows developers to quickly transform scripts into fully interactive web apps, with minimal code and without requiring front-end development skills. This approach enables you to focus on building interfaces using buttons, sliders, and text input boxes, allowing attention to stay on application logic instead of UI complexity.

Automatic UI generation

Streamlit’s auto-generated UI feature saves time by creating a web app interface directly from Python scripts, eliminating the need for HTML, CSS, or JavaScript. Simply running the code builds a user-friendly app where users can interact with your logic and data immediately.

Prebuilt widgets

For interactivity, Streamlit offers a wide range of widgets, from sliders and buttons to date pickers and file uploaders, making it easy to add input and control options. The widget library is continually growing, giving developers more flexibility in user input choices.

Instant code re-runs

The "hot-reloading" feature offers a smooth, real-time feedback loop while developing, which is particularly useful for POCs. This way, developers can see the effects of code edits or parameter adjustments right away, making testing quick and efficient.

Simple deployment and sharing

One-click deployment through Streamlit Cloud allows for easy sharing of applications, ideal for getting POC feedback quickly. There’s no need to set up complex web infrastructure, so you can deploy and share your app with collaborators or stakeholders via a single link.

What should I keep in mind?

While Streamlit offers many benefits for POC development, it’s not without its limitations. As development progresses, consider the following factors:

Performance constraints with large data or heavy computation

Due to its single-threaded nature, Streamlit may struggle with large datasets or computation-heavy tasks. For POCs involving extensive data processing, caching can help optimize performance, and for even heavier computations, offloading tasks to a separate backend might be necessary.

Customization limits

As a project scales, you might encounter limitations when customizing layouts, styling, or advanced UI elements. While custom components can be integrated, this may require front-end expertise and reduce the initial simplicity that makes this framework so appealing for POCs.

Deployment considerations

Streamlit Cloud is great for rapid deployment, though it requires your code to be hosted on GitHub, which may not align with every organization’s policies. Additionally, Streamlit Cloud is designed for small-scale deployment, so high-traffic or multi-user scenarios may require deploying on other platforms like Heroku, AWS, or a Dockerized environment for greater flexibility.

Conclusion

Streamlit’s ease of use, fast development, and interactivity make it an excellent choice for testing and validating application logic in POCs. Its native support for data visualization and simple deployment options make it especially valuable for early-stage prototypes. However, limitations in scalability, customization, and performance mean it’s best suited for small to medium-sized projects. For POCs that need to grow into production applications or require complex interactions, transitioning to a more versatile framework may eventually be necessary.

The Path to SnowPro Core Certification

Fri, 31 Jan 2025 00:00:00 GMT

Snowflake is a popular data lake/warehouse. It offers many different tools that data engineers can utilize for their data pipelines. As a data engineer that is just starting out in this world, Snowflake is a good and relatively simple platform to learn. It is great to get started and going for the SnowPro Core certification helps you understand many different concepts within the data engineering world, with a focus on Snowflake of course. Additionally, some of these concepts can be applied to other data platforms as well.

This certification and the learning materials provided by Snowflake give you a good baseline for the data engineering world. In this blog, we will go over my journey and all the materials that I utilized to achieve my SnowPro Core Certification, with the hope to inspire other people to do the same. Note that you do not need to follow this journey step by step, I encourage you to utilize the resources listed here and make your own journey to SnowPro!

The certification

On the Snowflake website, they provide an overview, exam breakdown and domain breakdown of the SnowPro Core Certification at the following url: https://learn.snowflake.com/en/certifications/snowpro-core/.

I will not go into much detail about this information. However, it is still important to take a close look at this, so you know the practical information about the exam and the scoring weights of each domain. Then you know which domains are more important than others.

Additionally, on this website in the right column there are some links to (free) courses that are interesting learning materials: Exam study guide, getting started tutorials, hands-on courses and the level up series. I will now discuss these in more detail.

Exam study guide

This guide provides an overview of everything related to the exam. Most importantly, it lists a lot of terms and concepts, with the relevant study resources, for each domain. Go through this list and take note of the items you know and do not know. This will make it easier later to focus on the concepts that you do not know yet. We will come back to the study resources later.

Getting started

There is a very small getting started workshop that explains some of the basic concepts of Snowflake, with demo code, that can be completed in about an hour or two. Parts of this course are listed in the exam study guide. So, for completion I list it here as well: https://quickstarts.snowflake.com/guide/getting_started_with_snowflake/index.html

I recommend completing the steps in the hands-on essentials courses on how to create a trial, then you can use it for both. However, I think that the hands-on essentials course is a better place to get started, as those courses explain the relevant concepts in more detail.

Hands-on Essentials

For myself, I know that I learn the most from practical experience. That is why I decided to start with the Hands-on Essentials courses: https://learn.snowflake.com/en/pages/hands-on-essentials-track/ . These are divided into 5 different parts and for each part you can achieve a badge on Accredible, which you can then show off to your friends and/or colleagues.

These courses are built around a real-world story, and all require you to get your hands dirty. You need to create a (specific) Snowflake trial to complete this course. Snowflake provides a testing suite, called DORA, that grades all your tasks/assignments. DORA will determine whether you get your badge or not. Most tasks you must complete are simple, with easy to follow and well explained ideas and instructions. However, some tasks require you to think and implement some concepts for yourself. From this, you can learn a lot!

Level up track

Now that we have completed the hands-on essentials courses and achieved our fancy new badges, it is time to dive into the theory of Snowflake. These level-up courses, available at the following url: https://learn.snowflake.com/en/pages/level-up-track/, provide an excellent overview of almost every component required to pass the SnowPro Core certification exam. It requires a lot of reading in the documentation, but this is well worth it as you get a lot of information from this! Study this documentation in detail, as questions can arise from everything on these pages. Including specific permission names, commands, syntax... These level up courses are also listed as a resource in the exam study guide.

Exam preparation

Once you have completed these courses and read through the documentation, it is time to test your knowledge and (potentially) fill in the gaps. There are several ways to do this, below I will list the ones that I used on my journey to certification, in addition to the exam study guide mentioned above.

SnowPro Core Certification On-Demand Preparation Course

This (paid) course, offered by Snowflake, is meant for last-minute preparation and provides a summary of all knowledge required for the exam. There are a significant number of questions in this course that can be used to test yourself. If you do not know the answer to a question, the relevant information is always listed on the previous pages in the course. This course really gives you a boost in both knowledge and confidence for the exam!

Online exam questions

There are many examples of exam questions available online. Some are paid; some are free. Snowflake provides a paid practice exam with 40 questions (for reference: the real exam is 100 questions): https://learn.snowflake.com/en/certifications/snowpro-practice-exams/.

I personally did not use the Snowflake practice exam, instead I utilized the ExamTopics exam questions available at https://www.examtopics.com/exams/snowflake/snowpro-core/view/. This is a community driven list of exam questions that were asked on previous exams. At the time of writing, about 300 questions are available for free, the rest is paid. People can vote on the correct answer for each question; in most cases, the indicated answers seem to be correct. However, there are some drawbacks: not all questions are updated with the latest answer as Snowflake is constantly updated causing some things to change. Additionally, some questions are configured wrong. For example: a question asks to indicate the 3 best solutions, but the suggested “correct” answer only selects two. If you ever doubt an answer, there is a community discussion section where people list a specific page of the documentation with the correct answer: thank you community! While this is not perfect, I found this to be extremely helpful as you get a very good idea of the kind of questions that are on the real exam, especially when combined with the on-demand preparation course.

Final words of encouragement

Once you have completed all these tutorials, courses and practice questions, you should be able to pass the certification exam for SnowPro Core. It was still a challenge on the exam itself, but that makes it all the more rewarding once you pass!

All the resources I mentioned are listed again, for your convenience 😉:

SnowPro Core: https://learn.snowflake.com/en/certifications/snowpro-core/
Snowflake getting started: https://quickstarts.snowflake.com/guide/getting_started_with_snowflake/index.html
Hands-on Essentials courses: https://learn.snowflake.com/en/pages/hands-on-essentials-track/
Level up track: https://learn.snowflake.com/en/pages/level-up-track/
Snowflake practice exam: https://learn.snowflake.com/en/certifications/snowpro-practice-exams/
ExamTopics example questions: https://www.examtopics.com/exams/snowflake/snowpro-core/view/

Best of luck! And remember that if you need Snowflake expertise; do not hesitate to contact us for one of the data consultants at Digital Hive!

The Tipping Point for Life Sciences: Embracing the Digital Revolution

Thu, 16 Jan 2025 00:00:00 GMT

The life sciences industry is ahead of a groundbreaking transformation where technology, data, and connectivity come together to enhance patient outcomes, streamline care delivery, and empower healthcare professionals (HCPs). This paradigm shift was one of the focus points in the Veeva Commercial Summit 2024, where life sciences industry leaders explored how recent innovations are going to be revolutionizing the industry.

Understanding Medical 2.0

Medical 2.0 represents the future of healthcare - a digitally connected, data-driven, and patient-centric ecosystem. By leveraging AI in healthcare, real-world evidence (RWE), and cloud-based platforms, this paradigm shift is paving the way for personalized care, smarter decision-making, and improved outcomes for both HCPs and patients.

Key Features for the Future of Life Sciences

Enhanced Digital Collaboration

For years there has been an emphasis on seamless collaboration between teams using digital tools. Platforms like Veeva Vault CRM and Salesforce Life Sciences Cloud enable a unified approach, integrating real-time data from clinical trials, patient outcomes, and scientific research to provide HCPs with a single source of truth.

Data-Driven Insights

Access to RWD and advanced analytics can empower organizations to tailor their digital strategies. By understanding patient behaviors, treatment adherence, and HCP preferences, companies can design personalized interventions that drive better engagement and more informed decision-making.

AI-Driven Solutions

Artificial Intelligence will be the core of Medical 2.0, offering tools that will transform how healthcare is delivered. From virtual assistants that streamline medical inquiries to predictive algorithms that forecast patient needs, or next best suggested actions for sales reps to take. AI will be enabling smarter and more productive systems within the industry.

Patient-Centric Care

The shift towards personalisation is another cornerstone of the future. By leveraging genomic data and patient-specific information, HCPs can deliver treatments tailored to individual needs, ensuring higher efficacy, reduced side effects, and better outcomes.

Integrated Omnichannel Engagement

There will be an increasing need to facilitates omnichannel engagement, ensuring that HCPs and patients receive consistent, valuable information across platforms—be it virtual consultations, in-person visits, or digital content.

The Impact on the Pharmaceutical Industry

Empowering Medical Affairs Teams

Medical affairs teams will be increasingly leveraging tools to align scientific communication with real-world evidence. This alignment ensures that HCPs receive timely, accurate, and actionable insights to improve patient care.

Redefining Field Engagement

Tools like pre-call planning dashboards and voice-enabled CRM assistants will transform field teams into strategic partners. This shift emphasizes the importance of AI in pharma sales and how digital tools can improve HCP engagement and collaboration.

Supporting Continuous Learning

Continuous Medical Education (CME) is evolving. Digital platforms offer personalized, on-demand learning modules that keep HCPs updated on the latest advancements in medicine and patient care strategies.

Driving Better Outcomes with Real-Time Data

By integrating real-time clinical data, pharma companies can respond swiftly to emerging trends, adjusting their strategies to meet the dynamic needs of HCPs and patients alike.

Addressing the Challenges of Digital Transformation in Pharma

While the future holds immense promises, the journey to adoption is not without its hurdles. Some challenges that we see include:

Data Integration: Connecting and integrating diverse data sources is vital for a seamless flow of information across systems and departments. This enables organizations to better understand patient needs, market trends, and treatment outcomes, fostering innovation and personalized care.
Change Management: Adopting new technologies requires helping teams adapt to new workflows through training and effective communication. Overcoming resistance and fostering a culture of innovation is critical to success.
Regulatory Compliance: Organizations must navigate complex regulations while leveraging these innovative tools. Balancing compliance with innovation demands attention to legal, security, and privacy requirements.
Resource Allocation: Investments in technology must be balanced with other organizational priorities. Allocating funds, time, and personnel strategically ensures long-term value and successful implementation.

To overcome these challenges, companies must adopt a phased approach to implementation, focusing on incremental wins that build momentum and demonstrate value. Many of the abovementioned activities are often underestimated or overlooked. Digital Hive has the experience to tackle these transformations and support pharma companies with a clear implementation strategy.

The Future of Life Sciences

The impact of the changes that will come within the pharma industry beyond technological innovation - it’s about creating a more human-centric healthcare experience. By combining cutting-edge tools with a focus on empathy and collaboration, we can empower HCPs to focus on what truly matters: improving the lives of their patients.

As the industry embraces the future of Life Sciences, the question isn’t whether to adopt but how to adopt effectively. Companies must prioritize strategies that align technology with their broader mission and values.

**At Digital Hive we are very much looking forward to a new and exciting era of innovation, where systems are more digitally connected, data-driven, and patient-centric to drive better outcomes. If you're ready to embrace this exciting transformation, contact us today for an introduction and a complimentary discovery workshop. Together, we can help you navigate the future of life sciences with industry experience and clarity! **

The path to the dbt Developer Certification

Fri, 20 Dec 2024 00:00:00 GMT

**As you might have seen in our previous blog posts (go read them if you haven’t), dbt is quite an amazing tool. My experience with it started about a year and a half ago, when I stepped into the world of data engineering. Before that, I had worked for two and a half years with Python in software engineering. Seeing how dbt is bringing numerous software best practices into data engineering, I really loved it from the start. After gaining experience for over a year and seeing all the things dbt has to offer, I just had to dive deeper into it. The dbt developer certificate gave me the perfect opportunity and guidance to get familiar with all of its best features. In this blog, I will go through all the steps I believe you need to take, in order to pass your dbt Analytics Engineering Certification Exam. **

The study resources

Before starting to study, dbt recommends having over six months of experience with dbt Core or dbt Cloud. Although I believe you could pass the exam without this, it will save you a lot of study time. Check out their overview here: https://www.getdbt.com/certifications/analytics-engineer-certification-exam

What’s covered in the exam?

Looking at the page, my first tip would be to get familiar with the What’s covered in the exam? section. Check out which of these items you already have experience with, which you might just be familiar with, or which are completely new to you. Getting this division upfront will help you when diving into the Study guide section. I had already heard or read about most of the topics on the list, but I did not have experience yet with all of them. By identifying the lesser-known topics, I knew where to spend more time still.

The Study Guide

Next on the list is the Study Guide, a detailed overview of everything you need to know, what to study and where to find the right study materials. It starts off with some practical information on the exam format (duration, length, passing score…), which are important to know, but less content heavy. After that, the topic outline returns followed by some sample questions. These are not the best in my opinion, as there are only five questions so it cannot really be used to test yourself.

The Learning Path is definitely the most important section of this document. It will give you a list of checkpoints to work on, each with their own guidance on how to prepare. Here you can find what courses to take to get the right experience and which blog posts or documentation pages to read to get a deeper understanding of all the topics. I recommend going through all these links in detail, as you will learn a lot about some dbt features you might otherwise not think of. Each checkpoint will list the experience you should have at the end of it and which dbt commands you should now master.

Going through all this will take quite some time, but you will be ready to start testing your knowledge!

Test your knowledge

Unfortunately, there are no great practice exams delivered by dbt. So, I had to look further for a way to check if I had prepared enough. Luckily, dbt has a great community on Slack with a separate channel to talk about the certificate. In there, I saw a lot of people referencing qanalabs.com where I could find a course with no less than 13 (!!) practice exams, each consisting of 65 questions. These were still on the previous version of the exam, so I had to take that into account when doing these tests. As some topics, like dbt metrics, are no longer part of the certificate topics. Luckily, the qanalabs course has now been updated for the new version, and it now consists of 11 practice exams of 70 questions each.

I would like to make two sidenotes on the practice exams. The first one is small, but good to know. Just like in the actual exam, all questions are multiple choice. In some cases, you only have to select one answer and in some multiple answers might be true. In the qanalabs questions, that differentiation is not as clear as in the real questions. There were multiple questions where this cost me some points, but luckily it is much better during the exam. So, no need to get frustrated when running into this.

Secondly, you should know that qanalabs is in no way supported or authored by dbt itself. The course is maintained separately, and the author is active on the dbt slack community. This means that not all questions are perfect, and I have seen people disagreeing with some answers. Even though this can be frustrating, I believe they will give you clarity on where your gaps are and where you might need some more studying. And that is exactly what they are intended for according to the author.

Take the exam

When you are finally done studying and getting good scores on the practice tests, you are ready to take the exam and get certified. Back in the certification overview from dbt, you can just click through to schedule your exam. You will be monitored during the exam, so make sure you plan it when you will not be disturbed, and you can work in a silent, clean environment.

I planned my exam in the early afternoon, so that I could start my day with a relaxing coffee. I had some time to go over all my notes again and prepare my set up. I sat down and I was ready. The exam consists of 65 questions, and I feel like they were quite hard. But luckily, I prepared well and after submitting, I passed!

Of course, all this is just my experience in this challenge. I would like to list up the before mentioned resources to finish up, so you can really decide what will work best for you:

Dbt overview: https://www.getdbt.com/certifications/analytics-engineer-certification-exam
Qanalabs practice tests: https://www.qanalabs.com/courses/dbt-developer
Join the dbt community on Slack and checkout the #dbt-certification channel

Best of luck! And if you are in need of more dbt expertise, don’t hesitate to contact us for one of our consultants!

A Salesforce Health Cloud implementation for CAR-T Treatment

Wed, 11 Dec 2024 00:00:00 GMT

Challenge

A global leader in healthcare and pharmaceuticals encountered significant operational challenges in managing the intricate and crucial logistics associated with CAR-T Treatment. CAR-T cell therapy is an innovative cancer treatment that harnesses the body's immune system to fight cancer. By genetically modifying a patient's T cells to target specific cancer cells, CAR-T therapy offers a powerful and targeted approach to treatment which has shown remarkable results, curing 78% of treated patients.

This process involves collecting a patient's T cells, freezing and shipping them, and genetically modifying them to attack cancer cells. These cells are then multiplied and infused back into the patient's bloodstream, where they target and destroy cancer cells, offering long-term protection. Throughout the process, potential issues like quality deviations, shipment delays, or the need for re-apheresis must be managed efficiently. Ensuring smooth procedures, accurate validations, and a user-friendly system is crucial for the success of this life-saving treatment, especially for cancer patients relying on CAR-T therapy as a last resort.

The healthcare organization partnered with Digital Hive to deliver a comprehensive CRM solution. Initially focused on CRM development and business analysis, Digital Hive extended beyond the original scope of work to provide continuous improvements across various dimensions of the project. This proactive approach was aimed at enhancing overall project efficiency and fostering a long-term, trust-based partnership.

Solution

To address these challenges and improve patient care and operational efficiency, the team devised a comprehensive solution leveraging Salesforce Health Cloud. Here's how they tackled the complexities:

Business Analysis: Ongoing business analysis and support were provided, conducting follow-ups on demands, incidents, and requests to ensure effective issue resolution.
Salesforce Customization and Development: The development team customized the Salesforce Health Cloud platform to align with the organization's specific requirements, ensuring seamless integration and functionality tailored to the CAR-T treatment process.
Project Improvements: Leveraging project experience, the team consistently evaluated their approach to optimize project delivery and coordination from various angles. This was essential in a fast-paced environment with numerous stakeholders.
Technical Support: The development team provided ongoing technical support, troubleshooting any issues that arose during the implementation and post-deployment phases, thereby ensuring smooth operation and user satisfaction.
Continuous Improvement: With the entire development team onboard, the system was continuously enhanced based on user feedback, evolving requirements, and emerging technological advancements, ensuring ongoing optimization and effectiveness of the solution.

Results

Digital Hive successfully developed an intuitive Salesforce Health Cloud solution, leveraging best practices from the pharmaceutical industry and utilizing out-of-the-box Salesforce solutions and components. This approach not only ensured a robust and scalable CRM platform but also laid a solid foundation for future growth across the EMEA region. The solution is set to expand its usage from 3 countries to 6, with plans to further scale to over 20 countries over the next coming years, positively increasing patient impact and operational efficiency. Some key achievements include:

Intuitive User Interface: Developing an intuitive user interface using Salesforce Lightning capabilities ensured clarity for end-users regarding treatment progress and upcoming steps.
Enhanced Efficiency: Notable improvements in team efficiency due to proactive process optimizations and early detection of project dependencies.
Improved Collaboration: Strengthened alignment and communication across technical and business teams, leading to a more cohesive working environment.
Recognized Value: The healthcare organization acknowledged Digital Hive’s contributions, particularly appreciating the team's willingness to go beyond their initial assignment and foster a partnership grounded in trust and expertise.

Conclusion

Digital Hive’s commitment to continuous improvement and proactive collaboration significantly enhanced project outcomes. By focusing on long-term partnership goals and delivering beyond the initial requirements, Digital Hive demonstrated its added value as a strategic CRM consultancy partner, reinforcing trust and solidifying its reputation for CRM expertise and advisement.

Digital Hive CRM | Transforming Sales Processes in a Multilingual Environment with Microsoft Dynamics 365 at M

Wed, 11 Dec 2024 00:00:00 GMT

Challenge

Milexia is a French company which specialises in high tech electrical components, used in consumer electronics, medical and scientific devices, military and space exploration. Expanding their operations in the past years, Milexia faced some significant operational challenges:

The need for a unified Sales Process supported by a CRM system that could serve as a single source of truth from lead to cash.
Integrating operations across 5 countries with differing data models, ways of working, and legal requirements, while also preparing for future expansion and onboarding from acquisitions and strategic partnerships.
The absence of a CRM system in three of the five countries, compounded by the use of various ERP systems, created difficulties in group reporting and pipeline visibility. The need for one unified and consistent data model was crucial for effective cross-country analysis and strategic alignment.

Strategic CRM Selection and Planning

Digital Hive was onboarded in the early phases of the project. During the first phase, we assisted Milexia in making strategic CRM decisions that aligned with their long-term growth and operational goals. To achieve this, our consultant worked with the Milexia internal team on the following areas to gather their needs and building the RFP.

Analysis and Platform Selection

Conducting comprehensive business analysis sessions with Sales and Marketing departments to identify 41 critical sales functionalities and 26 marketing requirements that the CRM would need to fulfil.
Performing a benchmark analysis of leading CRM platforms (Salesforce, Microsoft Dynamics 365, and SAP CRM) to evaluate the best fit for Milexia’s needs. Similarly, multiple Marketing platforms were analysed (ClickDimensions, Lemlist, Hubspot)
As an outcome of our benchmark analysis, we conducted an in-depth examination of Microsoft Dynamics 365 to ensure it met all requirements before final platform selection.

RFP Development and Partner Selection

Creating a detailed Request for Proposal (RFP) based on insights gathered during the analysis phase, which was shared with potential MS Dynamics implementation partners.
Assisting Milexia in evaluating potential implementation partners and navigating the selection process to identify the most capable partner for their needs.

Implementation Oversight and Strategic Guidance

During the implementation phase of the project, Digital Hive’s role expanded to safeguarding Milexia’s interests and ensuring success by providing:

Stakeholder Management

Facilitating effective communication between Milexia’s internal teams and the external implementation partner. This ensured alignment and clarity across all project activities.

Escalation Management

Proactively addressing challenges and escalations, such as data migration complexities, by designing collaborative solutions with the support of Microsoft’s technical services.

Scope and Project Management

Maintaining oversight of project milestones, protecting the project timeline and budget, and ensuring that deliverables met quality standards and Milexia’s strategic objectives.

Results

The successful implementation of Microsoft Dynamics 365, integrated with ClickDimensions Marketing Automation is currently being rolled out in the current Milexia countries. This platform will allow Milexia to reach their goals and prepare for the future. Not only will there be a centralized and clear dashboard for sales pipeline management, but Milexia will also be ready to seamlessly onboard newly acquired companies and expand their operations with a consistent approach.

Digital Hive’s early engagement and comprehensive support ensured that Milexia selected the right CRM solution and protected their strategic interests throughout the implementation. By facilitating strong stakeholder communication, proactive escalation management, and disciplined scope management, Digital Hive demonstrated its value as a trusted strategic partner. This collaborative approach laid the groundwork for long-term operational success and scalability.

Leverage DBT audit to ensure accurate model generation

Fri, 15 Nov 2024 00:00:00 GMT

Introduction

In the evolving landscape of data analytics and engineering, efficient and reliable data transformation tools are crucial. dbt (data build tool) has emerged as a leading solution, offering capabilities that streamline data transformation workflows, enhance data quality, and support collaborative data practices. Many companies might want to start using DBT, but are stuck using an older, legacy sql based etl tool. In this blog we give an example of using the dbt audit helper as a way of checking the results when moving your legacy query to dbt.

Prerequisites

In any move between tools the ideal first step is to create an overview of the various pipelines and their logic. This overview will function as the backbone from which the development can commence. Make sure to document the sources and destinations from which the data is read and written to, and the logic consisting of SQL queries , joins, filters, and other transformations.

Set up the dbt environment

We won’t go over the setting up of a dbt-airflow environment in detail here as the documentation on this subject is widely available. However, the basic steps are:

Install dbt: Follow the official installation guide.
Initialize a dbt project: Run dbt init followed by the project name to create a new dbt project.
Configure profiles: Set up a profiles.yml file with the necessary database connection details.
Create the sources: Define the data sources in dbt/models/sources/source_name.yml.
Set up ingestion pipelines: Use Airflow DAGs to run custom Python ingestions or choose from the wide range of existing operators depending on the requirements.
Fill ingestion layer: Run the ingestion pipelines to populate the ingestion layer with source data.

Once there is data in the ingestion layer which should have been defined in the source data we can begin migrating the pipelines.

Note: The ingestion layer should contain copies of the source data.

Translate sql to dbt

To create the models that will replace the current transformations, first, organize the models depending on how many layers the project has (staging, intermediate, reporting). For file, create a file in the relevant layer folder under models (dbt/models/staging/transformation_name.sql). During the transition phase, it might be useful to introduce prefixes in the table names or different schemas, such that the data generated by the legacy SQL and the ones generated by dbt can coexist in the same layer for comparison.

Note: Use tags to group the models such that they can be run in grouped dbt builds.

Test model accuracy with an audit model

An important step in creating the models is testing whether the behavior of the model transformations results in data that is equal to the data generated by the legacy tool transformations. Here we suggest using an audit model to generate a comparison audit. An example of such a model is shown below. To generate this model we made use of the audit_helper library. https://hub.getdbt.com/dbt-labs/audit_helper/latest/

-- define relation of the dbt model table {% set a_relation = source('snowflake', 'pharmaceuticals_india') %} -- define relation of the legacy table {% set b_relation = source('snowflake2', 'pharmaceuticals_india') %} {% if execute %} {% set audit_query = audit_helper.compare_row_counts( a_relation=a_relation, b_relation=b_relation ) %} {% set audit_results = run_query(audit_query) %} {% do audit_results.print_table() %} {% endif %} {% if execute %} {% set audit_query2 = audit_helper.compare_relation_columns( a_relation=a_relation, b_relation=b_relation ) %} {% set audit_results2 = run_query(audit_query2) %} {% do audit_results2.print_table() %} {% endif %} {% if execute %} {% set audit_query3 = audit_helper.compare_relations( a_relation=a_relation, b_relation=b_relation ) %} {% set audit_results3 = run_query(audit_query3) %} {% do audit_results3.print_table() %} {% endif %} {% if execute %} {% set audit_query4 = audit_helper.compare_all_columns( a_relation=a_relation, b_relation=b_relation, primary_key = "ID", ) %} {% set audit_results4 = run_query(audit_query4) %} {% do audit_results4.print_table() %} {% endif %} select 0 as audit_test

In the first step we use the audit_helper.compare_row_counts here we count the rows in both tables and return the total count.

In the second step we use the audit_helper.compare_relation_columns which checks the order and datatype of the columns in both tables.

In the third step we use the audit_helper.compare_relations. This returns the percentage of matching relationships.

In the final step we use the audit_helper.compare_all_columns. This checks the matching rows in both tables and see how many NULL values each one has for every column, and how many each column is missing compared with the other table.

The “select 0 as audit_test” the end of the models acts as a placeholder query to avoid returning data while ensuring DBT's requirement of returning a select result from every model is met.

Once all models that replace the legacy tool transformations have been generated, the next step should be setting up tests for each model.

Demo

Here we have a db called db1 containing two schemas Schema1 and Schema1__legacy each of the schemas contains a table called “pharmaceuticals_india”. Below we can see the output of comparing two 100% equal tables.

To show the result of a failed audit we removed a column from the table in Schema1 the output is as follows.

Conclusion

The audit model we presented provides a comprehensive method to compare data between legacy systems and newly implemented DBT models, ensuring data consistency and accuracy throughout the transformation process. This approach allows users to easily identify discrepancies between datasets, whether they arise from differences in row counts, column structures, or data content. By utilizing the audit_helper library, these comparisons can be done with minimal overhead, making it a low-impact testing approach that efficiently validates the accuracy of new models without requiring excessive resources or introducing complexity.

One of the key advantages of this methodology is that it offers granular visibility into potential issues. From basic row count mismatches to more intricate column and data integrity checks, this process enables users to pinpoint the exact nature of discrepancies.

Ultimately, by leveraging DBT and audit_helper for model generation, organizations can achieve a smooth, risk-free transition from legacy ETL tools to modern, cloud-based data pipelines. This ensures that transformations are correct, scalable, and maintainable in the long term. Such an approach builds confidence in the migration process, enabling teams to embrace DBT’s benefits—like modularity, version control, and collaboration—without sacrificing the accuracy of critical business data.

Empty files not showing up in Snowflake external stage?

Wed, 16 Oct 2024 00:00:00 GMT

When working with Snowflake to load data from external stage zero-byte (0b) or 1-byte (1b) files are not accessible. In this blog post, we'll explore these challenges and outline potential solutions for handling such file ingestion problems.

The problem - Querying Files from an External Stage

While querying an external stage, zero-byte (0b) and one-byte (1b) files are not accessible through the standard SELECT query in Snowflake, creating an obstacle when that file's metadata is part of your dataset.

For files larger than 1b the content of the queried files can be accessed via the $1 column, and metadata about the files can be retrieved using several built-in metadata columns. These include:

METADATA$FILENAME: The full path to the data file the current row belongs to.
METADATA$FILE_ROW_NUMBER: The row number of the record within the data file.
METADATA$FILE_CONTENT_KEY: A checksum of the data file.
METADATA$FILE_LAST_MODIFIED: The last modified timestamp of the data file.
METADATA$START_SCAN_TIME: The timestamp when the scanning of the file started.

Here’s an example of such a query:

SELECT $1, METADATA$FILENAME, METADATA$FILE_ROW_NUMBER FROM @my_stage (FILE_FORMAT => my_custom_format, PATTERN => '.*.csv');

This query will work for files with content greater than 1 byte. However, files that are 0b or 1b in size** are not accessible** via the standard SELECT query.

Workaround - listing the files

A useful workaround is to query the results of the LIST stage command or use a SELECT from the stage's directory to access the metadata for these small files.

Here's how you can list the files in the stage and retrieve metadata:

LIST @my_stage; SET qid=LAST_QUERY_ID(); SELECT “name”,“size”, “md5”, “last_modified” FROM table(result_scan($qid)) ;

Alternatively, you can use SELECT with the DIRECTORY option:

SELECT * FROM DIRECTORY(@my_stage);

This approach allows you to access the file names, sizes, hash key, and last modified dates, even for small files that cannot be directly queried.

Workaround - adding metadata

If you want to combine metadata from 0b and 1b files with the data from larger files that can be queried, you can use a UNION ALL to merge the results. This approach ensures that all relevant file information is captured. Here's an example of how this can be achieved:

WITH large_files AS ( SELECT $1 AS FILE_CONTENT, METADATA$FILENAME, METADATA$FILE_ROW_NUMBER FROM @my_stage (FILE_FORMAT => my_custom_format, PATTERN => '.*.csv') ), small_files AS ( SELECT FILENAME, LAST_MODIFIED FROM DIRECTORY(@my_stage) WHERE FILE_SIZE <= 1 ) SELECT * FROM large_files UNION ALL SELECT null AS FILE_CONTENT, FILENAME,null AS FILE_ROW_NUMBER FROM small_files;

This query selects the data from files larger than 1b and combines it with metadata from smaller files. The same can be done using the LIST stage command. In that case you will need to replace the small files query with the LIST command and query form the query results provided above.

Additional context

Understanding which file types can potentially be 0b or 1b is crucial when dealing with external stages. Below is a breakdown of common file types and their potential to be zero or one byte in size:

CSV Files:

CSV files can technically be 1 byte if they contain an empty column represented by a single character such as a comma. 0b CSV file, however, would be invalid as it wouldn't contain even the structural elements (like delimiters).

JSON Files:

JSON files cannot be 1 byte because they require at least {} to form a valid empty JSON object.

A 0b JSON file is similarly invalid.

Text Files (TXT):

TXT files can be 1 byte and still valid, as they may contain a single character or a newline. A 0b TXT file, though, would be empty and contain no useful information.

Parquet Files:

Parquet files are a structured format and can't be 1 byte. These files have a header and metadata that make it impossible for them to be valid at such a small size. A 0b Parquet file would be considered corrupt or incomplete.

Avro Files:

Similar to Parquet, Avro is a structured format with metadata, and it cannot be valid if it's only 1 byte. 0b Avro files would also be invalid.

Conclusion

Handling file ingestion issues in Snowflake, especially with 0b and 1b files, can be challenging. However, by using workarounds like LIST commands and DIRECTORY queries, and understanding which file types can be zero or one byte, you can better manage the ingestion process. These solutions will help you ensure that no data—regardless of file size—is left behind when querying from external stages.

If you're working with file types that may be susceptible to these issues, it’s important to account for them during your ingestion workflows. Keep in mind the tips outlined here to handle file ingestion more effectively and avoid common pitfalls.

The Reality Check: How Effective is AI in CRM?

Wed, 16 Oct 2024 00:00:00 GMT

In recent years, the AI revolution has taken the business world by storm, with promises of transforming how we interact with technology. Nowhere is this more true than in the world of Customer Relationship Management (CRM) systems, where AI tools claim to streamline and automate complex processes. But is AI in CRM really living up to its promise, or are we still far from the dream of truly intelligent systems that work seamlessly with our businesses?

The AI Promise vs. Reality

When AI in CRM was first announced, expectations were high. People envisioned a system where you could simply ask the AI anything—whether it be customer data, sales performance, or forecasting—and it would interpret your query, comb through your entire data model, and deliver an insightful, context-aware response.

But the reality today is quite different. AI tools still rely heavily on predefined prompts and manual context feeding to generate useful answers. They don’t yet have the ability to independently interpret all your business processes, roles, or data models. For AI to work effectively within your CRM, every business case still needs to be carefully crafted. The correct prompts must be created and documented, and the right data must be provided in just the right way.

What We Were Expecting: Full AI Autonomy

The expectation was simple: you’d be able to ask the AI any question related to your business or customers, and it would know what to do. Whether it’s pulling data from various departments, predicting customer churn, or analyzing sales patterns, we imagined AI would become an integral, self-sufficient assistant. However, what we’ve seen so far is that AI still requires a lot of manual input to interpret even basic business queries.

In short, we’re still far from a point where AI in CRM can understand your business autonomously. It can respond to specific, pre-programmed prompts—but don’t expect it to “get” your entire business context on its own.

A Big Gap From What Was Promised

Another limitation is that AI systems don’t fully integrate with your entire data model. They lack the flexibility to automatically understand complex business processes or extract relevant information without significant human input. Instead, AI solutions in CRM are still reliant on structured, predefined queries and often fall short when it comes to handling the full scope of unstructured or complex data.

This gap is particularly noticeable in areas like lead scoring, service ticket triage, or customer journey analysis—where AI can only be as effective as the prompts and data it has been fed. The world imagined a future where AI could autonomously pull insights, but that dream still feels far away.

Why We’re Not There Yet

Despite all the advancements in AI, the reality is that CRM systems still demand a lot of preparation. Every business use case must be carefully constructed, documented, and implemented with prompts and boundaries. AI is powerful, but it’s only as good as the inputs it receives. If you want truly autonomous AI, we’re simply not there yet.

The complexity of data environments, the need for robust business logic, and the integration of industry-specific processes make it difficult for AI to function without significant customization. This means AI in CRM systems remains more of an assistant that can perform limited, pre-defined tasks rather than the fully autonomous, intelligent system that was promised.

A Bright Future Ahead?

However, while we’re not yet living in a world where AI in CRM is fully autonomous, there is reason to believe that things will improve. Players in the AI market are rapidly iterating on their solutions. As AI technology evolves, we’re likely to see improvements in contextual understanding and data interpretation, allowing for smarter interactions and more automated processes.

One promising development is the possibility of predefined prompts created by third parties, which could work across different industries and business environments. This could significantly reduce the setup time and effort required to implement AI in a CRM system, bringing us closer to the autonomous future we’re all hoping for.

Looking Ahead

The AI-driven CRM landscape is evolving, and while we may not yet be where we want to be, there are significant improvements on the horizon. AI has the potential to transform businesses, but the technology is still in its infancy in many ways. Over the next few months and years, we expect that AI tools will become more robust, more intuitive, and more capable of interpreting complex business data autonomously.

As we keep an eye on these developments, it will be fascinating to see how the market evolves and whether AI will finally live up to its immense potential in CRM.

If you’re looking to explore AI in CRM or need expert consultancy on how to implement these systems effectively, reach out to Digital Hive. We’re here to guide your business through the next wave of AI-driven innovation.

A comparison of Spark pools in Synapse and Fabric

Mon, 14 Oct 2024 00:00:00 GMT

In this blogpost we try to show the main features of each option. This might help to make the decision that best fits your needs.

Synapse is a Platform-as-a-service (PAAS) that combines former standalone Azure services including ADF, KQL(Azure data explorer), Data Lake, Apache spark, Azure SQL DW. The services are oriented around big data and data warehousing. It provides one experience for ingesting, transforming, managing and serving data to other azure services such as PowerBI and Azure Ml.

Microsoft Fabric is the latest of Microsoft SAAS, it encompasses the functionalities of Synapse and builds further on those to bring together with analytics and management tools such as PowerBI, Azure ML, Purview and the latest AI features via Copilot. Besides these, Fabric comes with a novel data storage solution, One Lake, which serves as the single source of truth for all Fabric services.

Capacities, SCUs, vCores, SKUs and Fs

Regardless of the process one wants to run, be it spark pools or any other form of compute instance, capacity is needed. To be able to compare how capacity is measured we need to understand what SCUs, vCores, SKUs and Fs are and how they relate to each other.

SCUs (Synapse Commit Units) are specific to Synapse Analytics. They represent a combination of CPU, memory, and I/O resources that can be purchased and work like credits.

vCores are used across various Azure services to measure compute power and represent a virtual CPU core.

SKUs (Stock Keeping Units) or Fs define a specific configuration of compute resources, including vCores and memory. In the end Fs are just a direct representation of CUs(Capacity Units). The following table might help set the relation between these terms.

Fabric

Reserved

In Fabric you can opt for reserving compute or pay- as- you- go. By resolving the compute you can save 41% on compute costs compared to pay-as-you-go. This also means that if you do not require compute for more than 60% of a month then it will always be cheaper to opt for a pay-as-you-go subscription.

SKU	Capacity unit (CU)	vCores	Pay-as-you-go	Reservation · ~41% savings
F 2	2	0.25	€0.407/hour	€0.242/hour
F 4	4	0.50	€0.814/hour	€0.484/hour
F 8	8	1	€1.628/hour	€0.968/hour
F 16	16	2	€3.256/hour	€1.936/hour
F 32	32	4	€6.511/hour	€3.872/hour
F 64	64	8	€13.021/hour	€7.743/hour
F 128	128	16	€26.041/hour	€15.485/hour
F 256	256	32	€52.081/hour	€30.970/hour
F 512	512	64	€104.162/hour	€61.939/hour
F 1024	1024	128	€208.323/hour	€123.878/hour
F 2048	2048	256	€416.646/hour	€247.756/hour

For the most up to date prices turn to the Fabric pricing page: https://azure.microsoft.com/en-us/pricing/details/microsoft-fabric/

Starter Pool

When you run a notebook without a configured Spark pool, it will default to the Spark configuration and runtime environment provided by Fabric. This means you won't be able to customize the Spark version, node size, or other configuration options.

Additionally, certain features such as automatic pausing, high concurrency, and concurrency limits may be unavailable or function differently without a configured Spark pool.

Custom Spark Pool

A custom Spark pool allows users to specify dependencies, size nodes, auto scale, automatic pause, and dynamically allocate executors based on Spark job requirements. When enabled, autoscaling acquires new nodes within the max node limit specified by the user and retires them after job execution. Dynamic allocation allocates an optimal number of executors based on the data volume for better performance.

Synapse

SCUs

In Synapse there is also a discount on the compute when it is reserved in the form of SCUs. By paying for compute in this way you can save up to 28% on compute costs. For example if one buys 5000 SCUs at the price of € 4346,22 the SCUs will be used as if they represent the currency with which you would pay in a pay as you go situation. However, since you acquired the SCUs at a lower rate this becomes cheaper.

Keep in mind that the SCUs expire after 12 months if they are not used.

The SCUs can be used for below Synapse services:

Azure Synapse Analytics Dedicated SQL Pool
Azure Synapse Analytics Managed VNET
Azure Synapse Analytics Pipelines
Azure Synapse Analytics Serverless SQL Pool
Azure Synapse Analytics Serverless Apache Spark Pool
Azure Synapse Analytics Data Flow - Basic
Azure Synapse Analytics Data Flow – Standard

Tier	SCUs	Discount %	Price
1	5000	6%	€4,346.218
2	10000	8%	€8,507.491
3	24000	11%	€19,752.174
4	60000	16%	€46,606.252
5	150000	22%	€108,193.084
6	360000	28%	€239,689.292

For more up to date prices turn to: Pricing - Azure Synapse Analytics | Microsoft Azure

Pay-as-you-go

The pay-as-you-go option is more expensive than the SCUs per compute. However this approach can be interesting in two situations. The first situation is when you are setting up a new project and you are still figuring out how much compute you are going to need for the processes that you run. The second situation is when you know you will consume less than 5000 SCUs a year. Since the bottom line of SCU purchases is 5000 you will end up not using the remaining SCUs.

Type	Price
Memory Optimized	€0.143 per vCore-hour
GPU accelerated (public preview)	€0.157 per vCore-hour

Options for creating a spark pool.

Node size · Memory optimized	Instances count	Price/hour	Price/month
Small (4 vCores / 32GB)	1	€0.57	€417.34
Medium (8 vCores / 64 GB)	1	€1.14	€834.69
Large (16 vCores / 128 GB)	1	€2.29	€1,669.37
XLarge (32 vCores / 256 GB)	1	€4.57	€3,338.75
XXLarge(64 vCores / 432 GB)	1	€9.15	€6,677.50

For the most up to date prices turn to the synapse pricing page: https://azure.microsoft.com/en-us/pricing/details/synapse-analytics/

The Pools

After having looked at the pricing we can zoom in on the features that distinguish the spark pools in Synapse from the ones in Fabric. Fabric Spark pools offer both Starter and Custom pool options. Synapse Spark pools, on the other hand, are exclusively Custom pools which requires some educated choices to be made regarding node sizes and scale depending on the jobs. Synapse also supports high concurrency and has a configurable auto pause feature, whereas Fabric's auto pause duration is fixed. Additionally, Fabric's Spark pools benefit from novel features like V-Order and Spark autotune, which are not available in Synapse.

Feature	Azure Synapse Spark	Fabric Spark
Spark Pool Types	Custom pool	Starter pool, Custom pool
Spark Versions (runtime)	2.4, 3.1, 3.2, 3.3, 3.4	3.3, 3.4, 3.5 (experimental)
Autoscaling	Yes	Yes
Dynamic Allocation of Executors	Yes, up to 200 nodes	Yes, based on capacity
Adjustable Node Sizes	Yes, 3-200 nodes	Yes, 1-based on capacity
Node Size Family	Memory Optimized, GPU Accelerated	Memory Optimized
Node Sizes	Small-XXXLarge	Small-XXLarge
Auto pause	Yes, customizable minimum 5 minutes	Yes, non customizable 2 minutes
High Concurrency	No	Yes
V-Order	No	Yes
Spark Autotune	No	Yes
Concurrency Limits	Fixed	Variable based on capacity
Multiple Spark Pools	Yes	Yes (environments)
Intelligent Cache	Yes	Yes
API/SDK Support	Yes	No
Primary Storage	ADLS Gen2	OneLake
Notebook Languages	Python, Scala, Spark SQL, R, .NET	Python, Scala, Spark SQL, R
Notebook Concurrency	No	Yes
Pipeline Activity Support	Yes	Yes
Built-in Scheduled Runs	No	Yes
Retry Policies	No	Yes

Conclusion

In the end choosing between Spark pools in Synapse and Fabric depends on the needs of your project and its cost considerations. Synapse is tailored for users who need a comprehensive, integrated analytics service with flexible compute options via SCUs. Fabric, with its enhanced features and unified storage solution, caters to those looking for an all-encompassing analytics platform that integrates advanced AI and management tools.

Both platforms offer significant advantages, and understanding their distinct features and pricing models can help you make an informed decision that aligns with your organizational goals and budget constraints. Since Fabric is newer, it is clear that this will always be the more expensive option compared to similar amounts of compute in Synapse. However using Fabric does leave you more ready for the future, since there will very little new development in Synapse, with the focus from Microsoft being fully on Fabric.

Still feeling unsure about your Microsoft setup? We can help! Contact us for advice about a new setup or to take a loot at your existing configurations and cloud costs.

How AgentForce Revolutionizes Business with AI

Thu, 19 Sep 2024 00:00:00 GMT

**In today’s rapidly evolving business landscape, AI is no longer just a buzzword; it’s becoming a game-changer for organizations worldwide. Salesforce’s latest innovation, AgentForce, is at the heart of this transformation, providing businesses with tools to enhance efficiency, improve customer engagement, and automate complex tasks. But what exactly makes AgentForce stand out, and how does it work? **

**In this post, we’ll dive deep into how AgentForce works, dissect the key components like the Atlas Reasoning Engine, AgentBuilder, and the AgentForce Partner Network, and explore real-world use cases and efficiency gains in areas such as sales, service, and marketing. **

Understanding How AgentForce Works

At the heart of AgentForce is the ability to create AI-driven autonomous agents designed to function seamlessly across business units. While Salesforce has already established itself as a leader with tools like Einstein GPT and Copilot, AgentForce represents the next leap into fully autonomous operations.

AgentForce combines data from across the Salesforce ecosystem to enable AI agents to autonomously complete tasks like responding to customer inquiries, managing workflows, or even making complex sales decisions. These agents can be programmed to align with specific business objectives, leveraging machine learning models and vast data reservoirs to make smarter, faster decisions than human teams could manage alone.

Key Components of AgentForce

The Atlas Reasoning Engine

AgentForce owes much of its power to the Atlas Reasoning Engine. This engine is designed to ingest large quantities of data, rapidly analyzing and interpreting this information to guide autonomous decision-making. The key feature of the Atlas Reasoning Engine is its ability to simulate human-like reasoning by learning from past interactions, customer behavior, and sales trends.

For example, in a customer service scenario, the Atlas Reasoning Engine allows AgentForce to predict customer needs based on past issues and behaviors, crafting responses that feel personalized, timely, and relevant. The engine improves over time, creating a continuous feedback loop where each interaction becomes a learning opportunity, making the agent smarter and more effective with every engagement.

AgentBuilder

AgentBuilder is the interface that enables businesses to construct, customize, and deploy their autonomous agents with ease. Much like building an app on Salesforce, AgentBuilder allows administrators to design agents that can tackle specific business problems. The interface is no-code, enabling non-technical users to design agent behaviors by dragging and dropping various action modules.

One exciting feature is that these agents can be tailored to meet the specific requirements of different teams across the organization. Marketing can build agents that handle campaign management and lead nurturing, while the sales team can create agents to prioritize leads and automate follow-ups. Flexibility is key, and AgentBuilder empowers businesses to start small with simple workflows and gradually expand the agents' roles as confidence in the system grows.

AgentForce Partner Network

In addition to the powerful tools within AgentForce itself, Salesforce has launched the AgentForce Partner Network to help companies maximize the platform’s potential. This ecosystem of partners provides additional plugins, integrations, and custom AI models that can extend the functionality of AgentForce in ways that suit industry-specific needs.

For example, healthcare organizations might use a partner-built agent to automate appointment scheduling and reminders, while retail companies may leverage an agent for handling personalized product recommendations and managing online inventories. The Partner Network ensures that businesses of all sizes and industries can benefit from AgentForce's versatility, regardless of their unique operational demands.

Five Key Attributes of an Agent

Building an autonomous agent using AgentForce isn’t just about automating tasks. Each agent comes with five critical attributes that make them highly effective at carrying out responsibilities:

Role: Define the agent’s core responsibilities—what tasks need to be done, and in what contexts?
Data: Determine what data the agent can access, including structured and unstructured data such as CRM records, call transcripts, and emails.
Actions: Specify what actions the agent can take. This can range from answering inquiries to executing complex business logic like running flows or calling APIs.
Guardrails: Set clear limitations on what the agent should not do, leveraging the Einstein Trust Layer to ensure compliance with data privacy and ethical guidelines.
Channel: Identify where the agent will operate, whether on customer-facing platforms like WhatsApp or internally via tools like Slack.

These attributes ensure that each agent is highly specialized for its role, making them far more capable than traditional chatbots.

Real-World Use Cases: AgentForce in Action

Sales Automation: An automotive company used AgentBuilder to create AI sales agents that track leads and autonomously execute follow-up emails, freeing up sales teams to focus on closing deals. This led to a 15% increase in lead conversions, thanks to more personalized and timely engagement.
Service Optimization: A telecommunications provider employed the Atlas Reasoning Engine to automatically triage incoming service tickets. The AI prioritized issues based on urgency and customer value, resulting in faster response times and improved customer satisfaction.
Marketing Personalization: Retailers are deploying AgentForce to analyze customer purchase behaviors, creating highly targeted promotions. By predicting product preferences and automating campaign launches, businesses have seen a 20% boost in click-through rates and revenue from AI-driven recommendations.

Efficiency Gains: Driving Business Growth with AgentForce

Sales: AI-driven follow-ups, lead scoring, and opportunity tracking eliminate much of the manual labor associated with sales pipelines. By autonomously managing routine tasks, AgentForce frees up sales reps to engage in more meaningful conversations with high-potential prospects.
Service: Automated ticketing and intelligent escalation systems ensure that customer issues are resolved more quickly, reducing wait times and boosting satisfaction. AI agents can even handle routine queries, letting human agents focus on more complex problems.
Marketing: Hyper-personalized campaigns, predictive insights, and automated content generation make it easier for marketers to reach the right customers with the right message at the right time. This reduces campaign costs and increases engagement metrics.

At Digital Hive, we are excited about the potential of AgentForce. Our experience with Data, CRM, and AI has shown us firsthand how these technologies can revolutionize business operations. Contact us to learn more about how AI, CRM and Agentforce can benefit your organization, so together we can embrace the future of Data, CRM and AI.

Tosca CRM implementation

Thu, 05 Sep 2024 00:00:00 GMT

Client Overview

Tosca is a global leader in reusable packaging and supply chain solutions, offering products like reusable plastic containers and pallets. The company is committed to reducing environmental impact and enhancing efficiency by eliminating waste across the supply chain. Serving diverse industries, Tosca helps customers transition from single-use packaging to sustainable, reusable alternatives, supporting their sustainability and operational goals.

Project Overview

As Tosca extended its operations in EMEA, it faced the challenge of harmonizing business processes and systems across various countries. This expansion necessitated a cohesive approach to unify the company's CRM solution and business processes.

Challenge

The implementation of the new CRM system by Digital Hive faced several key challenges, each requiring tailored strategies for resolution.

Diverse User Preferences presented a significant challenge, as each country within Tosca's EMEA operations had developed unique systems and established business processes. Adapting users accustomed to these specific workflows was addressed by Digital Hive through a flexible implementation strategy. This approach accommodated various work styles, ensuring that the CRM system met the specific needs of different regions while maintaining overall consistency.

Resistance to Change is often a challenge in global CRM implementations, particularly from long-standing users hesitant to modify their established processes. Convincing stakeholders of the new system's benefits during its early stages was crucial. Digital Hive tackled this by emphasizing a clear future vision and effectively communicating the advantages, thereby fostering a more positive outlook on the transition.

Increasing User Adoption also became a crucial focus point. Post-implementation, many users had difficulties navigating the new system, and new team members required additional training. Digital Hive addressed this by investing in comprehensive training programs and implementing a "train the trainer" approach. This strategy, along with extensive workshops, enabled key users to assist their peers, supporting a seamless onboarding process and enhancing user confidence across the organization.

Solution

To address these challenges, we structured our approach with the following components:

Key User Structure: Identifying and training key users in each region to act as local champions of the new system.
Train the Trainer Approach: Empowering selected individuals to train their colleagues, ensuring widespread knowledge dissemination.
Workshops with Countries: Conducting workshops tailored to each country’s specific needs and business processes.
Proper Release and Project Management: Launching the system in phases with clear project management guidelines to ensure smooth implementation.
Data Quality Team: Establishing a team dedicated to maintaining high data quality across the new CRM system.

Tosca's commitment to unifying EMEA operations goes beyond just implementing a CRM system – it's about empowering teams to work cohesively, leveraging a common framework. Our approach addresses the human aspect of a CRM implementation, understanding that change can be challenging. By showing adaptability, highlighting the benefits, and providing the necessary support, we've moved away from resistance into collaboration, ensuring a future where every user, regardless of their background, thrives with the Microsoft Dynamics CRM solution.

The project successfully unified Tosca’s CRM and sales processes across the EMEA region. Our structured approach not only facilitated a smooth transition but also enhanced collaboration and efficiency. By addressing the human aspect of change, we helped Tosca create a cohesive and supportive environment, enabling all users to thrive within the new system.

Conclusion

Digital Hive's partnership with Tosca in implementing Microsoft Dynamics CRM underscores our commitment to a tailored consulting approach that consider the unique needs of each client. We go beyond technology integration, focusing on change management and user support to ensure long-term success while leveraging system and industry best practices.

Embracing the Future with Salesforce’s Latest AI Innovations

Mon, 08 Jul 2024 00:00:00 GMT

At Digital Hive, we pride ourselves on our commitment to transparency, trust, and a use case-first approach. As a leading CRM consultancy, we are excited to share insights on the newest Salesforce capabilities that are set to revolutionize the way businesses operate. In this blog post, we will explore the integration of generative AI in CRM, the groundbreaking Salesforce Prompt Builder, the innovative Salesforce Co-Pilot, and provide best practices for implementation. Let’s dive in!

Generative AI and Its Role in CRM

Generative AI, a subset of artificial intelligence, refers to systems capable of creating content, such as text, images, and more, based on a set of inputs. In the context of CRM, generative AI has the potential to transform customer interactions, streamline processes, and enhance decision-making. By leveraging vast amounts of data, generative AI can provide personalized responses, generate insightful reports, and automate repetitive tasks, thereby increasing productivity and improving customer satisfaction.

The integration of generative AI in CRM means that businesses can now offer more personalized and engaging customer experiences. Imagine a scenario where AI drafts personalized emails for sales representatives, generates insightful analytics reports for managers, or even interacts with customers directly through chatbots, providing timely and accurate responses. The possibilities are endless and profoundly transformative.

Introducing Salesforce Prompt Builder

The Salesforce Prompt Builder is a revolutionary tool designed to help admins create, manage, and deploy customized prompts for generative AI within the Salesforce ecosystem. This no-code solution allows admins to craft reusable prompt templates that can dynamically incorporate CRM data, ensuring that AI-generated responses are contextually accurate and personalized.

Key Features of Prompt Builder:

Grounding Data

The first step in using Prompt Builder is to ground your prompts with relevant data from Salesforce or Data Cloud. This means connecting your prompt templates with the specific data fields and records that will provide context for the AI-generated responses. For example, a sales email prompt might pull in data about the recipient’s recent purchases, past interactions, and account status to generate a personalized and relevant message.

Template Management

Admins can create and manage different types of prompt templates tailored to various business needs. These include:

Sales Email Prompt Templates

Automatically create emails for users by incorporating grounded data. These prompts can be used in the Email Composer to draft personalized emails to Contacts, Leads, or Person Accounts.

Field Generation Prompt Templates

Populate a single field on a single record with a summary or description created by an LLM. This is useful for generating case summaries or status updates directly within Salesforce records.

Flex Prompt Templates

Create prompts that incorporate records from multiple objects simultaneously, allowing for complex data-driven prompts that can be used across various workflows and processes.

Testing and Refinement

Generative AI responses can vary, so it’s crucial to test and refine prompts to ensure they meet the desired quality. Prompt Builder provides tools for previewing and fine-tuning prompts using actual CRM data. Admins can specify which LLM model to use or bring their own model to further tailor the responses.

Secure Deployment

The Einstein Trust Layer ensures that all data used in prompts is securely masked, and sensitive information such as personally identifiable information (PII) remains protected. This layer guarantees that any data shared with the LLM is not stored or used for training purposes, maintaining high standards of data security and privacy.

Unveiling Salesforce Co-Pilot

Salesforce Co-Pilot is an advanced AI-driven assistant designed to work seamlessly within the Salesforce platform. Co-Pilot helps users navigate complex workflows, provides real-time insights, and automates routine tasks. By leveraging generative AI, Co-Pilot can draft emails, generate reports, and even suggest next best actions based on historical data and current context.

Benefits of Salesforce Co-Pilot

Enhanced Productivity

Co-Pilot automates repetitive tasks, allowing users to focus on high-value activities. For example, sales representatives can use Co-Pilot to draft follow-up emails after a meeting, freeing up time to focus on strategic planning and customer engagement.

Improved Accuracy

By generating precise and relevant content, Co-Pilot reduces the risk of errors and inconsistencies. This is particularly useful in scenarios like customer support, where accurate and timely information is crucial.

Seamless Integration

Co-Pilot works within the existing Salesforce environment, ensuring a smooth user experience. Whether it’s suggesting next best actions during a sales call or providing real-time insights on customer data, Co-Pilot integrates effortlessly into daily workflows.

Capabilities of Salesforce Co-Pilot

Email Drafting

Co-Pilot can draft personalized emails based on previous interactions, customer preferences, and contextual data. This ensures that every communication is relevant and tailored to the recipient.

Report Generation

Managers can rely on Co-Pilot to generate detailed reports, highlighting key metrics and insights. This helps in making data-driven decisions without the need for extensive manual data analysis.

Task Automation

Co-Pilot can automate routine tasks such as updating records, scheduling follow-up actions, and managing to-do lists, ensuring that nothing falls through the cracks.

Real-Time Recommendations

During customer interactions, Co-Pilot can suggest relevant products or solutions based on the customer’s history and current needs, enhancing the overall customer experience.

Best Practices for Implementing Salesforce’s AI Capabilities

To maximize the benefits of Salesforce’s AI innovations, consider the following best practices:

Define Clear Objectives

Establish clear goals for what you want to achieve with generative AI. Whether it’s improving customer service response times, enhancing sales email personalization, or automating routine tasks, having defined objectives will guide your implementation strategy.

Start with a Pilot

Begin with a small-scale pilot project to test the functionality and gather feedback. This approach allows for adjustments before a full-scale rollout. Identify a specific use case, such as automating follow-up emails for a particular sales team, and measure the impact on productivity and customer engagement.

Leverage Existing Data

Ensure your CRM data is clean, accurate, and up-to-date. High-quality data is crucial for generating meaningful AI responses. Conduct regular data audits and cleansing activities to maintain data integrity.

Involve Stakeholders

Engage key stakeholders from different departments to ensure the AI solutions meet the needs of various users. Regularly communicate the benefits and progress of the AI implementation to gain buy-in and support.

Monitor and Refine

Continuously monitor the performance of AI-driven tools and refine prompts and workflows based on user feedback and performance metrics. Use analytics to track the effectiveness of AI responses and make necessary adjustments to improve accuracy and relevance.

Ensure Data Security and Privacy

With the increasing reliance on AI, it’s vital to prioritize data security and privacy. Use Salesforce’s Einstein Trust Layer to mask sensitive data and ensure compliance with data protection regulations.

Provide Training and Support

Equip your team with the knowledge and skills needed to leverage AI tools effectively. Offer training sessions, create user guides, and provide ongoing support to ensure a smooth transition and maximize the benefits of AI integration.

Foster a Culture of Innovation:

Encourage experimentation and innovation within your organization. Allow teams to explore new ways of using AI to solve problems and improve processes. Create a feedback loop where users can share their experiences and suggestions for further enhancements.

Use Cases and Practical Examples

Sales Email Personalization

Imagine a sales team needing to send personalized follow-up emails to prospects. Using Salesforce Prompt Builder, admins can create a custom email prompt template that pulls in relevant data such as the prospect’s name, company, and recent interactions. The generative AI then drafts a personalized email, allowing sales representatives to send tailored messages quickly and efficiently. This not only saves time but also ensures that each email is relevant and engaging, increasing the likelihood of a positive respnse.

Customer Support Summaries

Customer support agents often need to summarize case details for internal reporting. With the Field Generation prompt template, agents can click a button next to a case field to generate a concise summary of all open cases related to an account. This automation saves time and ensures consistency in reporting. For example, an agent can quickly generate a summary of all ongoing support cases for a major client, providing a clear overview for management meetings or strategic discussions.

Product Recommendations

Using Salesforce Co-Pilot, a retail company can provide personalized product recommendations to customers based on their purchase history and preferences. For instance, when a customer contacts support, Co-Pilot can suggest relevant products or services, enhancing the customer experience and driving sales. A customer who recently purchased a new smartphone might receive recommendations for compatible accessories, extended warranties, or special offers on related products.

Task Automation in Sales

A sales manager can use Salesforce Co-Pilot to automate the task of updating lead statuses and scheduling follow-up actions. For instance, after a sales call, Co-Pilot can automatically update the lead status in Salesforce, schedule a follow-up email for two days later, and set a reminder for the sales representative to check in with the lead after a week.

Real-Time Insights for Marketing Campaigns

Marketing teams can use Salesforce Co-Pilot to gain real-time insights into the performance of their campaigns. By analyzing data from various sources, Co-Pilot can provide actionable recommendations on how to optimize campaigns for better results. For example, if a particular email campaign is underperforming, Co-Pilot might suggest adjusting the subject line, targeting a different audience segment, or sending the email at a different time.

Conclusion

Salesforce’s latest AI capabilities, including the Prompt Builder and Co-Pilot, are poised to transform the CRM landscape. By integrating generative AI, businesses can enhance productivity, improve customer interactions, and make data-driven decisions with ease. At Digital Hive, we are excited to help our clients harness these innovative tools to achieve their business goals. Embrace the future of CRM with Salesforce’s generative AI and take your customer relationship management to new heights.

For more information on how Digital Hive can assist you in implementing these cutting-edge solutions, contact us today!

Salesforce & IQVIA: Partnership to transform Life Science Engagement

Tue, 07 May 2024 00:00:00 GMT

**In a groundbreaking move, Salesforce and IQVIA have announced a significant expansion of their global partnership, aimed at accelerating the development of Life Sciences Cloud. This collaboration marks a pivotal moment in the evolution of customer engagement within the life sciences industry. **

At the heart of this partnership lies the integration of IQVIA’s Orchestrated Customer Engagement (OCE) innovations into Salesforce’s Life Sciences Cloud. This strategic amalgamation promises to deliver a single, trusted, end-to-end engagement platform that revolutionizes how life sciences organizations interact with healthcare professionals and patients alike.

The synergy between IQVIA’s expertise in data, analytics, and technology and Salesforce’s prowess in AI CRM sets the stage for a transformative journey. By leveraging IQVIA’s OCE platform, Salesforce’s Life Sciences Cloud aims to redefine customer engagement, offering unparalleled insights and personalized experiences to users across the globe.

This collaboration is not just about technology integration; it’s about empowering life sciences organizations to make informed decisions, drive innovation, and ultimately improve patient outcomes. Through this partnership, customers can expect enhanced capabilities, streamlined processes, and a deeper understanding of their target audiences. Key elements of the expanded partnership include:

IQVIA licensing the OCE CRM software to Salesforce.
Collaborative efforts to expedite the development of Life Sciences Cloud for enhanced customer engagement, expected to be available in 2025.
Continued support for IQVIA’s existing OCE customers, totaling nearly 400 across 130+ countries, until 2029.
Joint marketing initiatives to promote the new offering, demonstrating a seamless transition from IQVIA’s OCE to Salesforce’s advanced life sciences solution.

Bernd Haas, SVP and Head of Digital Products and Solutions at IQVIA, emphasizes the shared vision of bringing the best of both worlds to life sciences customers. He highlights the commitment to innovation across various healthcare verticals, ensuring that the partnership delivers tangible value at every step of the journey.

Frank Defesche, SVP and General Manager, Life Sciences at Salesforce, echoes this sentiment, underlining the transformative potential of the collaboration. He envisions a future where life sciences organizations have access to a unified, intelligent platform that enables orchestrated and personalized engagement with their customers.

As we embark on this exciting journey, we invite you to join us in shaping the future of life sciences engagement. Together, we’re redefining what’s possible in the world of customer engagement within the life science industry, one innovation at a time.

Stay tuned for further updates or reach out to our industry expert:

Migrating multiple legacy systems into a cloud data warehouse at Essent

Mon, 29 Apr 2024 00:00:00 GMT

Essent is among the biggest energy companies in the Netherlands. The energy industry has been evolving rapidly for years now. So, to keep up with all the changes they are innovating their data landscape by migrating data from older, outdated systems to a modern cloud platform to make data more accessible and efficient for the business. This comes with significant challenges, both technical and organizational.

Challenge

At Essent NL, the goal to migrate multiple legacy systems into a cloud data warehouse is driven by the organization's strategic move towards cloud technology adoption, compounded by end-of-life considerations for aging technologies. An additional driver is the commitment to implementing a data mesh approach, where teams take ownership of their own data, to create a more flexible and decentralized way of working. Which brings along a challenge of reorganizing and change management on its own.

Solution

We worked together with Essent to tackle their challenges with a focus on improving the Snowflake platform and standardizing the data transformation practices.

Streamlining data ingestion

To enable the decommissioning of the outdated systems, we were responsible for the data ingestion of multiple sources into Snowflake. With a focus on efficiency, we improved the ingestion process and developed a Python tool for code generation to speed up dbt development. The Snowflake ingestion team can now make new data available in less than a day.

Standardizing data transformation

To improve the data landscape, Essent has now adopted dbt as the default data transformation tool. This way the transformation is standardized, and the ownership and development of all transformations are decentralized with each team being responsible for their data. In order to get this implemented, we helped organize workshops and training sessions to get more teams on board with dbt. Thereby playing our role in the organizational challenge of moving to a data mesh approach.

Implementing the logical data model

Next to ingesting new data and maintaining the Snowflake environment, our team also implemented the logical data model. The DMO (Data Management Office) discusses all entities in this model with the right stakeholders, ensuring the presentation of the data follows the company definitions and standards, ready to gather new insights through dashboarding and analysis.

"Digital Hive helped Essent with a quality injection within the Essential Insights team. They brought on a Senior Data Engineer that seamlessly integrated within the team. Additionally, it was proposed to bring in a junior counterpart which expanded the capabilities of the team even more. Their collaborative efforts have added significant value to our data operations."

Anthony Roes

Product Owner Essential Insights

Results

Looking back at our project at Essent, we have empowered the organization to make better data-driven decisions. Our role in the ingestion process has significantly improved the time to make data available. The integration of dbt as the default transformation tool has standardized the way-of-working and decentralized the data transformation efforts, creating a strong culture of ownership and accountability across teams. These changes have successfully helped Essent forward in current challenges, ready to adapt for future innovations in the energy industry.

ITZU

Sun, 21 Apr 2024 00:00:00 GMT

Design and branding, content creation and distribution, and advertising.

Challenge

ITZU Group sought to strengthen its online presence through effective social media management and support for the upcoming Hura rebranding. The challenge was to enhance brand communication and create engaging materials for internal and external audiences.

Solution

Our tailored solutions for ITZU's social media and rebranding support included:

Social media management Managed social media accounts for brands like EAZER and ITZU Talent Solutions, creating engaging content and optimizing LinkedIn pages.
Graphic design and branding Contributed to internal communication through the design of various materials, including flyers and documents. Designed social media templates for the forthcoming Hura rebranding.
Strategic participation Engaged in strategic meetings to align social media efforts with overall business objectives.
Website enhancement (Drupal) Updating and refining the EAZER website, utilizing the Drupal platform. This included implementing changes to improve user experience.

Results

The collaboration with ITZU Group led to great achievements. Elevated social media presence with increased engagement and brand awareness. Successful internal communication through visually appealing materials. Contribution to the smooth rebranding process, ensuring a consistent and appealing brand image.

These accomplishments underscore our commitment to not only meeting specific goals but also being a reliable and flexible partner that adapts swiftly to the dynamic needs of our clients.

BestBuro

Sun, 21 Apr 2024 00:00:00 GMT

Design and branding, content creation and distribution, and advertising.

Challenge

At BestBuro, they believe in transforming knowledge into growth. Their mission is to enhance well-being and drive business success by teaching ifficient working practices. Through their innovative approach, BestBuro equips individuals and organizations with the tools they need to thrive in today's fast-paced world.

They wanted to elevate their overall marketing strategy and execution. The primary challenge extended beyond conventional marketing goals; it was to find a high-quality partner with comprehensive (digital) marketing expertise capable of flexibly adapting to ad hoc needs and dynamically evolving requirements.

Approach

Content creation and distribution:

Executing a comprehensive content strategy, social media posts, engaging blog content, and weekly/monthly newsletters.

Design and branding:

Utilizing creative tools like Adobe InDesign and Illustrator to design visually appealing training guides and Efficiency Planners, perfectly aligned with BestBuro's brand.

Advertising campaigns:

Launching targeted advertising campaigns to boost brand visibility, drive website traffic, and enhance conversions.

Results

The implementation of our consultancy services showcase adaptability:

Adaptability in task prioritization: Demonstrating the ability to prioritize tasks based on changing client needs, ensuring seamless collaboration and project execution.
Dynamic focus adjustment: Flexibly adjusting our focus to align with evolving client priorities, showcasing agility in meeting unique business requirements.
Client-centric approach: Prioritizing client satisfaction through a tailored and client-centric approach, ensuring our services align precisely with the unique marketing requirements of BestBuro.

BooSt

Sun, 21 Apr 2024 00:00:00 GMT

Challenge

BooSt Body & Skin, a skin improvement and weight loss center in Lummen, sought a comprehensive online presence to showcase its offerings. The challenge was to create a user-friendly website and seamless webshop, integrating diverse features for optimal user experience and efficient sales transactions.

Solution

We embarked on a journey to transform BooSt's digital landscape. Key solutions included:

End-to-end website development

Crafted a visually appealing website on Wix, ensuring a modern design aligned with BooSt's brand identity. Implemented a user-friendly navigation structure for easy exploration of services and products.

Integrated webshop (Ecwid)

Integrated their webshop in Ecwid, enabling BooSt to manage inventory, process orders, and provide a seamless shopping experience for their clients.

Mobile optimization

Ensured mobile responsiveness, catering to the growing number of users accessing the website and webshop via mobile devices.

Strategic branding

Aligned the website design with BooSt's brand, creating a cohesive and professional online presence.

Results

The initial impact showcases a website that not only meets BooSt's aesthetic aspirations but also establishes a strong foundation for customer engagement and satisfaction. The website now stands ready to welcome a broader audience seeking personalized solutions for their skin and body wellness needs.

The website serves as a central hub for customers to explore BooSt's services, book an appointment easily and make purchases effortlessly.

Embocraft CRM & Stock Management

Sun, 21 Apr 2024 00:00:00 GMT

Embocraft: Enhancing Sales Operations and Stock Management with Salesforce.

Challenge

Embocraft specializes in selling medical devices, particularly catheters. With a focus on supporting their sales team and streamlining operations, Embocraft identified several challenges:

Supporting Sales Team: Enhancing sales operations to better support the sales team in effectively managing customer relationships and driving revenue.
Comprehensive Customer Information: Developing a 360-degree database for healthcare professionals (HCPs), including their department, hospital affiliations, communication history, and activity tracking.
Stock Management: Improving stock management processes by tracking product inventory, monitoring stock levels at customer sites, conducting stock counts, and generating invoices for orders.

Solution

To address these challenges, Embocraft implemented Salesforce Professional with Sales Cloud, offering a robust solution to streamline sales operations and stock management. Key components of the solution include:

Salesforce Professional: Leveraging the Salesforce Professional edition, Embocraft gained access to powerful sales and CRM tools to optimize their sales processes.
Sales Cloud: Implementing Sales Cloud allowed Embocraft to track customer interactions, manage accounts and opportunities, and drive sales productivity.
HCP Database: Developing a comprehensive HCP database within Salesforce enabled Embocraft to gain insights into healthcare professionals' preferences, interests, and engagement history. This 360-degree view facilitates targeted communication and relationship management.
Stock Management Module: Implementing a custom stock management module within Salesforce allowed Embocraft to efficiently track product inventory, monitor stock levels at customer sites, conduct stock counts, and generate invoices seamlessly.

Results

The implementation of Salesforce Professional with Sales Cloud has yielded tangible results for Embocraft. With enhanced sales operations and a comprehensive HCP database, the sales team can better manage customer relationships, resulting in increased sales revenue. Additionally, the streamlined stock management module has improved inventory control and invoicing processes, leading to greater operational efficiency.

Abbvie

Tue, 16 Apr 2024 00:00:00 GMT

A Salesforce Health Cloud implementation within Immunology and Neuroscience at AbbVie.

Challenge

Within the Exceptional Patient Experience (EPE) division at AbbVie, there were multi)faceted challenges that could have significant implications on AbbVie's ability to deliver optimal patient care, maintain regulatory compliance, and uphold its position as a leader in the pharmaceutical industry. Some key issues included:

Expansion of Health Cloud for Neuroscience:

The expansion of Health Cloud for the Neuroscience department was crucial for advancing patient care and research in neurological treatments. However, the challenge lay in seamlessly integrating and scaling the functionality.

Launch and Expansion of the Mobile App in 2022 and 2023:

Successfully launching the mobile app in 2022 and subsequently expanding it in 2023 were critical for AbbVie's patient engagement strategy. The app aimed to provide patients with accessible and personalized healthcare resources, thereby fostering better adherence to treatment plans and improving overall patient outcomes.

Integration of AE/PQC with Trilogy:

The integration of Adverse Events (AE) and Product Quality Complaints (PQC) with Trilogy represented a crucial aspect of pharmacovigilance and quality management. A seamless integration was essential for efficient tracking, reporting, and resolution of adverse events and product quality concerns.

Addressing these challenges required a holistic approach that considered both technological solutions and their broader impact on business operations and patient outcomes.

Solution

To overcome these challenges and enhance patient care and operational efficiency, our team implemented a comprehensive solution through our expertise in Salesforce Health Cloud. The approach included:

Patient Onboarding and Care Plan Maintenance:

Nurses were tasked with enrolling patients into the system with detailed care plans, encompassing medications, pump configurations, dosages, and ongoing care plan phases.
Maintaining the continuity and accuracy of care plans for patients, particularly those on long-term and home-based treatments, required a systematic approach.

Business Analysis and Support:

Working closely with operational managers, we acted as a supportive business analyst, conducting follow-ups on demands, incidents, complaints, and requests.
Weekly calls with cross-functional teams ensured effective communication and issue resolution.

Training and Knowledge Transfer:

Conducted training sessions, for new business units and regions, to empower the client's team in managing and utilizing Salesforce Health Cloud effectively.

Technology Integration:

Addressing the integration question, we explored possibilities to integrate with other systems such as Veeva CRM, ensuring a seamless connection between different systems.
Other integrations included handling Adverse Events, Product Quality Complaints, and Computer Telephony Integration (CTI) for calls.

KPI Reporting:

Providing monthly and weekly Immunology and Mobile App Key Performance Indicator (KPI) reports to BTS was essential for monitoring and assessing the performance of the Salesforce Health Cloud system.

Results

Our efforts yielded remarkable results and bolstered AbbVie's patient engagement strategy. Patients gained access to personalized healthcare resources, fostering better treatment plan adherence and overall improved patient outcomes. The project achieved notable milestones such as:

Onboarding Health Cloud in 3 countries and the Mobile App in 2 new countries.
Four global product readiness releases enhancing overall system stability and functionality.
Activation of new products in Europe, covering 7 countries (JP, IL, DE, NL, AT, SE, FI).
Three major releases in 2023, further refining the system and addressing evolving needs.
Successful migration of AU's entire data to Health Cloud Neuroscience.

These accomplishments signify a substantial improvement in patient engagement, operational efficiency, and the overall digital experience for both healthcare providers and patients within the Salesforce Health Cloud framework, and allows AbbVie to remain a market leader within the pharmaceutical industry.

Belgisch Centrum voor Geleidehonden VZW

Tue, 16 Apr 2024 00:00:00 GMT

Salesforce configuration, low code, and custom development to enable Vets to follow up on the whole breeding cycle from measurement of progesterone till the last day the puppies stay in the breed.

Challenge

BCGs mission is to raise and train exceptional dogs that have the potential to become lifelong companions for visually impaired individuals. However, managing the breeding process efficiently while maintaining accurate records posed significant challenges. The unpredictability of puppy litter sizes compounded the complexity of capturing vital data such as nutrition, weight, vaccinations, and medical records. Recognizing the need for an innovative solution, we sought to streamline data management for our dedicated veterinarians so they could work on their tasks in a more efficient manner such as:

Breeding dogs and managing external veterinarians.
Monitoring the health and well-being of mother dogs.
Leverage the breeding module to ensure the health and readiness of puppies.
Placing dogs with visually impaired individuals.
Supporting individuals in their daily activities.

Solution

To address these challenges, we developed a custom solution on the Salesforce platform, leveraging low code and Lightning Web Components (LWC). Our solution focuses on enabling veterinarians to efficiently manage the entire breeding cycle, from initial measurements of progesterone to the care of new-born puppies. Key components of our solution include:

Custom Lightning Web Components:

We created custom Lightning web components to simplify data entry for various objects on a single page, allowing veterinarians to manage multiple dogs simultaneously.

Lightning Flows with Invocable Apex:

Following Salesforce best practices, we utilized Salesforce Flows for automation within the breeding module. In cases where Flows were insufficient, we implemented a hybrid solution combining Flows and Apex to ensure seamless automation.

Comprehensive Reporting:

Our system provides operational and management-level reports, offering real-time insights into breeding activities. These reports enable informed decision-making and facilitate smooth operational workflows.

What sets our solution apart is our well-thought-through to ensure that the efficiency of data input does not compromise the integrity of our data model. This not only guarantees the accuracy and reliability of the information but also allows for the creation of comprehensive reports across various organizational levels.

Results

Our solution has significantly improved the efficiency and effectiveness of the breeding process at BCG. By centralizing data management and automating repetitive tasks, veterinarians can now focus more on the well-being of the dogs. The seamless integration of Lightning Web Components and Flows has reduced administrative burdens and improved overall productivity. Additionally, our robust reporting capabilities provide valuable insights for both operational and strategic decision-making.

We continue to provide support and enhancements to the breeding module, ensuring it meets the evolving needs of BCG. With our solution in place, BCG can better target donors and individuals interested in dog training programs. While we considered Salesforce Non-profit Success Pack, our tailored solution proved more suitable for their specific requirements, demonstrating our commitment to out use-case first approach which delivers the highest business value for BCG, while choosing the right solutions.