What we learned building Data Doula, our analytics agent

In early 2025, we knew our approach to analytics wouldn't scale. On the surface, it seemed like we were well-equipped: we had dashboards for our core business questions; our users were savvy and knew how to use our BI tools to self-serve; we had good lines of communication so analysts could answer questions not covered by dashboards. However, we saw that more and more questions were falling into that last bucket.

If someone had a question off the paved path, an analyst had to write a query or modify an existing dashboard. As Babylist grew, demand for answers grew faster than our team could supply them. Increasingly, stakeholders waited while analysts handled one-off requests instead of deep analysis, and engineers got pulled from platform work to support those one-off requests with new columns or tables. Ad hoc tickets started dominating our backlog.

Data Doula was our answer. It's an analytics agent that people can talk to in Slack, which is already the center of work at Babylist. It's also available through a Model Context Protocol (MCP) server for people who work in AI chat interfaces such as Claude.

The chat interface makes Data Doula accessible, but it doesn't automatically make answers useful or correct. For that, we had to engineer constraints and tooling.

Supporting new questions

Let's recap where we started. We'd already built a few dashboards used for a broad set of company KPIs, and they supported lots of exploration through the use of well-designed filters and drill-downs. But we still had to encode every aggregation or filter in advance. Questions like "Which channels drove the increase in purchasing activity among new registrants?" only worked if we'd already built every part of that question into the dashboard:

• Channels
• Purchasing activity
• User signup
• The relationships between all of those

We already had the data, but if the dashboard designer didn't anticipate a question, that question was effectively unanswerable until an analyst picked up a ticket.

Before AI, our default approach would have been to keep adding filters and columns, but each addition only expanded the set of expected questions. It didn't help with the next unanticipated question. We needed a way for someone to ask completely new questions without pinging an analyst.

Bounded agency

So how could we handle unanticipated questions? Our first idea was simply to point an agent at our data warehouse. After all, agents are very good at SQL.

It turns out that writing SQL is only part of the problem. We saw this during testing when two agent sessions produced different definitions of "active user," and both were plausible but wrong. The agent could write SQL that returned data, but left to its own devices, it couldn't always choose the right joins and metrics. We were sometimes able to use prompts to get the model back on track, but relying on prompts for every edge case would just be another flavor of the "just add another filter" problem we already had. So we adjusted the approach, using bounded tools:

1. Someone asks Data Doula a question in Slack or Claude.
2. The agent makes a tool call: it uses the prompt to choose from a list of semantic models, each of which describes a specific business domain.
3. The tool call runs a SQL query that's constrained to the set of tables, columns, metrics, filters, and relationships specified in the chosen semantic model.

This is an application of bounded agency - each tool (semantic model) exposes a defined subset of the data warehouse. The agent can choose among those tools, but it can't query arbitrary tables or invent relationships outside the chosen model.

The alternative is what we tried at the start: one general-purpose query tool with access to the entire data warehouse. We found that more narrowly scoped tools keep queries tied to trusted definitions, reducing the chance of a plausible but irrelevant answer. Those boundaries also make the system easier to inspect and maintain; if Data Doula can't answer a question, the agent traces tell us whether the problem was related to reasoning or the underlying tool - for example, overlap between semantic models, inconsistent definitions, lack of conceptual coverage, or simply an issue in the data model. This lets us zero in on the appropriate fix more quickly.

The tradeoff is that there will always be a "frontier" of completely novel questions that Data Doula is unable to answer because they're not captured in a semantic model. This tradeoff is acceptable to us - we'd rather have a correct agent that says "I don't know" than a more flexible agent that invents answers.

Maintaining correctness

I described the high-level tool architecture above: each tool is a semantic model, which is a slice of our semantic layer. But what is a semantic layer?

A semantic layer gives business meaning to data by defining how business concepts map to modeled data, including how they relate to each other. This includes join rules, metric definitions, column descriptions, synonyms for column names, common filters, and more.

This information helps keep LLMs from choosing the wrong definitions - or worse, hallucinating. When combined with an LLM's reasoning capabilities, the agent can take a term like "orders" or "active users" and identify which metric the user means, how it's calculated, which date to use, and how it relates to the other concepts or entities the user mentioned in their question. This means agents start from trusted definitions instead of raw tables, making answers more deterministic. In analytics, determinism is especially important because incorrect results can violate trust and create churn.

Maintaining these definitions can become laborious, but we already had much of this knowledge in dbt. Our models and their metadata describe the data our team maintains and its relationship to Babylist's world. Rather than create a separate copy for Data Doula, we built a small internal generator that reused dbt metadata to maintain the semantic models available to the agent. Reusing dbt metadata also makes the semantic layer composable. Registrant behavior matters across much of our business, so we define those concepts once. A narrower domain such as email marketing can reuse them and add its own metrics and filters (e.g. email clickthrough rate; email subscribers) without documenting registrant behavior again.

Extending our dbt metadata meant that analysts and engineers could update definitions through our existing dbt code review workflow instead of rebuilding the semantic layer in a separate system. This helped us launch a v0 in days without writing a new semantic model from scratch.

Feedback loops

Giving Data Doula constraints made its answers more reliable, which drove adoption. As the number of users rose, we observed people using the agent in ways we hadn't expected.

• For one time-sensitive request, a stakeholder used Data Doula for the initial question. The answer didn't quite match the underlying intent, but because the answer was in a shared Slack thread, the analyst supporting that team was able to inspect the query, extend the logic for edge cases, and complete the analysis.
• During one production incident, the on-call engineers used Data Doula to quickly determine the number of users impacted. As the incident progressed, they continued that thread to slice that population in different ways, such as grouping by when they signed up.
• "Dormant" teams that weren't using our BI tools showed up on our user activity leaderboards. These teams asked questions we had never anticipated - for example, one team wanted to know how many discontinued products were being added from guides, which would indicate that a guide needs to be updated.

This "field anthropology" is valuable product feedback. Observing real user behavior directly led us to improve how data tables are rendered or attached and how Data Doula expresses the assumptions it made. Unanswered questions also feed our semantic layer backlog - they represent gaps in the agent's knowledge. We could see user behavior because we put Data Doula in Slack and Claude - where our users already work - instead of a custom web UI. Slack also made analytics multiplayer, since everyone can participate in a shared thread.

Observing live threads is useful, but we also instrumented the agent at all layers: the agent itself emits telemetry and the Slack app logs interactions and feedback. Detailed telemetry has helped us monitor the agent's performance at scale without manually observing individual threads. For example, we look at adoption rates by team, track which semantic models have the most negative feedback, and use agents to analyze Data Doula's traces to categorize knowledge gaps.

From prototype to adoption

Building the agent aws only part of the work - we also had to get people to use it.

Data Doula actually started during one of our annual company hackathons, exploring whether an agent could help. The prototype won a prize - in fact, it was so popular that the prototype nearly crashed from users trying it out while we were demoing the app. The response convinced us to invest in taking it to production.

It's common for data projects to get stuck in a long cycle of data modeling without tangible results. To avoid this, we focused on depth over breadth: we dove deep on building the semantic model around registrant behavior, since it's a well-understood - but complex - domain relevant to the entire company. This helped us stay focused and deliver a high quality experience; instead of answering all questions poorly, we could answer a highly relevant slice of questions accurately.

We also pushed hard for people to use Data Doula in public channels, as more usage meant more data for our feedback loop. To reinforce this ask, our team made a habit of tagging Data Doula ourselves - especially when someone tagged an analyst for a question we knew Data Doula could answer.

What we learned

• Use the assets you already have. Reuse existing metadata and context so definitions stay reviewable, composable, and less likely to drift.
• Give the agent explicit, bounded tools. Scope each tool to a clear purpose and domain so you can trace failures to a tool, definition, or coverage gap.
• Stay close to your users. Data teams already work with the people who depend on their work, so use those interactions to find new use cases and knowledge gaps.
• Go to where the users live. Distribution changes whether people use a tool.
• Depth over breadth. For data, where trust is key, choose accuracy over a widely relevant domain over shallow but broad coverage.
• Be a user. Use the product yourself to teach by example.

What's next

We're building pipelines that use agent traces to automatically propose updates to the semantic layer. Repeated failures and knowledge gaps can become concrete proposals for the data team to review. We're also integrating a context layer to give Data Doula more relevant information before it chooses a tool and forms an answer.