Why trustworthy AI requires more than getting the answer right.
I’ve been thinking a lot lately about what makes an AI answer trustworthy.
Accuracy is the obvious place to start. If I ask a question about our data, I want the answer to be correct. I want the AI to understand what our organization means by an active donor, recognize that the same constituent may exist in several systems, use the appropriate time period, and calculate the metric correctly.
But as organizations begin using AI to ask increasingly complicated questions of their data, I’m starting to think accuracy is only part of the challenge. Sometimes no correct answer exists. The information may not exist. The question may ask the data to establish causation when it can only demonstrate correlation. Definitions may have changed halfway through the period we’re analyzing. Two populations may look comparable until we understand how we acquired them.
Sometimes the most accurate answer available is, “We don’t know yet.”
That isn’t necessarily a failure of intelligence. It may be evidence of it.
The Smartest Analyst in the Room
Think about the best data analysts you’ve worked with over the years. They really understand your organization, its constituents, and the sometimes peculiar ways information moves through its systems.
They rarely answer every question immediately with absolute certainty. They’re the ones who ask whether you’re looking at calendar year or fiscal year. They remember that the acquisition strategy changed three years ago, or that a particular source has always been a little messy. They know which numbers they trust, which ones require explanation, and which questions simply cannot be answered with the information available.
Most importantly, they understand that a technically correct calculation can still produce a misleading conclusion.
Consider a question nearly every nonprofit organization asks: Why did donor retention decline?
An AI assistant with access to a well-structured constituent data platform might determine that retention declined from 68% to 64%. It might find that the decline concentrated among first-year donors, or that certain acquisition channels changed significantly. Those findings are useful and narrow the investigation considerably.
But they don’t necessarily explain why retention declined.
Perhaps the organization dramatically expanded acquisition through a channel that historically produces lower first-year retention. Or an unusually successful emergency appeal attracted donors who were less likely to give again. Maybe changes in cultivation contributed to the decline. The data may help us evaluate those explanations, but they can’t definitively establish which one caused the result.
An experienced analyst understands the difference between identifying a pattern and explaining its cause. Rather than presenting a single, definitive explanation, that person might suggest additional comparisons, examine changes in donor composition, or ask whether the decline was actually unexpected given the organization’s strategy.
That’s the kind of analytical judgment we’ve historically depended on people to provide. As we increasingly turn to AI for analytical assistance, shouldn’t we expect something similar?
In From Data to Decisions: We Have the Dashboard. Now What?, I argued that analytics’ real value often comes from knowing how to follow an answer to a better question. AI makes that process faster and more accessible, but it doesn’t eliminate the need for judgment. If anything, AI’s ability to produce polished, authoritative-sounding explanations makes that judgment more important.
Confidence is not the same thing as certainty. And neither is the same thing as truth.
Trustworthy AI Needs to Understand Its Limitations
This distinction between producing an answer and understanding its limitations isn’t unique to nonprofit analytics. The National Institute of Standards and Technology addresses it more broadly in its AI Risk Management Framework, which identifies reliability, transparency, explainability, and interpretability among the characteristics of trustworthy AI. Importantly, NIST recognizes that trustworthiness depends on the context in which a system is being used and the human judgment involved in evaluating its outputs.
I think that’s an important consideration as organizations begin using AI to support increasingly consequential decisions. A mathematically correct answer isn’t necessarily a useful answer if it relies on an inappropriate comparison, an outdated definition, or an assumption the underlying data cannot support.
Imagine asking an AI assistant which acquisition channel produces the most valuable donors. The system might calculate lifetime giving by source and identify a clear winner. But what if one channel has been operating for fifteen years while another was introduced eighteen months ago? What if the organization changed its attribution methodology? What if one channel intentionally prioritizes younger donors whose giving potential may develop over decades?
None of those complications necessarily prevents a useful analysis. They do, however, affect what conclusions we should draw. The best response might account for donor tenure, explain the attribution methodology, and identify where the available evidence remains incomplete. It might even begin by asking what the organization means by valuable: cumulative revenue, projected lifetime value, retention, engagement, or something else?
That isn’t an AI assistant being difficult. It’s an assistant recognizing that the question contains assumptions worth examining. The goal isn’t to make AI perpetually hesitant. It’s to help it recognize when uncertainty actually matters to the decision someone is trying to make.
The Data Foundation Has to Help AI Recognize What It Doesn’t Know
One of the more interesting developments in conversational analytics is how much attention is shifting from the model’s intelligence to the information and context available to it. Anthropic’s How Anthropic Enables Self-Service Data Analytics with Claude describes the challenges it encountered building AI-powered analytical capabilities for its own organization.
Anthropic identifies three major sources of inaccurate analytical responses: ambiguity about which data represents the correct concept or metric, information becoming stale as systems and definitions change, and failures to retrieve the appropriate information even when it exists. Its conclusion is particularly relevant to organizations investing in AI: strong data foundations, maintained sources of truth, and reliable ways to retrieve the right information are essential to producing trustworthy analytical results.
That should sound familiar to anyone who has worked with nonprofit constituent data. An organization may have several definitions of an active donor, multiple representations of the same constituent, and years of historical information collected under changing business rules. An AI assistant might select the right field, perform a perfectly correct calculation, and still answer the wrong question because it misunderstood the organization’s question.
In Karen Taggart’s AI Requires a Strong Data Foundation, we explored why something like the Unite Analytics Common Data Model, identity resolution, and shared definitions matter for making organizational information useful for AI. But another dimension of that argument deserves more attention: a strong data foundation shouldn’t merely help AI find the right answer. It should help establish what the available information can reasonably tell us.
Consider an organization that changed its definition of an active sustainer halfway through a reporting period. A well-governed analytical environment should preserve enough historical context to make that change visible. An AI assistant might still calculate the numbers for both periods, but it should recognize that comparing them without qualification could be misleading.
The same principle applies to identity resolution, incomplete engagement histories, inconsistent campaign attribution, or data sources that haven’t been updated recently. These aren’t necessarily signs of bad data. They’re characteristics of real organizational information that people need to understand before making decisions. As I explored in Your Data Remembers the Past. Does It Understand the Present?, preserving what was historically true is only part of building a useful data foundation. We also need to understand how that history relates to the present.
But even a well-governed data foundation can’t answer every question. It can help an AI assistant recognize known limitations, identify missing information, and avoid inappropriate comparisons. It cannot manufacture evidence that was never collected or establish causation simply because someone asked why something happened.
Making Better Judgment Part of the Experience
This is becoming particularly relevant as organizations begin interacting with their data through conversational tools like Claude, ChatGPT, and Microsoft Copilot.
At ROI Solutions, we’re already seeing clients use Claude with data managed through Unite Analytics, and other AI interfaces offer similar capabilities. Snowflake provides the underlying data platform, while technologies such as Model Context Protocol (MCP) create new ways for AI assistants to interact with that information.
The experience is changing considerably. Instead of navigating dashboards, requesting reports, or writing SQL, someone can ask a question in plain language and keep exploring as new questions arise. That’s an exciting development, particularly for organizations where access to sophisticated analytics has historically depended on a relatively small group of technical specialists.
But removing the technical barriers to asking questions doesn’t eliminate the analytical complexity of answering them. Someone who has never worked directly with the underlying data model may not know that two fields represent different definitions of revenue. They may not realize that a particular source stopped updating several months ago or that an apparently straightforward comparison involves populations with very different characteristics.
A useful conversational assistant should help people navigate those complications rather than conceal them behind a friendly interface. Sometimes that means asking for clarification before beginning an analysis. Sometimes it means explaining that a conclusion depends on a particular definition or assumption. And sometimes it means identifying what additional information would be needed to answer the question with greater confidence.
None of this suggests that AI should replace the judgment of experienced analysts or organizational leaders. Quite the opposite. The opportunity is to make some of that analytical discipline more accessible, helping more people ask better questions and understand the evidence behind the answers they receive.
I suspect this will become one of the more important measures of progress as conversational analytics matures. We’ve spent a great deal of time evaluating how effectively AI can retrieve information, perform calculations, and explain results. Those capabilities matter, but so does the ability to recognize when the evidence doesn’t support the conclusion someone is looking for.
The people I trust most aren’t the ones who claim to know everything. They’re the ones who understand their subject deeply enough to recognize its uncertainties, explain those uncertainties clearly, and help everyone else decide what to investigate next. As AI becomes a more familiar participant in nonprofit analytics, I think we should expect the same.
Your AI should know what it doesn’t know. And it should be smart enough to tell you.
Is your organization using AI, and do you trust the results you’re seeing? Let’s Talk!
Key Takeaways
- Accuracy is essential for trustworthy AI analytics, but it’s not the only factor; sometimes, no correct answers exist.
- Experienced analysts question assumptions, recognize data limitations, and distinguish correlation from causation.
- AI should understand its limitations; reliable foundations promote meaningful analysis rather than false certainty.
- Conversational AI tools improve data accessibility but require users to navigate the complexities and assumptions behind their answers.
- Trustworthy AI analytics should support better judgment by acknowledging uncertainty and guiding users in their inquiries.