You don't trust AI in therapy? Good.

Neither do we, when it pretends to be certain.

There’s one sentence that comes up in conversations with therapists every single time the subject of artificial intelligence in the consulting room appears: “I won’t trust something I don’t understand.”

This isn’t technophobia. It’s a healthy clinical instinct. A therapist is accountable for every hypothesis they form about a patient - so they have the right, and indeed the duty, to know where that hypothesis comes from.

Yet most AI tools behave like a black box: you feed in the data, a finished result comes out, and the path from one to the other stays hidden - buried in millions of parameters no one can inspect. In a weather app, that’s no problem. In work with another human being, it is.

There’s also a second, more dangerous trap. A language model that hits a gap in the data has a strong tendency to fill it - to add what fits, what’s plausible, what looks like the textbook picture. That’s exactly what a hallucination is. And in a clinical context it’s more dangerous than silence, because it sounds credible.

So the starting point is simple: an AI tool in therapy doesn’t earn trust because it’s confident. It earns it when it’s honest - it shows the source, shows the reasoning, and admits what it doesn’t know.

This is easiest to see in a concrete case. The map opening this article is a fragment the system generated from a session transcript - let’s walk through its sections.

The system doesn’t invent the patient’s thoughts - it quotes them

In the automatic thoughts section there’s no “interpretation from the AI” anywhere. What appears are the patient’s verbatim statements from the transcript: “I feel like I don’t belong here”, “This is going to end badly. They’ll reject me”, “Some kind of flop”. Underneath, in the AI Rationale field, the system explains why it assigned those thoughts to a given category and which cognitive distortions might lie behind them.

This is the opposite of a black box. Instead of a verdict from nowhere, the therapist gets a message: here’s where I got this - check it yourself. Verification takes seconds, because the source is right there within reach.

It’s not a verdict - it’s a hypothesis

Next to every section sits one small but crucial word: (hypothesis). The system doesn’t announce: “this is the trigger”. It says: this is probably the trigger - verify it.

The same tone returns in the rationales: “appears to”, “may suggest”, “requires further clinical verification”. This isn’t linguistic caution tacked on at the end. It’s a built-in stance - epistemic humility written into the very way the system frames its conclusions. The decision always stays on the therapist’s side. The human never drops out of the loop, not for a moment.

And now the most important part: what the system does when it doesn’t know something

Let’s look at the physiology and somatic reactions section.

Anyone who works with anxiety knows the typical picture: muscle tension, racing heart, sweating, shallow breathing. A language model could simply add that - it would fit perfectly, it would sound credible, no one would notice.

The system doesn’t do that. It writes: “No data in transcript”.

And then it takes a step that changes everything. In the rationale it doesn’t just acknowledge the gap, it turns it into a clinical cue: it notes that the patient described “stress” but gave no specific bodily reactions, and suggests that a fuller ABC would call for asking what was happening in the body at the moment of entering and while standing beside the conversation.

This is exactly the moment where an ordinary model would hallucinate. ATS instead says “I don’t know” - and shows the therapist where the transcript ends and their own work begins.

The same precision shows up in the emotions section: where the patient gave a number (“stress 90/100”), the system records it. Where she didn’t - with sadness and grief - it states plainly: “intensity: not provided”. It distinguishes what was measured from what wasn’t. It doesn’t fake completeness.

One stance, not three features

Source. Hypothesis. Limit.

These aren’t three separate buttons bolted onto the product. They’re one consistent way the system approaches clinical material in the first place: it shows where it knows what it knows, it treats its conclusions as hypotheses to be verified, and it openly flags the places where data was missing.

A black box works the other way around - it hides the path and fills the gaps in order to look certain. In the consulting room, there’s no consent for that.

Skepticism toward AI in therapy is justified. So the point isn’t to talk you into trusting AI - it’s for the tool to prove, on its own, that it can be checked. Trust doesn’t come from confidence. It comes from honesty.

The Therapy Support Team