What we measure when we say "AI cites your firm"

TL;DR

We probed the same five NYC divorce firms with two methodologies — keyword-shape SEO queries and conversation-state buyer queries — across four LLM providers (OpenAI, Anthropic, Google, Perplexity). One firm, referred to below as Firm E, was cited on 37.5% of keyword-shape probes and 0% of conversation-state probes. Same firm. Same providers. Same pipeline. Same week.

That 37-percentage-point swing is not a measurement error. It is the construct-validity problem at the heart of every AI-visibility tool on the market right now: the choice of prompt shape silently determines the answer. A firm can be "highly cited by AI" or "invisible to AI" depending only on which kind of question you ask — and the industry has not been clear about which kind it asks.

Why this matters

A growing number of brands (and the agencies pitching them) are buying tools that claim to measure how often generative AI assistants recommend the brand. The pitch is simple: SEO is fading, AI is rising, and you need to know whether ChatGPT cites you.

The problem is that "cites you" is doing all the work in that sentence. Two methodologies can both honestly call themselves "AI citation audits" while measuring entirely different things — and producing entirely different verdicts about the same firm.

We discovered this empirically. Our own audit pipeline produced near-zero citation rates for firms with strong public reputations. The temptation was to conclude the firms were genuinely invisible. The reality was that our prompt shape and our public leaderboard's prompt shape were both biased in opposite directions, and the resulting numbers measured different constructs.

The two methodologies in plain English

Keyword-shape probes

These mirror the way buyers used to type into Google.

"best divorce attorney NYC"
"high net worth divorce lawyer Manhattan"
"top family law firm New York"

Short. Noun-driven. Optimized for SEO. The firm answering them well is the firm with the cleanest practice pages, the strongest backlinks, and the most matrimonial-specific landing pages.

Conversation-state probes

These mirror the way buyers actually type into ChatGPT in 2026.

"My husband filed for divorce last week and we have two kids and a house. I'm scared. Where do I even start?"

"I'm a surgeon. My wife is asking for half my future earnings. Who handles physician divorces in Manhattan?"

"We've been separated for six months and now I just found out he's been hiding crypto. What do I do?"

Long. Situational. Emotionally loaded. The firm answering these well is the firm whose website speaks the buyer's crisis back to them — not credentials, but situational language: "for surgeons facing income-shielding claims," "for parents navigating relocation hearings," "for spouses suspecting hidden assets."

Both query shapes are real. People type both. But the same firm will not perform equally on both, because they reward different content strategies.

The five-firm cohort

In June 2026 we ran our standard audit pipeline against five NYC divorce firms, all of which appear in the top of our public Atlas leaderboard for the category. The Atlas itself uses conversation-state methodology.

Finding 1 — Consensus-tier firms are systematically invisible on conversation-state probes.

Four of the five firms scored 0% on a 30-prompt conversation-state sample. These are not unknown firms. They are top-ranked, deeply credentialed, and routinely cited on broad recommendation questions — which is what drives their Atlas share. But on situational prompts that describe a specific buyer crisis (a physician's divorce, a hidden-asset case, an imminent custody hearing), the LLMs do not surface them. Their websites speak credentials, not situations.

Finding 2 — Firm E's 37-percentage-point swing.

Firm E scored 0% on conversation-state probes and 37.5% on keyword probes. Same brand, same week, same four providers, same probe pipeline. Only the prompt shape changed. A tool measuring only keyword shape would call this firm a star performer. A tool measuring only conversation-state shape would call it invisible. Neither verdict is wrong; they are measuring different things.

The construct-validity problem

When the same instrument can produce a 0% reading and a 37.5% reading on the same target depending on a single hidden parameter, the instrument lacks construct validity. It does not measure a single underlying property; it measures whatever the parameter currently happens to be.

We have isolated four such parameters in AI citation auditing. Each one, held constant or varied, can swing the result by tens of percentage points.

• Dimension 1 — Identity extraction. How the auditor extracts the firm's "real" identity from its website. Two competing tools given the same URL can produce different brand summaries — "matrimonial boutique" vs "BigLaw with matrimonial as one practice area" — which propagates into prompt generation and changes who gets surfaced.
• Dimension 2 — Prompt shape. Keyword vs conversation-state, demonstrated above.
• Dimension 3 — Probe format. Some tools send the LLM a free-form natural-language question and parse the answer. Others send a structured prompt asking for "the top 5 firms in ranked order, as JSON." The two formats elicit different response distributions from the same model. (See arXiv:2408.08656, NAACL 2025, for the formal result.)
• Dimension 4 — Model variant. GPT-4o, Claude Sonnet 4.6, Gemini 2.5 Pro, and Perplexity's Sonar-Pro do not cite the same firms at the same rates. A single-provider audit is a single data point. A four-provider audit is at least directional.

Any tool that varies on these dimensions without disclosing them is selling a number whose meaning the buyer cannot interpret.

What Viclaro does

We do not claim to have solved construct validity. We do claim to have disclosed it.

1. Four providers per audit. OpenAI GPT-4o, Anthropic Claude Sonnet 4.6, Google Gemini 2.5 Pro, Perplexity Sonar-Pro. Every per-check rate is an average across the four. Single-provider numbers are too volatile to anchor a strategic decision.
2. Both query shapes when both are measurable. For brands that map to a tracked Atlas vertical, we report the conversation-state rate (how real buyers query AI in 2026) and the keyword rate (how SEO has historically been measured). The gap between them is the most useful single number on the report — it tells the brand which content strategy they are currently optimized for.
3. Anchored against a public benchmark. Every audit cover renders the brand's Atlas category share next to their per-audit citation rate. A firm whose audit reads 5% but whose Atlas category share is 18% is in a very different situation than one whose audit reads 5% and whose Atlas share is also 5%. Single-number reports cannot tell the difference; we refuse to ship one.
4. Freeform probes by default. The structured-JSON probe format is faster to parse, but the response distributions diverge from natural-language probes in ways that materially affect which firms surface.
5. Disclosed sample sizes and confidence intervals. At N=30 prompts × 4 providers (120 measurements), the 95% Wilson confidence interval at p=0.5 is approximately ±9 percentage points. We say so. A tool reporting "your AI visibility is 7.2%" with no confidence interval is selling false precision.
6. Deep, not at scale. A Viclaro audit takes 10–35 minutes of compute and produces a 40+ page report. We do this once, deeply, for a paying customer. We do not yet operate a self-serve dashboard that would be tempting to over-claim against.

What this means for the buyer

If you are evaluating an AI visibility tool, the question to ask the vendor is not "what is our score." It is:

1. Which prompt shape does your benchmark use — keyword, conversation-state, or a mix?
2. How many providers do you query per prompt? Which ones? On which model variants?
3. What is the 95% confidence interval at your sample size?
4. If you ran the same brand twice, six hours apart, what is the test-retest reliability of your number?
5. How does your benchmark handle brands whose website is generalist versus specialist?

A vendor that cannot answer all five is not yet measuring a thing. They are selling a number.

Limitations

Five firms in one vertical (NYC Divorce & Family Law) at one point in time. The directional finding — that prompt shape can swing citation rates by tens of percentage points — is robust across the five firms. The specific numbers will move as LLMs are retrained, as firms update their websites, and as buyer query patterns evolve.

We did not control for query order or time-of-day. Neither has shown a material effect in internal replication runs, but we have not formally measured the effect size.

The Atlas itself uses conversation-state methodology, which means our "conversation-state per-check rate" and our "Atlas share" are not independent — both ultimately query LLMs with situational prompts. The 37pp keyword-vs-conv gap is independent of the Atlas; it is internal to the audit pipeline.

Methods (short version)

Pipeline. Site crawl (DOM-rendered, JS-rendered fallback) → identity extraction (Claude Sonnet 4.6) → prompt generation (N=30 conversation-state, weighted 50% broad-recommendation / 30% situational / 20% validation) → probe each prompt against four providers in freeform mode → entity-name normalization → citation determination → aggregate.

Citation rule. A firm is "cited" on a given probe if the LLM response names the firm, names a person publicly listed on the firm's website as an attorney, or names the firm via a clear synonym or DBA.

Confidence intervals. Wilson score interval at α=0.05. For N=120 (30 prompts × 4 providers) at p=0.5, ±9pp. At p=0.1, ±5pp.

Atlas snapshot. Snapshot #33, NYC Divorce & Family Law, v3.1 prompt set.