Research note 002

Why one AI response cannot establish visibility

AI systems generate answers probabilistically. The same question can produce different firms, in a different order, minutes apart. A single response is an observation — not a ranking.

Variability

A single AI response is an observation, not a ranking.

Generative systems do not retrieve a fixed list. Their answers depend on sampling, retrieval results at the moment of the query, model version, wording and context. Asking the same question twice can produce a different set of firms. That is normal behaviour, not an error.

Observation versus pattern

An observation tells you what one system said, at one time, for one prompt. A pattern requires repeated observations under defined conditions. Only patterns support measures such as appearance frequency, recommendation frequency or stability.

Screenshots are useful but limited

A screenshot is genuine evidence that an answer occurred. It cannot show whether the same answer would occur again, whether it holds across other prompts or AI systems, or whether it was typical. Presenting a screenshot as proof that a firm “ranks #1 in ChatGPT” overstates what it shows.

What stability research tries to measure

The number of repetitions needed depends on the study design. There is no single universal number, and we do not claim one.

  • Appearance frequency — How often a firm appears across repeated relevant observations.
  • Recommendation frequency — How often it receives recommendation treatment.
  • Recommendation stability — How consistent the recommended set is across repeated runs.
  • Cross-model consensus — Whether different AI systems surface the same firms.

What this means for the free check

The FirmRanker check is a snapshot of what AI systems returned at a point in time. It is useful, but it does not produce the same quality of evidence as a controlled, repeated study, and FirmRanker does not present it as a ranking.

FirmRanker is an international research and intelligence platform studying how AI systems discover, surface, cite and recommend law firms.

One ecosystem. Three ways to help your firm.

Measure with FirmRanker. Understand with Law By Dan. Act with Practice Proof.

FirmRanker: Measure

Find out whether AI includes your firm in your local market, wherever you are based. Start with a free check, then explore the research behind the answers.

Check your firm free

Law By Dan: Understand

Make sense of the shift in legal discovery. Read Dan Toombs’ analysis and follow The Future Client briefing.

Read Dan’s analysis

Practice Proof: Act

Put the findings to work. Get help with your firm’s positioning, reputation, content, search and technology.

Talk to Practice Proof

Follow The Future Client, Dan Toombs’ free weekly briefing for law-firm leaders.

Get The Future Client

Implementation services are separate from research. Being a Practice Proof client does not influence FirmRanker findings.