Research note 002
Why one AI response cannot establish visibility
AI systems generate answers probabilistically. The same question can produce different firms, in a different order, minutes apart. A single response is an observation — not a ranking.
Variability
A single AI response is an observation, not a ranking.
Generative systems do not retrieve a fixed list. Their answers depend on sampling, retrieval results at the moment of the query, model version, wording and context. Asking the same question twice can produce a different set of firms. That is normal behaviour, not an error.
Observation versus pattern
An observation tells you what one system said, at one time, for one prompt. A pattern requires repeated observations under defined conditions. Only patterns support measures such as appearance frequency, recommendation frequency or stability.
Screenshots are useful but limited
A screenshot is genuine evidence that an answer occurred. It cannot show whether the same answer would occur again, whether it holds across other prompts or AI systems, or whether it was typical. Presenting a screenshot as proof that a firm “ranks #1 in ChatGPT” overstates what it shows.
What stability research tries to measure
The number of repetitions needed depends on the study design. There is no single universal number, and we do not claim one.
- Appearance frequency — How often a firm appears across repeated relevant observations.
- Recommendation frequency — How often it receives recommendation treatment.
- Recommendation stability — How consistent the recommended set is across repeated runs.
- Cross-model consensus — Whether different AI systems surface the same firms.
What this means for the free check
The FirmRanker check is a snapshot of what AI systems returned at a point in time. It is useful, but it does not produce the same quality of evidence as a controlled, repeated study, and FirmRanker does not present it as a ranking.
One ecosystem. Three ways to help your firm.
Measure with FirmRanker. Understand with Law By Dan. Act with Practice Proof.
FirmRanker: Measure
Find out whether AI includes your firm in your local market, wherever you are based. Start with a free check, then explore the research behind the answers.
Check your firm freeLaw By Dan: Understand
Make sense of the shift in legal discovery. Read Dan Toombs’ analysis and follow The Future Client briefing.
Read Dan’s analysisPractice Proof: Act
Put the findings to work. Get help with your firm’s positioning, reputation, content, search and technology.
Talk to Practice ProofFollow The Future Client, Dan Toombs’ free weekly briefing for law-firm leaders.
Get The Future ClientImplementation services are separate from research. Being a Practice Proof client does not influence FirmRanker findings.