Study · Validation

System Validation 001: how AI recommends Sydney personal injury lawyers

A controlled Sydney public-liability dataset used to validate FirmRanker's research pipeline before larger benchmark studies.

Study ID
SV-001
Market
Sydney, NSW, Australia
Practice area
Personal Injury
Prompt archetypes
10
AI systems
4
Observations
40
Status
Validation
Methodology
v1.0

Why this study exists

System Validation 001 is not designed to establish which Sydney law firm is “best” or to publish an AI visibility ranking.

Its purpose is to test whether FirmRanker can reliably preserve AI answers, identify firms, resolve name variants, classify response treatment, capture sources and support human review.

The study is therefore research infrastructure rather than a market benchmark.

Study design

10 unbranded prompt archetypes → × 4 AI platforms → × 1 repetition → = 40 observations

Model identity returned by each provider is recorded with every observation.

  • GPT-5 with web search
  • Gemini 2.5 Flash with Google Search
  • Claude Sonnet with web search
  • Perplexity Sonar Pro

The original answer is never rewritten

FirmRanker preserves for every observation:

  • Exact prompt
  • AI provider
  • Returned model
  • Timestamp
  • Raw answer
  • Raw source URLs
  • Searches, where available
  • Later corrections may change — Extraction, canonical entity, response treatment, position and source classification.
  • They do not change — The original observation.

Human QA

Automated extraction → Frozen machine result → Assisted review → Final human review → Accuracy vs frozen output

Automated extraction produces the original machine result, which is frozen. Assisted review may propose corrections. Final human review creates a gold-standard comparison layer, and machine accuracy is measured against the frozen original output.

Current status

Status: Validation in progress.

Firm-level outputs remain preliminary until entity resolution and human QA are complete. No firm-level results, league tables or accuracy metrics are published from this dataset.

Limitations

  • Only one market
  • One practice area
  • One repetition per prompt/model combination
  • Machine extraction pending and subject to QA
  • AI outputs may vary
  • Provider behaviour may change
  • Model versions change
  • Source retrieval differs by provider
  • Results should not be generalised beyond tested conditions

FirmRanker is an international research and intelligence platform studying how AI systems discover, surface, cite and recommend law firms.

One ecosystem. Three ways to help your firm.

Measure with FirmRanker. Understand with Law By Dan. Act with Practice Proof.

FirmRanker: Measure

Find out whether AI includes your firm in your local market, wherever you are based. Start with a free check, then explore the research behind the answers.

Check your firm free

Law By Dan: Understand

Make sense of the shift in legal discovery. Read Dan Toombs’ analysis and follow The Future Client briefing.

Read Dan’s analysis

Practice Proof: Act

Put the findings to work. Get help with your firm’s positioning, reputation, content, search and technology.

Talk to Practice Proof

Follow The Future Client, Dan Toombs’ free weekly briefing for law-firm leaders.

Get The Future Client

Implementation services are separate from research. Being a Practice Proof client does not influence FirmRanker findings.