Short answer: the experiment should test whether improving first-party consistency and provenance changes answer quality and source behavior, not whether a "hack" increases citations. Anthropic documents web search and citations, but does not publish a health-authority score or selection formula. Primary outcomes must be factual accuracy, first-party conflict rate and source-support quality; citation mix remains an external outcome.

Hypothesis

For a set of public questions about services, providers and locations, reducing first-party conflicts and clarifying ownership will improve factual accuracy and may change the source mix compared to a comparable cohort without full rollout.

The hypothesis must be written before the intervention.

Population

Choose two comparable groups of services or locations. Avoid putting the most popular services in treatment and marginal ones in control.

Document exclusions: ongoing relocations, suspended services, rebrands, or pages undergoing a major rewrite.

Baselines

For each query it keeps:

  • the text of the question;
  • language;
  • the target entity;
  • factual accuracy;
  • source-support classification;
  • source mix;
  • first-party conflicts;
  • time stamp;
  • provider/location status;
  • severity.

The intervention group

Apply only controlled changes:

  1. clarify provider/facility/service mapping;
  2. correct location and service data;
  3. align author/reviewer where relevant;
  4. fix critical controllable external profiles;
  5. add provenance and owner for material claims;
  6. consolidate duplicate intent only where the evidence justifies.

Don't simultaneously rewrite the entire site and add backlinks as part of the same test.

The control group

Keep a comparable set without full temporary rollout. If a P0 safety issue or incorrect clinical information occurs, immediately fix and flag the contaminated control.

The log of changes

For each edit it saves URL, old value, new value, owner, timestamp, reason and expected effect.

Without a change log, you cannot separate the intervention layer from collateral changes.

Observation window

Define the period before. Keep the same frequency and the same query set. If a rebrand, relocation or major service change occurs, close the version and start a new one.

Don't just expand the window until more citations appear.

Outcome 1: factual accuracy

Classify claims correct',incomplete', wrong',unverifiable'. Separate medical and operational.

Outcome 2: source-support quality

For citation claims, check that the source actually supports the claim. A display URL is not enough.

Outcome 3: first-party conflict rate

It measures contradictions between provider profiles, location pages, service pages and articles.

Outcome 4: entity consistency

Check that provider, facility and service are correctly mapped in the response.

Outcome 5: source mix

It separates first-party, publications, institutions, directories and review platforms.

Outcome 6: owned citation rate

Report separately, with clear denominator: eligible observations with sources shown.

Do not turn it into a quality score.

Confounders

  • PR;
  • provider lifecycle;
  • relocations;
  • rebranding;
  • external profile updates;
  • content rewrites;
  • Search changes;
  • AI platform/model changes;
  • query-set drift.

Stop criteria

Stop if:

  • the control receives the same intervention;
  • cohorts become incomparable;
  • the query set changes materially;
  • a location or service is undergoing reorganization;
  • sample size becomes too small;
  • a safety issue requires immediate remediation.

Negative control

It includes some questions where the first-party was already coherent and the intervention should not change the verdict. If these are also "improving", check evaluator bias or rubric drift.

Blind rating

On one sample, hide treatment/control and classify factuality and source support under the same heading.

How do you treat the positive result

If factual accuracy increases and conflict rate decreases in treatment, the intervention layer has direct value. If the source mix also changes, report the association with limitations.

How do you handle the null result

If citations do not change, but first-party conflicts decrease, the method may remain useful for governance and safety.

How do you deal with divergence

If one location responds and another does not, analyze external footprint, provider stability and source freshness. Do not average until the difference disappears.

Replication

Repeat the protocol in another service or market with the same heading and denominators. Standardize only if primary outcomes reproduce.

Acceptance criteria

The experiment is valid when:

  1. the hypothesis is predefined;
  2. the population is versioned;
  3. the control is comparable;
  4. intervention layer is delimited;
  5. the change log is complete;
  6. observation window is fixed;
  7. the denominators are explicit;
  8. stop criteria are respected;
  9. confounders are logged;
  10. the conclusion does not claim unproven causality.

How do you handle source changes during the test

An institution, directory or publication can update the cited page during the observation window. Keep logical snapshot by timestamp, title, URL and verified claim. If the source content changes materially, mark the observation and do not directly compare two rounds as if the external evidence were identical.

How do you treat the difference between provider discovery and medical explanation

Questions about where a provider works or what services a clinic offers have a different evidence structure than educational questions about a procedure. Do not mix these two classes into one accuracy rate. Preserve task type and expected source family so that source mix change can be interpreted correctly.

How do you treat new pages

A first-party page published during the experiment may change the natural source behavior. Flag new_page_event and separate before/after results. Do not attribute any difference to the cleanup if the intervention layer also includes the appearance of a completely new surface.

Sustainability criterion

A protocol is worth extending only if it reduces conflicts and factual errors without requiring continuous rewriting or disproportionate manual monitoring. Measure rework per cohort and keep operational cost next to outcome.

Claim ledger

  • FACT/EVIDENCE: Anthropic documents web search and source citations.
  • PRACTITIONER GUIDANCE: healthcare citation experiments must separate factuality, source support and entity consistency.
  • INFERENCE: reducing first-party conflicts can contribute to more stable outputs.
  • NOT PROVEN: a universal tactic that produces Claude citations or a health-authority score.

Conclusion

A good experiment for Claude web citations in healthcare does not try to demonstrate a shortcut. It tests whether a more coherent public system produces more correct responses and whether source behavior changes in the same direction. Primary proof remains quality and consistency, not the mere appearance of the domain.

Sources reviewed

Related insights