Short answer: an editorial A/B for Claude web citations in finance must test a controlled intervention on first-party clarity and provenance, not a "citation hack". Primary outcomes can be factual accuracy, source-support quality, product-market consistency and first-party conflict rate. Anthropic documents web search and citations, but does not publish a full source selection formula or a financial authority score.

Hypothesis

For a versioned set of financial questions, an intervention that aligns product identity, market, effective dates, and source ownership will reduce factual errors and source-support failures relative to a comparable cohort without full rollout.

The hypothesis must be written before the treatment and not changed after the first observations.

The experimental unit

Choose product families or plan groups, not arbitrary URLs. If ten pages inherit the same pricing source, they are not ten independent observations.

Keep product, market, lifecycle and release cadence in matching.

The population of queries

Includes questions about rates, fees, eligibility, product comparisons, terms and general explanations. Preserve wording, language, market and product ID.

Excludes queries that ask for personal data or sensitive individual situations.

Baselines

For each observation it saves query, claim set, product-market mapping, displayed sources, factuality, source support, effective-date fit and timestamp.

In parallel, save first-party expected state from product page, terms, pricing or other appropriate owner.

The intervention group

Apply only bounded changes:

  1. clarifies product-plan relations;
  2. align market/jurisdiction;
  3. add effective dates for volatile fields;
  4. eliminates first-party contradictions;
  5. map source owners;
  6. consolidate duplicate explanations only where the task is identical.

Do not change pricing, product positioning, backlinks and campaigns at the same time if you want to interpret the result.

The control group

Use comparable products such as maturity and update cadence. If a P0 factual error occurs in the control, immediately repair and mark contamination; experiment purity does not justify keeping risky information.

Primary outcome 1: factual accuracy

Classify claims correct',incomplete', wrong',unverifiable'. The denominator is the total of verifiable claims.

Primary outcome 2: source-support quality

Check that the cited source accurately supports the claim. A Topically Relevant URL may not support the stated value, condition or period.

Primary outcome 3: product-market consistency

It measures observations where the product, plan, and market are simultaneously correct.

Primary outcome 4: first-party conflict rate

Number of contradictory material claims between verifiable sources. This is a direct outcome of internal governance.

Exploratory outcome: owned-source rate

You can track if more first-party sources appear, but don't turn it into a quality score. An increase in domain citation is not enough if factuality does not improve.

Exploratory outcome: source mix

Separate first-party, official/regulator, publishers, comparators and review platforms. Analyze change descriptively, not as an automatic winner.

Observation window

Set the period before. Use multiple rounds and the same frequency. If there is a major product or market change, close the version and start a new one.

Stop criteria

Stop or reschedule if:

  • cohorts become incomparable;
  • the control receives the same intervention;
  • product pricing changes massively;
  • the query set changes materially;
  • a regulatory update modifies the expected state;
  • sample size becomes insufficient;
  • rubric evaluator changes without versioning.

Confounders

  • PR and media coverage;
  • product launch or retirement;
  • rate changes;
  • merger/rebrand;
  • market expansion;
  • publisher updates;
  • Search updates;
  • platform/model changes;
  • seasonality;
  • campaign traffic.

Negative control

Includes queries where first-party was already coherent. If they "improve" significantly without real change, check evaluator bias or platform drift.

Blind rating

On one sample, hide treatment/control and ask the rater to classify factuality and source support with the same rubric.

How do you treat variable rates

It stores the formula and the index, not just the point value. If the intervention layer clarifies the mechanism, it measures whether the answer preserves the formula and conditions, not just the figure.

How do you treat terms updates

If terms change during the test, mark event data. A response change can be determined by the truth change, not by the intervention layer.

How do you handle external comparators

Do not remove them from the analysis. Classifies source roles and checks that data is current. A comparator can be useful, but it does not become a product owner.

How do you interpret a positive result?

If factual errors and source-support failures decrease more in the treatment, the intervention layer has operational evidence. External source mix can be discussed separately.

How do you interpret null result

If the citation mix does not change, but first-party conflicts decrease, the experiment may still be successful on primary outcomes.

How do you interpret a negative result?

If the first-party simplification removes relevant conditions or produces product ambiguity, rollback and redefine the intervention layer.

Replication

Repeat the protocol on another product family or another market before standardization.

Acceptance criteria

The experiment is valid when:

  1. the hypothesis is predefined;
  2. the experimental unit is clear;
  3. the cohorts are comparable;
  4. the query set is versioned;
  5. intervention layer is delimited;
  6. the denominators are clear;
  7. observation window is fixed;
  8. stop criteria are documented;
  9. confounders are logged;
  10. external outcomes are separated from primary proof.

Claim ledger

  • FACT/EVIDENCE: Anthropic documents web search and source citations.
  • PRACTITIONER GUIDANCE: financial citation experiments must separate factuality, source support, product-market consistency and source mix.
  • INFERENCE: first-party consistency can reduce ambiguity and change the behavior of observed sources.
  • NOT PROVEN: a universal tactic that produces Claude citations or a financial authority score.

Conclusion

Editorial A/B for Claude web citations in finance only makes sense if the primary outcomes are about factuality and source support. If the treatment produces more sources, but the answers remain wrong or incomplete, the experiment failed. Measure what you can prove and treat visibility as a separate layer.

Sources reviewed

Related insights