Judge a polished London AI demonstration by adding ordinary, difficult and out-of-scope cases before treating the result as product evidence.

Direct answer: Trust it only as evidence that the demonstrated setup handled those selected examples. Before treating it as purchase evidence, add ordinary cases, difficult but valid cases and cases the system should refuse or route to a person. If the presenter cannot show limits, corrections or the exact product access used, do not generalise the stage result to your work.

Perfect examples have a legitimate role. They help a presenter explain a workflow quickly and reduce the chance that an event slot is consumed by setup. The problem is not curation itself. It is an unsupported jump from a curated success to ordinary reliability.

A London live event offers one advantage over a promotional video: you may be able to ask what would break the result. Use that access carefully instead of trying to embarrass the speaker with an irrelevant trap.

Applies to: live AI product demonstrations, startup pitches and vendor workshops in London. No current event or product is assessed, and “live” should be confirmed with the organiser rather than inferred from the listing.

Use the demo difficulty ladder

The demo difficulty ladder is an editorial method with four levels:

  1. Prepared success: the presenter's selected input under known conditions.
  2. Ordinary variation: a representative change in length, wording or format.
  3. Difficult valid case: ambiguity, missing field, noisy input or important exception.
  4. Out-of-scope case: something the system should decline, flag or hand to a person.

The purpose is not to make every system fail. It is to see whether the workflow recognises changing conditions and whether a person can review and recover.

The parent guide How to Choose the Right AI Tool Without Being Sold by the Demo recommends testing representative and messy examples. The ladder turns that principle into a short event question and a later trial.

Establish exactly what succeeded

Write a one-sentence observation which names the conditions. For example: “The vendor's enterprise demonstration account produced a paragraph-referenced checklist from the supplied text document during the session.”

That wording does not say every account can do it, that every checklist will be complete or that the product is better than an alternative. It preserves what you actually saw.

Record:

  • application and account tier;
  • model or service named, if documented;
  • file type, size and source;
  • connected retrieval, search or other tools;
  • prompt or instruction visible to the audience;
  • human edits, retries and selected outputs;
  • time between input and final result;
  • review method.

Do not fill gaps with assumptions. If a backstage operator or preloaded knowledge base may be involved, mark it unknown and ask. If the presenter cannot disclose part of a private setup, the demo may remain interesting but cannot support a comparison with your available configuration.

Add a useful next rung

Choose the smallest change which represents your real work. If a demo summarises a neat report, an ordinary variation might be the same kind of report with headings in a different order. A difficult valid case might include a footnote which reverses a general rule. An out-of-scope case might be an unreadable scan that should be flagged rather than guessed.

Ask before supplying a case. Use synthetic or public information and explain the acceptance condition. A good event question is:

If this fictional file contains one exception in a footnote, how does the workflow show the reviewer that the exception was captured?

That question examines process, not merely output. The presenter may reasonably decline an unplanned live test. Ask how the case should be evaluated later and which access would be required.

My recommendation is to give more weight to a demo which exposes a limitation and shows recovery than to one which produces only flawless output. The decision changes when the demonstrated question is a simple binary compatibility check, where one transparent success may directly answer it.

Require a review path, not just an answer

A useful workflow should show where a person checks the result, what source they check and what happens when the check fails. Look for:

  • source links or exact references;
  • uncertainty or missing-input markers;
  • editable output;
  • approval before an external action;
  • an activity record where appropriate;
  • a way to restore the original or use a manual fallback.

Government AI assurance guidance describes performance testing as assessing a system against predetermined requirements and treats assurance as context-dependent across the system lifecycle. The official portfolio lists multiple techniques rather than a single universal test. A successful prompt is therefore one observation, not a complete assurance method.

When the product performs an action, the recovery path matters more. Sending a message, changing a record or granting access has a different consequence from drafting text. The event should show preview, scope restriction and reversal where those controls exist. If it cannot, test in an isolated environment later.

Work a test set and interpret it correctly

This worked example uses illustrative test results, not a measured product benchmark. A small business evaluates an enquiry-classification demonstration with 12 fictional cases:

  • four ordinary cases;
  • four difficult but valid cases;
  • four out-of-scope cases which should be sent to a person.

Observed illustrative results after the event:

  • ordinary: 4 correctly routed out of 4;
  • difficult valid: 2 correctly routed, 1 flagged for review, 1 incorrectly routed;
  • out of scope: 3 sent to a person, 1 incorrectly routed.

Count acceptable handling only after defining it. If a review flag is acceptable, then acceptable cases are 4 + 3 + 3 = 10. The observed proportion is 10 ÷ 12 = 83.3%. If the difficult-case review flag is not operationally acceptable, the count becomes 9 ÷ 12 = 75%.

Neither figure is a general product accuracy rate. The set is tiny, fictional and selected for this task. The decision-changing result is that two cases were routed incorrectly, including one outside scope. The next test should examine the consequences and whether controls catch those failures.

The original stage demo might have shown four prepared successes, or 4 ÷ 4 = 100%. Comparing “100%” with “83.3%” as competing benchmark scores would be misleading because the case sets differ. The ladder explains why the evidence changed.

Resist stage effects without assuming bad faith

High production quality, fast replies and confident narration make a demonstration easier to follow. They can also make uncertainty less visible. At a crowded London showcase, separate presentation quality from system evidence on your notes before the next speaker resets the comparison.

UK guidance on unfair commercial practices says objective product claims should be supported by evidence, and that an overall presentation may be misleading even where isolated information is true. The CMA guidance applies in consumer commercial contexts and explains the relevant legal tests. Do not accuse a presenter based on polish. Simply record what the evidence establishes and what it does not.

Customer applause, a well-known venue and a large audience do not add test cases. Nor does a surprising failure automatically disqualify the product. The response to failure may reveal useful controls.

Do not make secret recordings to capture evidence. Follow the venue and organiser rules, and keep identifiable attendees out of screenshots or notes where possible.

Run the next test within three days

At the London event, record the prepared success and one question for the next rung. Within three days, find the current official documentation and use equivalent permitted access to run a small representative test with synthetic data.

Define acceptable handling before seeing the outputs. Record success, review, refusal and harmful or incorrect action separately. Compare the product with your existing process, including checking and correction time.

Proceed only when the workflow handles the difficult cases at a level appropriate to their consequences and when a person can recognise failure. Stop when you cannot obtain the demonstrated configuration, the presenter will not identify material conditions, or the system acts incorrectly without a practical control. Do not pay for a larger trial merely to explain an undisclosed stage setup.

Frequently asked questions

Why do vendors use perfect examples in demonstrations?

Because a short event needs a clear narrative, predictable timing and inputs the presenter has permission to show. Preparation can make a complex workflow understandable. It becomes weak decision evidence when the selection is hidden or the result is presented as typical without broader support. Ask what was prepared, which account was used and where failures appear. A transparent curated example is useful for learning the intended workflow. It simply needs representative and difficult cases before purchase. The exception is a demonstration of a deterministic documented function, where one run may answer a narrow compatibility question.

Is an unexpected live example a fairer test?

Not automatically. An audience example may be poorly specified, unsafe to share or outside the product's intended scope. Surprise also prevents the presenter from checking permission and setup. A fairer test uses a representative synthetic case, a stated acceptance condition and equivalent access for alternatives. Send it in advance when possible. If the presenter refuses it, ask for the supported test route rather than assuming failure. The exception is a simple public input which the event explicitly invites and which requires no confidential data, specialist setup or disputed rights.

What does a good refusal look like in an AI workflow?

A good refusal identifies that the input or request is outside the workflow, avoids inventing an answer, preserves the original and directs the case to a suitable person or method. It should not expose hidden information or make a consequential decision while claiming uncertainty. The exact behaviour depends on the task. Refusal is not always success; a tool that rejects normal work too often may be unusable. Define which cases should proceed, pause or stop before testing. For high-impact domains, a qualified professional should set that boundary and review the result.

Can I use the presenter's published case study as proof?

Use it as first-party evidence about the described implementation, subject to its method and disclosures. Check task, configuration, customer selection, baseline, measurement period, review effort and exclusions. A headline improvement does not transfer automatically to your organisation. Ask whether support, pricing or product access differed from the ordinary offer. The case study can help you design a test and identify dependencies. The exception is a precise factual compatibility detail that matches your environment, but even then run a reversible check before depending on it.

How many test cases are enough after the demo?

There is no universal number. Use enough cases to cover ordinary variation, material edge conditions and out-of-scope requests for your task. Consequence matters more than a convenient sample size. A low-risk personal formatting task may need a small practical check, while customer, employment, financial or safety decisions require specialist evaluation and stronger evidence. Do not publish a percentage from a tiny hand-picked set as a product benchmark. Stop increasing the sample when a known failure already makes the proposed use unacceptable or when the test no longer represents the intended task.

Should a visible failure make me reject the tool?

Reject it when the failure has unacceptable consequences and the workflow lacks a reliable way to prevent, detect or recover from it. Do not reject every tool for making any error; compare the failure with the current process and intended role. A system used only to draft may tolerate a correctable error which would be unacceptable in automatic sending. Ask whether the failure is documented, reproducible and affected by the configuration. The exception is a security, privacy or rights breach that requires immediate containment and qualified review before any further testing.

Sources and verification

No current London event, product, account tier or demonstration was evaluated. Every test result and percentage in the example is illustrative and must not be represented as a measured benchmark.

Twokq Tech

This article is practical guidance. Apply it in proportion to your tools, evidence, risks, and responsibilities.