Find the account, model, context, integration and input differences that can make an AI tool behave differently after a live demonstration in practice.

Direct answer: The stage and your workspace probably differ in configuration, context or input, even when the product name is the same. Compare the application, account tier, selected model or mode, persistent instructions, connected sources, source file and human interventions one variable at a time. If the presenter will not identify the demonstrated setup, the result cannot be reproduced and should remain a vendor claim.

AI output can vary between runs, but variation is not a complete diagnosis. A live demonstrator may use a prepared account, cleaned data and preloaded context. Your account may expose a different feature, use a different mode or lack the same integration.

At a London meetup, the presenter may also use a venue network, a managed demonstration laptop and staff support that attendees do not have afterwards. Those conditions belong in the comparison because they can change uploads, connected tools and recovery.

The right response is a controlled comparison, not repeated prompting until the stage result appears. This is a narrow troubleshooting application of A Useful AI Workflow for Focused Knowledge Work: keep the question, evidence and judgement with a person while changing one condition at a time.

Applies to: documentation-based diagnosis across generative AI applications. Product features, plan names, model availability and data terms change, so verify the exact account and date rather than following remembered workshop steps.

Use the configuration delta trace

The configuration delta trace is an editorial diagnostic method. Make two columns, "live demo" and "my run", then compare seven layers in order:

  1. Application and access route.
  2. Account, plan and administrator settings.
  3. Model, mode and optional tools.
  4. Persistent instructions, memory or project context.
  5. Connected files, search indexes and integrations.
  6. Exact input and prompt sequence.
  7. Human selection, editing and reruns.

Do not change several layers together. If you switch the model, rewrite the prompt and clean the document at once, a better result will not tell you which change mattered.

Establish what happened on stage

Start with evidence from the London demonstration. Record the displayed application, any model or mode label, the type and size of source, visible connected tools and the output shown. Distinguish what you saw from what the presenter said.

Ask whether the workspace contained earlier instructions or uploaded material, whether the input was prepared, and whether the answer was selected from earlier runs. A presenter may reasonably compress these details to fit a London session. The problem begins only when an edited, selected or prepared result is presented as the default experience without qualification.

Do not infer hidden settings from visual appearance. Obtain documentation or a direct statement. If neither is available, mark the stage configuration "unknown" and do not promise colleagues that you can reproduce it.

Check the access route first

A model is not the same as the application that provides access to it. Applications can add retrieval, formatting, safety controls, memory, integrations and administrator policies around a model. Consumer subscriptions, workplace accounts and API access may also have different capabilities and data terms.

That distinction explains why copying words from a demo prompt may not copy the result. Record whether the presenter used a web application, a workplace deployment, a developer interface or a feature embedded in another product. Then confirm what you used.

The UK Government AI Playbook says its government audience should understand the system's limitations and the full lifecycle, including setup, maintenance and assurance. It also states that AI output is not guaranteed accurate. Although it is not a private-sector product manual, it supports treating the surrounding system as part of the evaluation. AI Playbook for the UK Government

Compare context before rewriting the prompt

Context includes more than the message currently visible. It may include system instructions, project files, an indexed knowledge source, prior messages, remembered preferences or the output of a connected search tool. The specific mechanisms vary by application and account.

Use a fresh test workspace if the product supports one, but do not describe that as a guarantee of no retained or administrator-supplied context. Check current documentation. Supply the same synthetic source file and the same prompt. If the demonstration used a real document you cannot lawfully or contractually access, reproduce its structure with invented content instead.

Privacy comes before reproduction. Do not upload client material to discover whether it makes the answer better. The ICO's data-minimisation principle requires personal information to be adequate, relevant and limited to what is necessary for the purpose. The regulator notes that guidance is being updated following recent UK legislation, so recheck it for a deployment. ICO data-protection principles

Use a small controlled test grid

Suppose an illustrative analyst wants to compare two available accounts, two documented modes and three synthetic documents. The full grid contains:

2 accounts × 2 modes × 3 documents = 12 runs

Allow 4 minutes to run and record each case, plus 22 minutes to prepare the files and rubric. The test costs 12 × 4 + 22 = 70 minutes.

By contrast, twenty unsystematic reruns at 5 minutes each cost 100 minutes and still mix prompt, context and configuration changes. The controlled grid releases 30 minutes and produces attributable evidence. That is released time, not cash saved, and all figures are illustrative assumptions. If outputs are high impact, twelve runs are not sufficient assurance; the example only shows why controlled comparison beats random repetition.

For each result, check required fields, source support, material omissions and reject behaviour. Do not score tone as accuracy. Preserve the prompt, configuration label, input version, time and output so another person can inspect the comparison.

Account for human intervention

A stage result may have been edited, shortened or selected. Ask whether you are comparing raw output with raw output. If the facilitator spent eight minutes correcting a two-minute generation, include ten minutes in the method.

This is not a reason to reject the product. Human review is expected for many useful workflows. It is a reason to measure the complete process. A five-second generation with fifteen minutes of repair is not faster than a ten-minute manual task.

NIST describes testing, evaluation, verification and validation as work carried out across the AI lifecycle. Its AI Risk Management Framework is voluntary and under revision, so it should guide your test discipline rather than serve as a product approval badge. NIST AI Risk Management Framework

Decide what the difference means

Classify the result after the trace:

  • Reproduced: the relevant output meets the same rubric under a documented matching setup.
  • Explained difference: a known account, mode, context or input difference changes the result.
  • Uncontrolled variation: material differences remain after known conditions match.
  • Unreproducible claim: the stage setup or source cannot be established.

An explained difference may justify changing account or workflow, but only after you assess price, privacy, access and maintenance. Uncontrolled variation may still be acceptable for brainstorming, where wording changes do not affect a decision. It is unsuitable where the workflow assumes stable facts or actions without an independent check.

What to do in the next two working days

Today, write the two-column trace and gather the London presenter's public documentation. Tomorrow, create three synthetic cases, define the acceptance rubric and run the smallest test grid that isolates the suspected variable. Keep every other condition fixed.

Stop when the missing configuration cannot be verified, the required account is unavailable, or a material failure persists across controlled runs. Do not spend a week reverse-engineering a demonstration for a task the current manual process already handles well. Decide whether to request one precise vendor answer, redesign the workflow, or reject the claim.

Frequently asked questions

Does a different answer mean the model has changed?

Not necessarily. First check the application, account, mode, context, tools, source and prompt sequence. Generative output can vary even under similar conditions, but blaming an unseen model change skips more observable causes. Look for official release notes if the provider identifies a version change. Do not infer one from wording alone. If the task requires stable facts, use source-linked checks or a deterministic method rather than expecting repeated generation to prove which answer is correct. Preserve the two outputs so the comparison remains inspectable.

Can I reproduce a demo with the same prompt?

Only if the prompt was the material difference. The stage account may contain persistent instructions, files, integrations or administrator controls that are not copied with the visible text. The input may also have been cleaned or shortened. Record all known conditions and use the same synthetic source. If the provider cannot disclose an essential dependency, label the demo unreproducible. You can still test the available product independently, but you should not claim that your test confirms or refutes a setup you cannot inspect.

Should I keep rerunning until I get the stage result?

No. Repeatedly selecting the closest answer hides the failure rate and encourages confirmation bias. Define the number of runs and rubric before starting, then retain every output. If one of ten runs works, the procedure does not have a 100 per cent success rate because you preferred that answer. The exception is a creative task where selection among alternatives is deliberately part of the workflow. In that case, count generation and selection time, and ensure no factual claim escapes review.

Could my laptop hardware cause the difference?

It can if processing occurs locally, or if the browser, operating system, memory or graphics capability affects a documented feature. For a cloud service, hardware may still affect uploads, rendering, local extensions and network-dependent steps, but it does not automatically explain a different generated answer. Check provider requirements and observe where the failure occurs. Do not buy new hardware until you can reproduce a hardware constraint. A missing account feature will not be repaired by a faster laptop. Compare on a supported device before spending.

What if the demo used confidential vendor data or private tools?

Then the public result may not be reproducible. Ask which parts are generally available and which relied on private material. Do not request access to data you are not entitled to receive. Evaluate the available product on your own permitted synthetic cases. If the product's value depends on a private knowledge base, assess the work, rights and security required to build and maintain your own. A polished private setup is evidence about that setup, not the product in isolation. Record the limitation in any recommendation you make.

How much variation is acceptable?

Acceptable variation depends on what changes. Alternative wording may be harmless in brainstorming; a changed total, deadline, quotation or eligibility decision can be material. Define required invariants before testing. The output must preserve those fields or trigger review. Avoid inventing a universal percentage threshold. Measure errors on representative cases and consider their consequence. If a person must recheck every field anyway, compare the total review effort with the non-AI process before keeping the workflow. Reject variation that defeats the task's purpose.

Sources and verification

Twokq Tech

This article is practical guidance. Apply it in proportion to your tools, evidence, risks, and responsibilities.