Compare two AI tools shown at one London event by recording equivalent tasks, access levels, controls, review effort, costs and failure cases.

Direct answer: Record the task, input, account tier, connected services, human intervention, output check and failure route for each demonstration. Do not compare applause, speed or output quality when the presenters used different cases or access levels. Use the event to identify candidates, then rerun the same representative and difficult cases under equivalent conditions before choosing.

Sharing a stage does not create a controlled comparison. One vendor may show a prepared enterprise integration while another answers an improvised question through a standard web application. The first can look more capable even though the event has changed several variables at once.

A fair comparison does not require laboratory perfection. It requires enough common ground to know which difference affected the decision.

Applies to: multi-vendor AI showcases, conferences and demo nights in London. No current event, vendor, model, application or plan is recommended.

Use the like-for-like demo ledger

The like-for-like demo ledger is an editorial method with one row per tool and one column per decision-changing condition. It prevents a presenter from defining the comparison through the strengths chosen for the stage.

Use these columns:

  • user task;
  • exact input or case type;
  • application, model or service named;
  • account tier and region;
  • connected data and tools;
  • prepared instructions or examples;
  • retries and human edits;
  • acceptance check;
  • failure and recovery route;
  • export and exit route;
  • complete cost for equivalent use.

Mark unknown rather than guessing. You may leave the London event with two incomplete rows. That is better than a confident ranking built from different conditions.

This method narrows the principles in How to Choose the Right AI Tool Without Being Sold by the Demo: define the job, use difficult cases, include review and plan for exit. The event supplies observations; the ledger decides what still needs testing.

Fix the comparison task before the first demo

Write a bounded job which both products plausibly support. Include the input, output and unacceptable failure. For example:

From a supplied fictional service policy, draft a customer checklist with a source paragraph for every requirement; do not invent a requirement when the policy is silent.

This is better than “Which tool writes better?” It gives you completeness, source fidelity and unsupported additions to inspect.

If the event announces tasks in advance, prepare one ordinary case and one difficult case. If it does not, use the presentations to build a later test rather than trying to force a public comparison. Do not upload private files or ambush speakers with unsafe material.

Record whether each tool performs the whole task or only one component. A model may generate text while the application retrieves files, applies hidden instructions and exports the result. Product names alone do not tell you which component caused the difference.

Make access levels equivalent

Ask which plan is on stage and whether an attendee can obtain it in the UK. Enterprise contracts may enable administration, connectors, support or data controls absent from consumer access. A special demonstration environment may not be purchasable at all.

Equivalent does not always mean identically priced. It means comparing configurations capable of the required job and including the cost of reaching that capability. If Product A needs an enterprise connector and Product B uses a supported file upload, record those as different routes to the same task.

The National Cyber Security Centre's secure AI guidance recommends considering model and provider choice alongside deployment environment, oversight, legal requirements, data sensitivity and supply-chain security. Its secure-design guidance also recommends due diligence for external providers and controls on data sent to external services. The document targets AI system providers and organisational decision-makers; use it as risk guidance, not as certification of either product.

My default recommendation is not to declare a winner at the event. The condition that changes this is a narrow, directly observed compatibility question under truly equivalent access, such as whether each product exports the required documented file structure. Even then, check terms and repeat before depending on it.

Compare controls and review work

Output quality is only one criterion. A tool which produces a strong first draft but hides sources or makes correction difficult may cost more to operate than a less fluent tool with traceable output.

Criterion Evidence to collect Failure that matters
Accuracy Claim checked against supplied source Incorrect or unsupported material statement
Completeness Required fields mapped to source Missing condition or exception
Control Preview, edit, approve and reverse External action before approval
Privacy Current data flow and terms Unauthorised or unnecessary disclosure
Accessibility Required input and review methods work A reviewer cannot inspect or correct
Portability Usable export with source context Work trapped in the service

Measure review time separately. Fast generation followed by slow correction is not a fast workflow. Record the reviewer, source and action for every material check. If the event shows only the final output, ask how much preparation and correction happened beforehand.

Government AI assurance guidance includes performance testing against predetermined requirements and emphasises using techniques suited to the system and context. The official portfolio makes clear that no single technique covers every concern. Your ledger is a small decision aid, not a formal assurance or certification.

Work a transparent trial comparison

The following is an illustrative example, not a result for real products. A team tests Product A and Product B on eight fictional policies: four ordinary, two difficult and two outside the intended scope.

With acceptable handling defined in advance:

  • Product A handles 7 of 8 acceptably and takes 240 review minutes across the set.
  • Product B handles 6 of 8 acceptably and takes 165 review minutes.

That is 30 minutes of review per case for A: 240 ÷ 8 = 30. For B, 165 ÷ 8 = 20.625 minutes, or 20 minutes 38 seconds. Product A handles one more case acceptably but requires 75 additional review minutes.

Now compare an illustrative three-month trial:

  • A access: £45 × 3 = £135.
  • B access: £25 × 3 = £75.
  • A setup and controlled review work: 7 hours at £25 = £175.
  • B setup and controlled review work: 8 hours at £25 = £200.

Complete illustrative trial cost is £135 + £175 = £310 for A and £75 + £200 = £275 for B. The £35 difference does not decide the choice. The team must inspect which cases failed and whether the 75 extra review minutes buy a result that matters. Time value is not a cash saving unless those hours produce or avoid real expenditure.

Do not publish 7 out of 8 as an accuracy rate for the wider product. The small set supports only this task, configuration and selection of cases.

Use stage claims correctly

When a presenter says the product is faster, more accurate or less expensive, ask for the comparator, configuration and method. A current official feature page can support availability. A documented benchmark may support its own task and conditions. Neither proves your workflow result without transfer evidence.

Current UK advertising guidance says objective comparative claims require suitable substantiation and may mislead when the basis is unclear. The ASA's guidance explains the need to support objective claims. This does not determine whether a statement at a particular event is legally an advertisement. It gives you a sensible editorial habit: record the evidence and scope beside the claim.

Do not ask one vendor to explain the other's product. Verify each through its own current documentation and your equivalent test. If a vendor identifies a limitation in a rival, turn it into a test case instead of repeating it.

Run the fair comparison within one week

Before the event, write the common task and ledger columns. During it, fill only what you observe or receive from an official source. Note London event conditions, including network, supplied accounts and presenter intervention.

Within one week, obtain permitted equivalent access and run the same ordinary, difficult and out-of-scope cases. Preserve inputs and raw outputs, use the same acceptance rules and record review effort. Compare both with your existing non-AI process.

Choose only when one option meets the required quality and control at an acceptable complete cost. Stop when access cannot be made equivalent, a material condition remains undisclosed or neither product beats the current process enough to justify new risk and maintenance. “No purchase” is a valid comparison result.

Frequently asked questions

Can I compare the speed shown in two stage demonstrations?

Only if the task, input size, application, account, connected tools, network conditions, start and finish points and human preparation are genuinely comparable. Stage timing often excludes upload, setup, retries, checking and export. Record observed elapsed time as one event fact, not a general performance result. Test end-to-end time on the same permitted cases later, including correction. The exception is a narrow live operation performed under visibly identical conditions, but even then one run does not establish reliability or ordinary variation.

Should I compare models or applications?

Compare the product configuration you would actually use. A model generates or analyses content, while an application may add retrieval, tools, stored context, permissions, hidden instructions and an interface. Two applications using a similarly named model can behave differently, and one application may switch models. Record both when documented. Do not assume a model benchmark predicts the application workflow. The exception is a direct model evaluation through equivalent interfaces and settings, which is a specialist test and should still state the exact versions and tools.

What if one vendor will not disclose the setup?

Mark the material conditions unknown and restrict your conclusion. You may still learn the intended workflow, but you cannot compare performance or availability fairly. Ask for current documentation, an equivalent trial or a written explanation after the event. Do not reward secrecy with an assumption that the setup is ordinary. Conversely, legitimate security or customer confidentiality may limit disclosure. The decision remains the same: if you cannot verify that you can obtain and govern the capability, do not depend on the stage result for procurement.

Is a weighted scorecard a good way to choose?

It can organise evidence when weights reflect a real decision, but it can also hide an unacceptable failure inside a good total. Set non-negotiable gates first, such as permitted data handling, required export, accessibility and a maximum consequence of error. Score only the remaining trade-offs, document each weight and test how a small change affects the result. Do not invent precision from subjective ratings. The exception is a formal procurement method with defined governance, where a scorecard may be required; follow that process and retain qualitative failure evidence alongside totals.

Should price be compared monthly or by usage?

Compare the complete cost for the same workload and period. Include required plan, usage charges, integrations, setup, training, review, administration, support, migration and exit. A monthly subscription and usage-priced service are not directly comparable until you model the same task volume. Verify current pricing and tax treatment for your region and account type at the time of decision. The exception is an early functional test, where a short controlled budget may be enough, but do not extrapolate it into annual cost without representative usage.

What if neither product performs well on the difficult cases?

Keep the current process or narrow the proposed AI role. Examine whether the cases are genuinely within scope, whether inputs or acceptance rules need clarification and whether a person can safely handle exceptions. Do not weaken the test merely to produce a winner. You may use either tool for lower-risk drafting while excluding the failed decision or action. The condition for another trial is new evidence, a materially changed configuration or a redesigned workflow with better controls. Otherwise, stop spending evaluation time and record why no product was chosen.

Sources and verification

No current London showcase, vendor, product, plan, price or benchmark was evaluated. Every result, price and time in the worked comparison is illustrative.

Twokq Tech

This article is practical guidance. Apply it in proportion to your tools, evidence, risks, and responsibilities.