Episode 9 · 18 Aug 2026

Evaluations, demos, and the difference

What a vendor means by an evaluation, what a buyer needs, and the three questions that separate them. Plus the review policy for this run.

Show notes

• "Evaluation" means three different things to a vendor, a buyer and a reviewer. Ask which one is on the table before agreeing to anything.
• The three questions that separate a demo from an evaluation: who chose the data, who chose the tasks, and who can stop the clock.
• Scores here are composite, to one decimal, and every axis is published.

Chapters

• The week in five lines
• Three meanings of one word
• How reviews get scored here
• Listener mail

Transcript

This episode runs as written notes rather than a recording. The argument, in order.

A vendor evaluation is a guided tour. It is built from data the vendor chose, on tasks the vendor chose, ending at the moment the vendor chooses. None of that is dishonest on its own; it is simply a sales artefact, and it should be read as one.

A buyer evaluation is the opposite exercise. You supply the data, you supply the tasks, and you decide when it is finished. If a vendor cannot survive that arrangement, the product is not ready for the claim being made about it.

The review policy follows from the same logic. Products are bought, used for a stated period, and scored on axes that include the ones vendors would rather avoid: pricing honesty, support, durability under sustained use, and exit cost. The composite is an average of published axes, to one decimal, and revisit dates are set at the time of publication.