Notes on evaluating AI companies

• Evaluation

  Why an accuracy number is not evidence

  Most AI accuracy claims are produced on evaluation sets that overlap training data. What to ask for instead, and what a genuinely held-out test set looks like in a data room.

• Defensibility

  Model dependency is a supplier concentration problem

  When core capability is rented from one provider, pricing, latency and quality are set outside the company. How to price that exposure rather than describe it.

• Economics

  Inference economics and the flat-fee trap

  Consumption-based cost under seat-based pricing produces margin compression that scales with success. Where to find it in twelve months of provider invoices.

• AI Agents

  Diligencing AI agents

  Agent systems fail differently: escalation rates, tool-call error handling, and the gap between marketed autonomy and observed human review.

• Data Moat

  Does the data moat actually compound?

  Volume is not advantage. The test is whether measured task performance improves across retraining cycles, and whether labels are linked to outcomes.

• AI Governance

  AI governance questions that change deal terms

  Retention settings, sub-processor terms and training-use language routinely differ from the summary assurances given in management sessions.

Full write-ups are published periodically. To be notified, get in touch.