Accuracy leadership that did not survive a held-out test
A growth investor was underwriting a document-processing company on a claimed accuracy advantage over incumbents, with win rates in the model built on that advantage persisting.
- Technical observation
- The published accuracy figure was produced on an internal evaluation set that overlapped the fine-tuning corpus; no frozen held-out set existed, and production traces showed a materially smaller margin on four of six task types.
- Business consequence
- The differentiation the sales motion is priced on is narrower than represented, and there is no internal mechanism that would have detected the erosion, so it can widen without anyone noticing.
- Investment implication
- Win-rate and pricing-premium assumptions require re-underwriting on the two task types where the advantage holds. An independent evaluation with a frozen test set should be a pre-close condition and a recurring covenant.
How the client used it
The investor renegotiated the entry multiple and made an independent evaluation programme a funded first-year initiative.