Nine AI Vendor Red Flags Worth Walking Away From

The single most reliable AI vendor red flag is refusing to run the demo on your data. Everything else on this list is a variation of the same underlying problem: a product that performs well under conditions the vendor controls and poorly under yours.

The nine

1. The demo only runs on their data. Every capable vendor will run a pilot on a sample of your worst inputs. One that resists has usually tuned the demo to a dataset chosen for the purpose. The vendor evaluation framework covers the protocol.

2. No baseline comparison offered. If a vendor cannot tell you what to measure against, they are not planning to be measured. Ask what your current error rate or handling time is, and how they would know if the product beat it.

3. Vague model claims. "Proprietary AI" that cannot be described at even an architectural level is usually someone else's model with a wrapper. That is not disqualifying in itself — but a vendor unwilling to say so is telling you how the rest of the relationship will go.

4. Accuracy stated without conditions. Any accuracy number is meaningless without the dataset, the task and the measurement method. A vendor quoting a single figure across all customers is quoting marketing, not measurement.

5. Your data trains their model, by default. Sometimes acceptable, often not, always a decision you should make consciously. Check the default setting rather than the sales assurance, and get the answer in the contract.

6. No exit path. Ask how you would get your data out, in what format, and what happens to derived artifacts. A vendor without a clear answer has built lock-in into the product, whether deliberately or not.

7. Pricing that scales on something you cannot predict. Per-token, per-task or per-seat pricing is fine when you can forecast the driver. When the driver is a volume nobody has measured, you have signed an open-ended commitment.

8. Security answers that arrive from sales. Data handling questions should be answerable by someone technical, in writing. If a security questionnaire is treated as an obstacle rather than a routine step, the compliance posture behind it is thin.

9. Pressure tied to a discount deadline. Real enterprise software gets sold on fit. A discount that expires this quarter is a statement about the vendor's sales cycle, not about your readiness.

What to do with a red flag

Not every one of these is fatal. Numbers 3, 7 and 9 are often negotiable and sometimes just clumsy sales practice. Numbers 1, 5 and 6 are structural — they describe how the product and contract are built, and they will not improve after signature.

The disciplined version of this is a written evaluation method applied to every vendor equally, which is what the AI vendor evaluation framework sets out. For data-handling questions specifically, NIST's AI Risk Management Framework is a reasonable public reference to anchor your questionnaire against, so you are not inventing the standard yourself.

The meta red flag

If the vendor conversation is happening without anyone who will own the outcome in the room, the evaluation is already compromised. Someone has to be accountable for whether this purchase worked a year from now — which is precisely the gap a fractional Chief AI Officer fills in companies too small for a full-time seat.

Frequently asked questions

What is the biggest AI vendor red flag? Refusing to run a pilot on your own data, especially your messiest examples. Demos are tuned to datasets chosen to flatter the product, so a vendor unwilling to test on your inputs is protecting the demo.

Should I be worried if a vendor uses another company's model? Not necessarily — most do, and it is often the sensible choice. The concern is a vendor unwilling to say so, because that unwillingness predicts how forthcoming they will be about everything else.

What contract terms matter most for AI tools? Data ownership, whether your data trains their models by default, export format on exit, and what happens to derived artifacts. Negotiate these at purchase, when you still have leverage.

How do I evaluate accuracy claims? Ask for the dataset, the task definition and the measurement method behind the number. A single accuracy figure quoted across all customers describes marketing rather than performance on your data.

Further reading

• The AI vendor evaluation framework
• Build vs buy AI
• The 90-Day AI Playbook (free PDF)
• AI strategy and advisory