Most vendor evaluations fail because they rely on polished proposals and hand-picked references. This framework gives you seven criteria: language, AI-native capability, technical depth, process transparency, team stability, cultural fit, contract clarity, each with a question to ask, how to verify the answer, and the red flags to watch for.
Table of Contents
- Why standard vendor evaluation usually fails
- What is the 7-criteria evaluation framework?
- Seven criteria, the questions to ask, and the red flags
- How to use this framework in a real evaluation session
- Four common mistakes when evaluating a vendor
- When this framework is not enough
- Frequently asked questions
Why standard vendor evaluation usually fails
Most offshore vendor evaluation processes apply the exact formula used for domestic vendors: read the proposal, call a reference, compare pricing. That formula breaks down in three specific ways.
A proposal is marketing, not evidence. Any sales team good enough will produce a polished capability deck. What matters is what happens after the contract is signed, not before.
Reference calls are pre-filtered. The references a vendor volunteers are always their happiest clients. The real value comes from asking for references the vendor did not volunteer.
Price comparison misses the real total cost. A vendor quoting a lower rate can end up costing more once rework, extra management overhead, and the cost of misunderstood requirements are counted (none of which show up in the initial quote).
The seven criteria below are designed to surface exactly what a standard evaluation misses.
What is the 7-criteria evaluation framework?
The 7-criteria evaluation framework is a structured set of questions for testing a software development partner's real capability, rather than relying on what a proposal claims.
Each criterion has three parts: the question to ask, how to verify the answer (not just listen, but require evidence), and the red flag: an answer that signals the vendor falls short on that criterion.
Unlike an ordinary checklist that simply marks a capability present or absent, this framework forces the evaluator to verify, not just ask. That distinction matters most: a vendor can answer every question correctly and still fail a criterion if the answer comes with no verifiable evidence.
Seven criteria, the questions to ask, and the red flags
Criterion 1. Language: don't take their word for it, verify it
Question: Does the team actually building your product communicate directly in your language, or does "language support" just mean one account manager who translates?
How to verify: Request a direct technical discussion with the actual engineers assigned to your project, not the sales team. Ask the tech lead to explain a recent technical decision, including why an alternative was rejected.
Red flag: "We have a PM who speaks your language" (a translation layer, not direct communication). Every technical question routes back through sales before you get an answer.
Criterion 2. AI-native capability: an occasional tool, or a standard workflow?
Question: Is AI use a habit a few individual engineers happen to have, or is it the team's standard operating practice?
How to verify: Ask which specific AI tools are in use, not a vague "we use AI." Request a concrete example of how AI is used in automated test generation or code review.
Red flag: "We use ChatGPT for documentation" is the only answer given, with nothing concrete about testing or design. Only a few engineers use AI tools; it is not yet a team-wide standard.
Criterion 3. Technical architecture depth: can they think, or only implement?
Question: Does the team have senior engineers capable of making architecture decisions, or only people who execute exactly what they're told?
How to verify: Present a real architecture problem and ask for 2-3 approaches with trade-offs for each. Ask about the hardest architecture decision they've made, and which alternatives were rejected.
Red flag: The proposed solution always matches your initial spec exactly, with no pushback and no alternatives offered.
Criterion 4. Process transparency: can you actually see what's happening?
Question: When quality or progress starts slipping, do you find out before it becomes a crisis?
How to verify: Ask specifically what happens when a sprint goal is missed. Request to see an actual retrospective record (anonymised is fine) from a past project.
Red flag: "We send weekly reports" is the only answer, with no mention of specific metrics or a retrospective mechanism.
Criterion 5. Team stability: who is actually staying on your project?
Question: Will the team on your project six to twelve months from now still be the same people who started it?
How to verify: Ask for average engineer tenure on currently running long-term projects, not a company-wide average. Ask whether handover is documented in writing when an engineer leaves a project.
Red flag: A vague answer about tenure. No documented handover process, meaning knowledge lives only in individual heads.
Criterion 6. Cultural fit: a partner, or just a contractor?
Question: Does the team proactively raise problems and suggest improvements, or does it wait for instructions?
How to verify: Ask about a time they disagreed with the client's technical direction, and how they handled it. Ask what they do when they spot an issue outside the assigned scope.
Red flag: They've never once pushed back on a client. Success is defined purely as "delivered on time," with no mention of business outcomes.
Criterion 7. Contract clarity: is the risk split fair?
Question: When something goes wrong, are the contract terms enforceable and fair to both sides?
How to verify: Ask specifically how IP ownership is defined. Ask whether they accept milestone-based payment. Ask how the contract handles rework costs caused by a misunderstood requirement.
Red flag: IP ownership is vague or requires an extra fee. They refuse to discuss milestone payments at any stage. Every risk sits on the client's side.
How to use this framework in a real evaluation session
The framework works best as a script for one direct conversation with the actual people who will work on your project, not a questionnaire emailed to a sales team.
Suggested order: start with criterion 1 (language) and criterion 3 (technical depth). These two surface the gap between proposal and reality fastest, usually within the first 15-20 minutes. If either shows a clear red flag, it's reasonable to end the evaluation early rather than working through all seven.
Write answers down in the room, during the conversation, not from memory afterward. Comparing notes across multiple vendors side by side surfaces differences far more reliably than evaluating each vendor separately over time.
Four common mistakes when evaluating a vendor
Only talking to sales, never meeting the actual engineers. Consequence: every answer has already passed through a marketing filter and doesn't reflect the real capability of the team who will do the work. Avoid it by requiring the assigned engineers to be present at the first meeting, as a condition of the evaluation.
Treating the lowest price as the deciding factor. Consequence: rework and management overhead get ignored, and they usually outweigh the initial price gap. Avoid it by using criterion 7 (contract clarity) to estimate hidden cost risk before comparing quotes.
Asking the questions but never requiring evidence. Consequence: you get answers that sound right but can't be verified. Avoid it by always pairing each question with its "how to verify" step, not stopping at the question itself.
Evaluating once and never following up. Consequence: a vendor performs well in the evaluation but doesn't sustain it a few months in. Avoid it by folding criterion 4 (process transparency) and criterion 5 (team stability) into recurring project check-ins, not just a one-time vendor-selection exercise.
When this framework is not enough
The 7-criteria framework filters out weak vendors, but it has three real limits worth knowing upfront.
It doesn't replace actual legal review. Criterion 7 helps you ask the right contract questions, but it doesn't substitute for a lawyer reviewing governing law, jurisdiction, and any industry-specific compliance requirements your project needs.
It doesn't measure capability in an industry you haven't tested. A vendor can score well on all seven criteria and still have never worked in your specific regulated industry (healthcare, finance, manufacturing with its own compliance rules). Ask for a concrete case in your exact industry rather than inferring it from general capability.
It doesn't fit a decision that has to be made in a few hours. This framework needs at least one 45-60 minute direct conversation with the actual project team to deliver real value. If your timeline is too tight for that, the framework won't do what it's meant to do.
Frequently asked questions
Do I need to cover all seven criteria in one meeting? No. Start with criteria 1 and 3. A clear red flag on either is enough reason to end early. If both look solid, continue through the rest in the same session or a follow-up.
A vendor refuses to let engineers join the call, offering only a PM instead. Is that a bad sign? Per criterion 1, yes, it's a red flag worth noting. It doesn't automatically disqualify them, but it's worth asking directly why, and weighing the answer carefully before deciding.
Does this framework apply to domestic vendors too, not just offshore? Yes. The seven criteria don't depend on a vendor's location. The "great proposal, different reality" problem happens with domestic and offshore vendors alike.
What if a vendor fails one or two criteria but is strong everywhere else? No single criterion is an automatic disqualifier, except a serious red flag on criterion 7 (all contract risk sitting on the client). Weigh it holistically, but prioritise whichever criterion matches your project's biggest actual risk.
At what stage of vendor selection should this framework be used? After you've already shortlisted 2-3 candidates from an initial proposal review. Use this framework for the final direct-interview round, not the first screening pass, since it takes more time than reading a proposal.
Conclusion
- Standard vendor evaluation (proposals, references, price) usually fails on three points: proposals are marketing, references are pre-filtered, and price comparisons miss the real total cost.
- The 7-criteria framework forces verification, not just questions: each criterion pairs a question with a verification step and a red flag.
- The framework doesn't replace legal review, doesn't measure industry-specific capability, and needs real direct conversation time to deliver value.
If you're evaluating software development partners and want to run this framework in a real conversation, book a consultation with VAON: we'll answer against this exact seven-criteria structure, including the parts we haven't gotten right yet.
