20 Questions to Ask Before Hiring a Development Agency

Ask five agencies the same five questions and you will get five confident, incompatible impressions. Gut feel fails at exactly this job — it converts charisma into trust and quiet answers into doubt. For a 35-employee insurance brokerage we will call Meridian Risk Partners (an illustrative example we will follow throughout), the fix was arithmetic: 20 questions, five themes, one 0-2 score each, 40 points on the table.

This guide is that question sheet, with the answers that should encourage you, the answers that should end the conversation, and the rubric that turns two calls and a proposal into a number you can defend. We use the same structure when clients ask us to judge a shortlist. Every number is attributed to a published source or marked as an illustrative model, like Meridian.

Here is the short version.

Key Highlights

• 20 questions across 5 themes — process, team, ownership, communication, post-launch — scored 0-2 each for a 40-point sheet.
• Only about 31% of software projects finish fully successful (The Standish Group CHAOS research); structured vetting is how you buy better odds.
• In the modelled brokerage vetting, five finalists scored 36, 33, 28, 24 and 17 of 40 — the sheet ended two conversations inside a week (illustrative).
• "We will assign your team after signing" is the most common auto-disqualifier, surfacing in roughly a quarter of modelled failed vetting rounds (illustrative).
• Repo, code and IP in your organization from day one — two of five modelled finalists resisted, and both were cut for it (illustrative).
• High communication scorers held a 3-5 hour median written reply in month three; low scorers drifted to 19 hours (illustrative).
• Bands that make it actionable: below 24 of 40, walk away; 35 and above, hire-worthy; in between, probe or buy a trial.

Why a Score Sheet Beats a Gut Feeling

Unstructured vetting has two failure modes. The first is halo effect: one slick demo colors everything else the agency says. The second is inconsistency: you ask agency A about QA and agency B about pricing, then compare notes never collected on the same form.

A score sheet fixes both. Same questions, same order, same 0-2 scale for every candidate — which turns the debrief into a comparison of evidence instead of a contest of impressions. Meridian's operations lead and office manager scored the five finalists independently, then reconciled; their two disagreements were both resolved by replaying recorded answers.

The sheet does not replace judgment — it organizes it. The signed agency led or tied on every theme, but the deciding gap was ownership, where the median rejected finalist scored 3 of 8. That gap was invisible in the pitches and obvious on the sheet.

Key takeaway: Same questions, same order, same scale, every candidate — a score sheet converts charisma into arithmetic and disagreements into evidence.

Group 1: Process Questions (1-5)

Process questions test whether delivery is a system or an improvisation. Listen for artifacts — named phases, written reports, checklists — and discount adjectives. "Agile" without named ceremonies is a word, not a process.

Process evidence shows up in the references, not just the interview. Across the modelled reference projects we use for training, agencies that produced written sprint reports delivered 88% of sprint commitments on time, against 64% where updates were verbal and 41% where no sprint cadence existed. Ask the question in the interview, then verify it with one reference call.

Key takeaway: Ask for artifacts, not adjectives — a sample sprint report tells you more about delivery than any methodology diagram.

Group 2: Team Questions (6-10)

Team questions test whether the people you met will actually build your project. The pitch team and the delivery team are different departments at more agencies than owners expect. Every good answer here comes with names.

Question 9 rewards agencies that measure themselves. Industry surveys commonly put annual software-developer turnover in the 10-20% band, so a straight answer near that range is credible — and a refusal is an answer of its own. Turnover is not disqualifying; hiding it is.

Continuity is the hidden variable in every delivery timeline. When a lead developer leaves mid-build, the replacement spends weeks learning code they did not write, and the schedule absorbs the difference. Meridian's eventual partner answered the team questions with names, dates and a voluntary org chart — the only finalist that did.

Key takeaway: Collect names, tenure and turnover before you fall for the pitch — the team that signs you is not always the team that builds for you.

Group 3: Ownership Questions (11-14)

Ownership questions decide what you are left holding if the relationship ends — and relationships end for reasons that have nothing to do with software. These four answers should be unambiguous enough to enforce from the contract alone.

In the modelled vetting rounds, fewer than four in ten projects started with the repository in the client's organization — and the agency-repo arrangement is where hostage situations are born. Make day-one ownership a written condition before the first call, not a negotiation after the quote. Meridian eliminated two of five finalists on this single theme.

The same logic covers everything the agency touches: cloud accounts, app-store listings, domain records, analytics properties, third-party licenses. If it is not in your organization or transferable to it, someone else holds leverage over you. The healthcare patient app we built ships with compliance-sensitive data rules that only worked because the infrastructure lived in the client's accounts from the first commit.

Key takeaway: Day-one ownership of the repo, cloud and IP is the single most important answer on the sheet — resist it once and you will negotiate it forever.

Group 4: Communication Questions (15-17)

Communication failures rarely announce themselves during the sales process, because sales is the one phase where agencies are maximally responsive. These three questions make the operating rhythm explicit and testable instead of assumed.

The modelled reply-time curve shows why the question matters more than the answer. High scorers promised one business day and held a 3-5 hour median through month three; low scorers promised the same thing and drifted to 19 hours as the project stopped being new. Ask how the promise survives month three — dedicated slots, escalation names, response-time reporting — not just whether it exists.

Key takeaway: Responsiveness is a system, not a personality trait — demand named contacts, standing demo slots and an escalation path before you need them.

Group 5: Post-Launch Questions (18-20)

Post-launch is where the cheapest surprises live, because the contract is already signed by then. Three questions close the most expensive gaps. Ask them in the first interview rather than the final one — the answers shape the quote.

Warranty terms double as an honesty probe: agencies confident in their QA quote 30-90 days without flinching, while agencies that skip testing charge for the same bugs twice. Maintenance answers reveal the same character — an agency that says "we will figure it out after launch" has just told you its post-launch plan is improvisation. The voice-AI ordering system we run for restaurant clients carries a standing support arrangement that was negotiated into the original contract, not discovered after it.

Key takeaway: Launch is the middle of the relationship, not the end — price the first 90 days and the first year before you sign, not after.

How to Score the Answers: The 40-Point Sheet

Score every answer 0, 1 or 2 immediately after each call while the recording is fresh. The scale is deliberately blunt: 0 is a red flag or a refusal, 1 is a vague or partial answer, 2 is a specific answer with evidence. Twenty questions at two points each gives a 40-point maximum.

Interpret the totals against bands, not vibes. Below 24 of 40, walk away regardless of price. From 24 to 29, the risk outweighs the save unless a trial changes your mind; 30 to 34 is hireable with named gaps written into the contract; 35 or more is hire-worthy. In the modelled brokerage vetting, the five finalists scored 36, 33, 28, 24 and 17 — the sheet shrank the trial budget to two candidates in a week.

The band-to-outcome relationship in our modelled projects is the argument for the whole exercise: launch reliability climbs from 42% to 93% as scores rise from below 24 to 35 and above. Then verify the top scores where verification is cheap — call two references per finalist and ask what went wrong as well as what went well, and run a paid one-to-two-week trial on a real slice of the project. The headless ecommerce storefront we delivered began as exactly such a trial; the contract clauses that protect what the trial proves are covered in our guide to choosing an app development partner.

Key takeaway: Score 0-2 after every call, act on the bands, and spend your verification budget — references and a trial — only on the scores worth verifying.

Red Flags That Override Any Score

Some answers end the process no matter how the rest of the sheet reads. These are the disqualifiers we treat as non-negotiable in the vetting rounds we run for clients:

• Resistance to putting the repository and cloud in your organization from day one.
• Unwillingness to name the team before signing.
• No demo cadence, or demos "when there is something to show".
• "We have never had a failed project" — no honest veteran says this.
• Warranty or post-launch support priced only after launch.
• More than 40% payment upfront, or any demand for full prepayment.
• Pressure to sign within days, especially with a discount that expires.

Treat the list as arithmetic, not judgment. An agency that trips one disqualifier may still be the right partner for someone else's project — but the fix always reduces to the same clause or the same promise you would have to extract under pressure later. The 40 points measure quality; the disqualifiers measure risk, and risk sets the floor.

Key takeaway: A high score never rescues a disqualifier — walk-away answers are the floor, and no ceiling justifies a floor that is missing.

What Your Business Should Do

Here is the 90-day sequence, from candidate list to signed contract.

Days 1–30: Build the sheet and the long-list.

• Adapt the 20 questions to your project, write your preferred answers down, and set your auto-disqualifiers in advance.
• Write or refresh the project brief the questions assume — goal, scope, constraints, budget band.
• Build a long-list of 12-20 agencies across at least two supplier classes and send the brief.

Days 31–60: Interview and score.

• Run two interviews per finalist — process and team first, then ownership, communication and post-launch — and record both.
• Score 0-2 after every call, and reconcile independent scores before impressions harden.
• Eliminate on disqualifiers first, then on bands; probe every 30-34 score in writing.

Days 61–90: Verify, trial, sign.

• Call two references per top scorer and ask what went wrong, not only what went well.
• Run a paid one-to-two-week trial on a real slice of the project for the top two.
• Sign with IP and day-one repository ownership, milestone payments, a 30-90 day warranty and a written change-order process.

The Bottom Line

Twenty questions will not make an agency good, but they will make the difference between agencies measurable — and measurable is most of the battle. Score every answer, act on the bands, honor the disqualifiers, and the pitch stops being a performance and becomes a preview.

If you want a second pair of eyes on a shortlist, a quote or a scored sheet, that is a fast conversation for us. Book a free strategy call and we will pressure-test your candidates with you — or send us a message and we will respond within one business day.