How to vet an AI/ML engineer when you're not technical yourself

01.1. Ask about a model that failed

Good engineers have clear stories about what went wrong and what they changed. Vague answers are a warning sign.

Ask: "Tell me about a model or AI feature that did not work. What happened?" Listen for specifics. A strong answer names the problem, the cause, what they tried and what they learned. Perhaps the training data did not match live traffic. Perhaps users asked questions the system was not built for.

If someone says nothing has ever failed, they have either not shipped much or are not being honest. Either is a concern.

02.2. Ask how they knew it worked

Listen for test sets, metrics and comparison with a baseline, not just "users liked it".

Follow up: "How did you decide it was good enough to release?" A strong candidate will talk about an evaluation set, the measure they chose and why, and what they compared against. They will also mention cases where the system was weaker. You do not need to understand the metric. You need to hear that one existed and that they can explain why it mattered.

03.3. Ask what happened after launch

Strong candidates talk about monitoring, drift and retraining. Weak ones stop at deployment.

Ask: "What did you monitor once it was live? What changed over time?" Experienced engineers will tell you about drops in quality, new kinds of input, rising costs or slow responses. They will describe how they noticed and what they did. Candidates who describe only the build phase may have handed over before the hard part.

04.4. Give a small, real problem

A short paid task on your data beats any whiteboard puzzle.

Pick a problem close to your actual work. For example, ask them to review a sample of your documents and outline how they would build a search feature, or to look at output from your current model and say what is going wrong. Keep it to a few hours and pay for it. What you want to see is how they think: what questions they ask, which risks they spot, and how clearly they write up a result.

05.5. Check communication

Can they explain a trade-off to a product manager in two minutes? You will need that every week.

Ask them to explain a technical decision as if you were the client. For instance: "Why would you pick a smaller model over a larger one?" The answer should be plain, short and tied to your goals: cost, speed, quality or risk. If you leave more confused than before, expect that in daily work as well.

06.A sample scoring approach

Score each candidate from one to four on six areas: failure story, evidence of measurement, post-launch experience, the paid task, communication and curiosity about your data. Write one sentence of evidence for each score. Avoid an overall gut score until after the evidence is on paper. When two candidates score alike, choose the one whose task output you would be happiest to inherit.

07.6. Use a partner who has already filtered

Our four-gate vetting passes about 3% of applicants, and every placement carries a 100% replacement guarantee.

A partner that screens for you reduces the number of weak interviews you have to run. The four gates are a technical test, a senior-engineer interview, a culture and communication check and a background check. You still interview the shortlist and decide. The guarantee covers the case where a placement does not work out.

08.Red flags worth noting

Some answers should slow you down. A candidate who cannot name a single thing that went wrong. One who describes only a model's accuracy and nothing about the data. One who blames earlier teammates for every failure. One who proposes the most complex approach first without asking about your constraints. Or one who claims to know every tool on a long list. Depth beats breadth in this field, and a good engineer will tell you where their experience ends.

09.Green flags worth noting

Look for the opposite. They ask what a wrong answer costs your business. They want to see your data before proposing a design. They suggest a simple baseline before anything complex. They talk about how they would hand work over, so you are not dependent on them. And they give you a plain "I have not done that, but here is how I would find out".

10.Using a non-technical reviewer well

If you cannot judge the technical answers yourself, borrow help without turning it into a long process. Ask a trusted engineer to listen to one interview and give you their view in ten minutes. Record the paid task output and have someone technical read it. Keep your own role focused on communication, judgement and fit, which you can assess well.

11.A quick interview checklist

• A failure story with cause and fix.
• A clear account of how quality was measured.
• Evidence of post-launch monitoring.
• A sensible approach to your small problem.
• Plain-language explanations.
• Questions about your data, which shows they know it matters.

12.What to listen for in each answer

On failure stories, listen for ownership. Strong candidates say "we" for successes and "I" for mistakes. On measurement, listen for comparisons: against what, over what period, on whose data. On monitoring, listen for specific signals, such as input changes, latency or cost per request. On the paid task, read the questions they asked before you read their answer. Good questions often reveal more than a good solution, because they show what the candidate believes matters.

13.Common mistakes

• Hiring on tool names alone.
• Skipping reference checks on past projects.
• Asking only theory questions.
• Rejecting candidates who say "it depends", when that is often the right answer.
• Judging confidence instead of evidence.
• Ignoring how they handle disagreement or uncertainty.

14.A note on contracts and trial periods

Start with a clear scope and a short review point, such as the end of the first month. Agree how performance will be tracked from the first day: tasks completed, quality of reviews, communication and whether the engineer unblocks others. If the fit is wrong, say so early. Early feedback is cheap, and late feedback is not.

15.What to do next

Write down the three outcomes you need from the hire. Prepare the six questions above and agree scoring with someone on your team, even if they are not technical. Compare answers across candidates.

If you want a shortlist that has already passed vetting, Brandsmashers can provide matched profiles within 72 hours of a scoping call.