Judge the method, not the demonstration. Every agency you speak to will be able to show you something impressive. Very few will be able to tell you, in specific terms, how they decide what to build, who owns it afterwards, and what happens when the first thing they try does not work. That second set of answers is what separates the ones who deliver from the ones who present.
Fair warning before you read on: we run one of these, so discount this accordingly. We have tried to write the version we would want if we were the ones buying, including the parts that do not flatter us.
Why is this a commercial decision rather than a technology one?
Because the failure mode is operational, and you carry it.
An engagement that stalls halfway does not just waste the fee. It occupies your people, produces half-integrated processes nobody trusts, and makes the next attempt harder because the organisation now has a story about AI not working here. That cost lands on you and it outlasts the invoice.
Writing in Harvard Business Review in November 2025, Jin Li, Feng Zhu and Pascal Hua put the pattern plainly: firms struggle to capture real value from AI “not because the technology fails” but because “their people, processes, and politics do.” That is worth holding in mind while you evaluate people selling you technology.
Which is why the selection criteria that matter are mostly not technical. Can they scope tightly? Will they tell you when something is a bad idea? Does the work survive them leaving? Those questions predict outcomes better than any model comparison, and they are harder to fake in a first meeting than a demonstration is.
What separates real capability from a good pitch?
Four things, and none of them show up in a slide deck.
How they choose. Ask an agency to walk you through how they decide which of your workflows to touch first. A strong answer is specific and unglamorous: they will want to know what you already measure, where work gets redone, which jobs are frequent and structured. A weak answer talks about possibilities and use cases in general terms. This one question sorts the field faster than anything else.
Whether the work outlives them. Ask what your team is able to run and change after the engagement ends. Some arrangements are designed so you cannot maintain anything without the agency. That is a business model rather than a delivery failure, but you should choose it knowingly instead of discovering it later.
Ownership. Every automated process needs a named person on your side who is accountable for the output being right. If an agency has not raised this by the second conversation, they have not implemented much.
What they will not do. A partner who cannot name anything they would decline is either inexperienced or selling. The most useful question in the whole process is: where do you think AI is the wrong answer for us?
What are the trade-offs between the different kinds of firm?
Three broad models, each genuinely good at something different.
| Type | What they are strong at | Where they struggle |
|---|---|---|
| Large enterprise consultancies | Complex multi-year programmes, organisations with internal AI teams and formal governance, board-level assurance | Cost and pace. Their delivery model assumes a scale most Australian businesses do not have, and you rarely get the people from the pitch |
| Technical AI labs and dev shops | Custom model work, genuinely novel technical problems, deep engineering | Commercial framing and adoption. Strong builds that never get used, because the workflow and training side was somebody else’s job |
| Focused specialist agencies | Applied work inside existing processes, speed, defined scope, adoption | Depth on genuinely hard technical problems. If you need a custom model trained, this is the wrong door |
The honest read: most small and mid-sized Australian businesses need the third, occasionally the second, and almost never the first. The services a specialist actually offers should map to workflows you recognise rather than to technology categories. But if you are in a regulated sector with a board that needs formal assurance, the first exists for a reason, and a specialist telling you otherwise is selling.
What should I actually ask?
Take these to every conversation. The answers are more revealing than any case study.
- How do you decide what to build first? Listen for whether they ask about what you measure today.
- Where is AI the wrong answer for us? Silence here is disqualifying.
- What happens if the first project does not work? Everyone has one. The good ones tell you without being pushed.
- Do you resell or earn margin on any platform you would recommend? Not automatically bad. Undisclosed is.
- Who owns this internally when you leave, and what can they change?
- What does the scope end look like? A defined end with a fixed fee focuses the mind. Open-ended arrangements remove the pressure to produce anything.
- What have you declined recently, and why?
Fair questions to be asked in return: what you already measure, who would own this internally, and what you have tried before that did not work. If nobody asks that last one, they are not really listening.
Why do pilots so often fail to scale?
Because a successful pilot and a working process are different achievements, and the gap between them is organisational rather than technical.
A pilot is run by motivated people, on a case they chose, with someone watching. Daily operations have none of those conditions. What closes the gap is unglamorous: standard inputs, a named owner, a defined review rhythm, training that shows people their own job, and a decision point where you either keep it or stop.
Ask any prospective partner how they handle that transition specifically. If the answer is essentially “we hand it over,” you will be doing the hard part yourself. More on how we structure that in Audit, Build, Partner.
How much should governance factor into the decision?
More than most buyers think, and less than most vendors sell.
For a business of your size, governance means writing down a handful of answers: what customer information can go near these tools, who is accountable for the output, what happens when something is wrong, and who is allowed to change the setup. That is a page or two of plain language.
Two reference points if you want them. The NIST AI Risk Management Framework and the OECD AI Principles both cover transparency, accountability and risk management, and a competent partner will be able to talk about them without reaching for a brochure. Your version should be shorter and more specific than either.
The practical reason to care is not compliance theatre. It is that teams use these tools cautiously when nobody has told them what is allowed, and cautious use produces no return. Clear rules get people using the thing.
What should I avoid?
- Choosing on brand. A recognisable name tells you about their marketing, not about who will actually do your work.
- Buying the demonstration. Demonstrations are built to succeed. Ask what it took to build, and what it would take on your data.
- Accepting an open-ended engagement because it feels flexible. It removes the deadline that makes results happen.
- Skipping the reference call. Ask to speak to a client where something went wrong. The refusal is the answer.
- Assuming a proposal is a plan. A document that has not been tested against one of your real workflows is a hypothesis.
- Treating price as the variable. The expensive mistake is not the fee, it is a year spent on the wrong project.
How should I run the selection?
Keep it short. This does not need a procurement process.
- Write down the problem, in your own words, before you speak to anyone. “Quotes take three days and we lose work” is workable. “We should be doing something with AI” will get you sold to.
- Speak to three, of different types, so you can hear the difference in how they answer.
- Ask the same questions to each and compare the answers rather than the presentations.
- Take a reference call, and ask specifically about something that did not go to plan.
- Scope the first piece small enough to judge within weeks, with a fixed fee and a defined end. If a proposed first step cannot show you anything for a quarter, ask why it is that large.
- Agree what “it worked” means before starting, in a number you already track.
If a first conversation does not leave you with a clearer view of your own problem, that tells you something regardless of what they are selling. The people worth hiring tend to be useful before you have paid them anything.
Ours is a discovery call, and the FAQ covers what usually comes up first.
FAQ
How do I choose an AI consulting agency in Australia?
Judge the method rather than the demonstration. Ask how they decide what to build first, where they think AI is the wrong answer for you, who owns the result internally, and what the scope end looks like. Speak to three firms of different types and compare the answers to identical questions.
What questions should I ask an AI consultant before hiring them?
The most useful is where they think AI is the wrong answer for your business. Then: what happens if the first project fails, whether they earn margin on any platform they would recommend, what your team can run and change after they leave, and what they have declined recently.
Should I hire a big consultancy or a specialist agency?
It depends on scale and assurance needs. Large consultancies suit organisations with internal AI teams and formal governance requirements. Most small and mid-sized Australian businesses get better value from a specialist working inside their existing processes, because the work is faster, cheaper and more likely to be adopted.
How much should an AI consulting engagement cost?
The structure matters more than the number. Look for a defined scope with a fixed fee and a fixed window rather than an open-ended retainer. If the first step cannot be judged within weeks, it is scoped too large.
Why do AI pilots fail to scale into production?
Because a pilot runs under conditions daily operations do not have: motivated people, a chosen case, and someone watching. Scaling needs standard inputs, a named owner, a review rhythm, and training. Ask any prospective partner how they handle that transition specifically.
Do I need to worry about AI governance at my size?
Yes, but less than vendors sell. A page or two covering what data is allowed near these tools, who is accountable, what happens when output is wrong, and who can change the setup. The NIST AI Risk Management Framework and OECD AI Principles are reasonable reference points, but your version should be shorter and more specific.
What are the warning signs of a bad AI consultant?
They cannot name anything they would decline. They lead with technology rather than asking what you measure. They will not disclose platform commercial relationships. They resist a defined scope end. And they cannot give you a reference where something went wrong.
