A potential client recently sent me an AI-generated prototype of some software he was designing. It was impressive. It looked good visually. It showed the major pathways through the app. It brought the app to life more than any text document, and it triggered us to discuss several what-if scenarios, leading to some genuinely useful conclusions. More importantly he had used the prototype to develop commercial and funding relationships.

While reviewing the prototype, I had a look under the hood at how the AI had built it. The AI had chosen Supabase1 to store all its data. I asked the client about this choice, and he had no idea why it had been made, or even what Supabase was.

The AI made an important decision, and the human it was working for knew nothing about it.

In February 2026 researchers discovered that Moltbook, an AI-built app2, was insecure, exposing 35,000 email addresses and 1.5 million API keys. The AI used Supabase for this build too, but failed to configure its security settings at all.

Its creator, Matt Schlicht, has said “I didn’t write one line of code for @moltbook. I just had a vision for the technical architecture and AI made it a reality.” I think we can assume his vision for the technical architecture didn’t include a giant security hole, and that he didn’t know the AI had left one. It’s the same pattern, except this time with a decision that was not made.

The app I reviewed didn’t make that mistake, but Supabase was a poor choice for this scenario for a different reason.

The problem in this case was that the client wanted to launch into a highly regulated industry, and storing all their data on another company’s service would make passing their regulatory checks much harder. That’s not a critique of Supabase, which is a good product, just that it was a poor fit for this case.

A hand-drawn stick figure sits at a desk smiling at a computer screen, pleased with what they see. Hidden behind the screen, where they cannot see it, a robot holds up a clipboard with a checklist of four items: the first two are ticked, the last two are crossed out.
The app looks finished. The decisions behind it are out of sight.

So if an AI makes important decisions about your software without telling you, and can (and in my experience, often does) get those decisions wrong, how do you know what to trust it to build?

What AI is good at

AI-built software’s biggest impact is helping you work out, quickly and cheaply, what software will actually help your business.

Businesses only build software tools when they think the business will benefit from it somehow. Yet, the most common problem I see is when the software “works” but doesn’t actually help the business.

I’ve seen this in many forms:

  • staff members not using the software
  • staff and customers finding the software too limiting and finding workarounds
  • everyone uses the software, but it just doesn’t produce the business result it was expected to

In other words, cases of successfully building the wrong thing.

With AI, you can address this by building many different bits of software quickly and cheaply. And because this experimentation is so cheap, it no longer matters as much if one bit of software fails to deliver. You can try more and more ideas till you find the one that actually helps your business. Trial and error.

Sometimes it’s the first attempt that works. The prototype I mentioned above opened doors with customers and investors, driving the business forward.

I recently caught up with a former client who had used AI to build an idea of his own. He planned an app that would take over cumbersome bits of admin and paperwork, so his staff could focus more on the face-to-face part of their service, which was particularly critical in his business. He just described his idea to an AI coding tool (Claude Code in this case) over a weekend and was able to try it out the following week. Both staff and service users reported being much happier with how the face-to-face sessions felt.

Of course, sometimes it takes two, three (or more) attempts before you find the right software for your business.

The decisions you can’t see

The downside of AI-written code is that it can, and does, make poor decisions. My client’s prototype was built using a design choice that wasn’t bad but was a bad fit for his project. The Moltbook codebase ignored security, a bad choice for a live project going online, but an OK choice if you’re building a quick local prototype, or a tool that won’t be accessible online. My own experience shows the same pattern. Choices that are defensible in one context, but not right for the task I want done.

These poor decisions are what make coding with AI challenging. It’s also understandable. We are asking the AIs to balance multiple vague goals. The obvious goal is to do what we asked it to build, but the models come with other goals baked in:

  • make easy-to-use software
  • don’t burn too many tokens building it
  • make good software design choices
  • be ethical3
  • make the user happy4

Some of those goals are very ambiguous and subjective. Ask two software developers what counts as a good system design, and you’re liable to get at least three well-thought-out answers.

Having said that, no experienced developer would leave security turned off for a live site or choose Supabase for my client’s business. But he couldn’t have caught that, because he didn’t know there was a decision to catch, and so couldn’t judge if it would clash with rules in his industry.

He didn't know there was a decision to catch.

This is a great example of the “Knowledge Paradox” that Addy Osmani wrote about in The 70% problem: Hard truths about AI-assisted coding. If you already have the knowledge and experience to navigate these hard decisions, AI can help you by picking up the grunt work, making it fast to explore the trade-offs. But if you don’t have that knowledge, AI tools are notably less helpful. Non-developers are in that second group, which is fine, it’s not their job, but it’s a gap the AIs are not closing well. And there are consequences.

AI coding tools tend to make poor decisions about things you can’t see initially. Things like making your application safe from attack, able to cope with growth and change, compliant with the rules of your industry, and reliable about telling you the right thing.

Sometimes those are things that don’t matter for your project, and sometimes getting one wrong can sink a business. That’s why I prefer not to ask “Can AI build software?” but “What software is it safe to have an AI build without the help of an expert?”

What’s safe to let AI build

The answer is any project where none of these things matter. Which means one of two things: something you are going to throw away, or something that’s a tool only for you, or for a small group.

Prototypes are the clearest case. You can quickly build something, sometimes in a few hours, and find out how well it fits. Do people use it? Do they like it? Does it drive the sort of behaviour you expected? As long as they are throwaway prototypes, none of the ways AI tends to go wrong will bite.

If you can answer those questions over a few days or weeks, rather than over months or years, the result can genuinely change your business for the better. You can test out lots of ideas quickly and keep only the ones that lead to the results you want.

Your kitchen, or a restaurant

The other time AI-coded apps are useful is when it’s just for a small group. Robin Sloan came up with the metaphor of a home-cooked meal. When you are cooking at home, there are many things you don’t have to think about that would be critical if you were a professional chef in a restaurant.

A home cook only needs to make the food safe enough to eat for you and their guests; they don’t have to worry about passing safety inspections or allergen-aware menus. They don’t need their meals to taste the same, order after order, or to think about another chef reproducing their recipe if they ever want a night off. A professional chef’s reputation stands or falls on whether the food tastes good. If a home cook burns dinner, they can always order a pizza.

A hand-drawn kitchen scene. On the right a cook in an apron looks down, dismayed, at a smoking pan. In the middle four friends sit happily around a table with wine. On the left a delivery rider is already coming through the open door with a pizza box.
Dinner is ruined. Nobody minds.

Just as a home-cooked meal is often the cornerstone of family life, a home-cooked app can be a cornerstone of your business. As long as you keep in mind its limitations.

I have a few home-cooked apps that I use more and more. I have one that tracks many aspects of my business, helping me keep on top of admin and other dull-but-necessary tasks. I have another that guides me through learning guitar.

I don’t use a home-cooked app to analyse the results of medical tests and suggest treatments. That’s something where an error could be very bad (in an extreme case, literally life and death). Though I do use one to visualise some of my data, so I can see trends and look into the underlying data by eye.

My rule is that I only use these apps for things where it doesn’t matter if it goes wrong, and I can still do it myself by hand. I use them to save time, not to do things I couldn’t already do.

When to get help instead

I don’t let AI create an app for me if I need it to work, scale, be secure, and so on. In those cases I take the reins. I do let AI help with much of the laborious work, but I guide it and direct it on the decisions that I know to be important. That’s something I can do because of the years of experience I have building apps by hand. If you don’t have that, you should consider getting support from someone who can.

And beware of thinking you can just ship your prototype as a finished production app. That’s dangerous. It’s like taking a home-cooked meal and putting it on the menu at your local restaurant with no changes. It might work out fine, but so many things could go wrong. Maybe it uses expensive ingredients and you can only sell it at a loss, or it will trigger an allergic reaction in one of your customers.

Both my clients did this right. The one who sent me his prototype kept it as a prototype. The other, after proving that it helped his business, was able to work with an expert to produce a production version of it. In his case it was a small app, so the AI-coded version only needed small changes to make it production-ready.

So what is it sensible to ask AI to build? Two things: software you are going to throw away, and software only you or your team will ever use.

AI coding doesn’t save you from the important technical decisions. Ones where there are multiple answers, and which one is right depends on things the AI isn’t considering, or doesn’t know. For a prototype, or a home-cooked app, that isn’t too important. But when it’s software your business relies on, an entirely AI-built app can be not an asset, but a liability.

Everything else deserves a second pair of eyes before you rely on it. If you’re thinking about having AI build something for your business and you’d like a hand with it, get in touch — or have a look at what I do.

Footnotes

  1. Supabase is a company that stores your app’s data on their computers, allowing you to skip setting up your own data storage. Many developers choose to use it, and AIs even more so. ↩

  2. Rather bizarrely, Moltbook isn’t just a site written by AI, it’s a site for AIs. It’s a social network where AIs can get to know each other. Very odd, but that’s the world we live in now. You can check it out at www.moltbook.com ↩

  3. A lot of time is spent trying to make AI work in an “ethical” way, which is both difficult, and vague as most humans don’t agree what ethical is in the first place. ↩

  4. AI gets trained on how humans rate its answers. As we like answers that tell us we are great, AI gets trained to tell us we are great. ↩