This three part series is not a warm, fuzzy post about some AI strategic theater, though there are both strategic and tactical issues included. It’s a walk down some paths regarding practical issues. It’s about some of the challenging practical realities we face once product people try learning about or making tools work in actual workflows. You may feel some of my personal scar tissue in some of these passages. This is for semi-technical product people working with or learning several AI tools and dealing with some of the practical gotcha’s in making them go. It’s also raising a hand and calling BS on the spew of feed drivel about how “all you have to do is just set all this stuff up and crap magically happens.”

What’s Ahead? failures, data, workflow ops, context and prompting, evals, governance and kill switches, costs, complacency, and more, mostly related to smaller or mid-sized projects; though some ideas apply to all. This is a rather long series, even for me. But when I study and do things, I tend to go deep. As is often the case, these posts are really based on my notes to myself in my personal wiki over time, cleaned up somewhat and posted here to share.

These tools are great and I really enjoy them, even if there are some challenging spots, which we’re about to explore. Some of this stuff feels magical! And fun if you have that attitude. It really can be just fun and satisfying to see a workflow executing well if you’re working hands on. However, issues can quickly become a major hassle. It’s rarely as it is in so much of the feed fawning I see. On the surface, it looks like product managers, all of us really, are being sold a fantasy of frictionless AI tooling. It makes me wonder if some of these folks are actually using these tools for real. My own experience, and interacting with others or via Reddit and so forth, shows a usually more challenging and meandering path. That’s ok. I get it. The Happy Path is easier to write about and slap together a LinkedIn post or YouTube or whatever. Unfortunately, it also obscures some likely realities.

Here’s my message as we drive through the messy parts. You’re ok. It’s not you. These things can still be a little sloppy. Just keep pushing through.

Time to step away from the AI hype and look at real costs for everyday AI use cases. This article focuses on internal tooling and operational spend, not so much more exciting user facing products; though the same cost dynamics apply.

My motivation here is I keep hearing AI will let teams “vibe code” their way out of SaaS, but AI can also turn the resulting products’ predictable Operating Expenses (OpEx) into volatile, usage-based costs. So I want to think through ongoing variable costs. Not ROI or quality risk, which matter too. Or the reality that AI talent may still be hard to find; just really more ongoing variables costs, which are different than our more predictable past tools.

Spoiler Alert: I’ll run through the thought process and offer up a spreadsheet. But the bottom line is this choice can be very business specific. Building your own with AI can easily spin up costs faster than you’d expect, as can burning tokens with SaaS solutions that add AI. But even speedier app development with AI assisted coding might cost you more than SaaS. Scroll down and just grab the spreadsheet or read on for all the details. It’s about more than just tokens vs. seats.

Quick Summary: SaaS often seems pricey per year, while “vibe coding” your own replacement can look cheaper early but gets expensive once you count labor, operations, and risk. The biggest drivers are seats, interaction complexity, and hidden costs like evals, testing, and ongoing auditability. DIY can win for small, short efforts, but at enterprise scale SaaS can still be cheaper on TCO even if the subscription line stings. In either case, for AI enabled products, there’s inference/token and other costs. Here’s the sheet if you’re skipping the rest of this article. (Note that older non-AI enabled build vs. buy examples are there just for historical reference and comparison.)

• On Scott’s Github
• In Scott’s Google Sheets

Also note that the cost assumptions I’ve put in here to start are radically higher than what simple token pricing might be compared to what you’ll find on a vendor’s chart. I’ve tried to add blended costs for things that reflect real production costs, like API costs, vector database queries for RAG, caching and so on. The point is for you to plug in your own numbers. If you want you can split out more granular costs to their own lines.

And, oh yes… don’t forget token costs are not the whole story. This piece is focused on costs, but when you shift to pricing and ROI, it’s worth reading John Rowell’s Context Is the Next Frontier in AI Economics.