TL;DR

Following are some things I’ve been coming across as I work, (and play), with some of our new tools. These days, whatever your level in Product, chances are you’re building or at least looking at something involving Agentic workflows or AI.

Given the increasing ease of use of new tools, you’re probably working directly with them. Hint: Even if you’re a CPO or VP, if you’re not getting your hands dirty, you’re similar to the CEOs as recently as the 1990s who were proud of the fact that they didn’t use email. Is that really the box where you’d like to constrain yourself? As with so many things, you don’t likely need to be an expert here. You just need to understand what the tools in your box can do and certainly some of the risks we face. Whether as Individual Contributor (IC), or senior manager, there’s risks and basic practices you need to have on your checklist that should be table stakes. Many of us now build with Low Code No Code or are VibeCoding our way to MVPs. Great. These tools are awesome for prototyping. For production though?

Watch out.

Stop here and just ponder that thought if you like. For details… read on. (Note that some of the upcoming examples are for any product person, whereas others are for more technical PMs or their developer partners. Regardless, they should give you a sense of what kinds of things you need to check off your list.)

tl;dr

Are you a builder? Probably if you’re here. Either way, we’re all getting carried along in several recent trends. As we race to automate everything with shiny new AI tools, we’re also stacking up fragile workflows and hidden dependencies, setting ourselves up for a quiet kind of chaos I’m calling Agent Rot. It’s like Link Rot, but for smart systems. The short answer is to just pay attention to this. For more, here’s why it’s happening, what could go wrong, and what we can do to keep our bots from forgetting what they were built to do.

Agent Rot will put Link Rot to shame. It’s not quite here yet, but I’m giving it a name. Because for now our burgeoning bright, shiny, new AI/GPT automation tools still have that “Fresh New Workflow Scent!” But what challenges will we face when they start to get creaky?

A couple of weeks ago I deployed a small WordPress website in support of a particular medical community. My goal was to test using WordPress vs. A Gen AI website builder and see how well a moderately technical product manager type, (but not a developer), could use some tools to make digital things. I kind of joked at the end that “I’ll probably toss in a GPT Chat feature because, well, I mean… it’s 2025.” I wasn’t actually going to bother, but, really now… it is 2025 after all. The site should have a GPT. So let’s have at it! (Here’s the end result by the way: ACL-GPT.)

tl;dr

• I wanted to test building a small website comparing WordPress to a custom AI generated site to support people with a medical issue.
• Bottom line is the generative site is easily competitive with WordPress, and very flexible.
• Final choice is to to go with the reliable WordPress option though; so I don’t have to worry about ongoing issues, but the generative site capabilities were somewhere between kind of cool and astounding. (I personally think it’s just going to be a relatively short time before I’ll be choosing to use them more.)
• The end results include the ACL support site generated with a Gen AI tool, (which is not content complete at this point), and the go to market site, aclsupport.com, which is managed hosting WordPress. (No attempt to make them match visually or with content; just get the basics up.)

That’s it! if you want the full story, here you go…

What are We Doing?

We’re going to look at a plaything I built in just a handful of hours while digging into agentic AI a bit just for fun. (Well, for career-related things as well, but mostly for fun.) The toy is an AI enabled Digital Product Manager assistant app where you can ask questions about product management. Which, as it turns out, could actually be a real product. I built it as a toy project for fun, but might actually soft launch this thing. Because, why not? Can you just do this with any GPT? Sure. But this one is tuned specifically to product management in general and digital product management in particular. (There are others like it though, so maybe just leave it as a personal tool. We’ll see.)

Why do this? Why bother. And why do you care? Since the vibe coding thing is so very in right now, (maybe for another few months), I figured it was time to jump in a bit. While I’ve built some agent workflows in the past and built a variety of apps, it’s usually been team based. This is one of those, “Hey, if I can do this… anyone can” type posts. The question, of course, is do you have a reason / use case? But the whole argument that used to exist about some things being “too hard” or “too technical” or “too much time” are fading away with some of these tools. Not completely. And some things absolutely – my opinion – require “real” developers. But others? Lower risk things? Personal productivity things? Not as much anymore. So I’m going to go through my process just at a super high level. My goal is to convince other product manager types to dive into this area more deeply than just watching a webinar and learning some of the lingo. Even for senior roles and beyond individual contributor roles, I personally think it’s useful to get a visceral feel for how things work. Doing so helps offer better context for what teams might be going through as well as understanding in what might be possible. And also, getting a sense of what budget implications might be if you’ve got P&L responsibility for a product.

Next we’ll get into the details, but if you want to see the end result, it’s here: (But note, the functionality won’t work as I’ve got the public webhooks turned off so as not to burn up my paid quotas on the services in use. To see it working, check the Loom Demo.)

Direct Link to the App Test Website

App Demo / Loom Video

In GenAI and Search: Differences from a Product & UX Perspective, we started looking at differences between search and GenAI from a user needs perspective. We looked at the tools in terms of use cases.

Now we’re going to turn more towards how we present things. This skips over everything in the Machine Learning Ops (MLOps) flow. And that’s okay. Because maybe we should be starting with the goal. After all, before you spend what could be millions, whether it’s for a consumer facing startup or an internal Enterprise tool, it’s probably wise to do some prototyping and testing anyway. (For MLOps, See: What is MLOps? (Amazon), What is MLOps? (Google), Why You Need MLOps.) By the way, I’m focusing here on user facing products, whether consumer or business. (As opposed to internal tools for analysis or production, marketing tools, etc. Though these certainly can have UX concerns as well.)

Figuring out how to design products to better serve users is of course not entirely your job. Whether you’re an entry level product manager or senior leadership, you’re ideally living more in the customer problem space. You’re looking across all things. Yes, you’re looking at features, functions, benefits. But also the dozens of other things to do. Which is why you work with your talented Design Leads. Whether direct reports or as a shared service, your design partners need to be getting up to speed on GenAI if they haven’t already. So your job is likely more along the lines of figuring out where or if AI is useful for your business; either for internal production or your actual products. And if it is, you might be the one – or at least be among the several – advocating for the resources to build out capabilities.

The classic question for Product Managers remains… How much do you really have to know in a specific domain; code, design, whatever? If you go too deep in any one area, chances are you won’t be very effective at your cross-functional tasks. This topic treatment is intended for skimming and basic understanding so you can work well with your talented design and tech colleagues. Our goal here isn’t to get anyone in Product to a practitioner level. It’s more to give you the tools to contribute effectively and have customer focused conversations with your specialist colleagues and drive requirements of value. Depth here once again depends on the type of product person you are. If you’re solidly on the business side, your whole product might be a P&L exercise for you and most functions and other team members are ‘just’ a line item on a spreadsheet as far as you’re concerned. A Technical PM? OK, you’re in the weeds with everything from APIs to whatever. Most of us are somewhere in between. One thing generally seems true though… if something comes up in a new domain where no one has clear responsibility yet, chances are you’re going to own it while an organization gets up to speed with where to put the new thing.

What is a Rubric for AI Products?

A rubric is about evaluation and quality control, but also standardization, consistency and more. The origin of rubrics is from education and assessment so the term may be new to a digital product person. The general idea is to have a highly structured way to evaluate qualitative judgments. This seemed to be somewhat parallel to what was needed to evaluate AI output, so the model was adapted for that purpose. Rubrics for AI evaluation are used in academia, by tech companies, and regulatory and standards bodies. For traditional development, we have a variety of QA standards. A lot of them involve unit and integration testing and in modern workflows is often part of a continuous development and deployment plan. Rubrics can also be used along a development path, during early evaluation and fine tuning, pre-deployment, and for ongoing testing. However, at least a rough model must be fully available.

In the case of AI model quality assessment, a rubric is a structured framework for evaluation.