tl;dr

Are you a builder? Probably if you’re here. Either way, we’re all getting carried along in several recent trends. As we race to automate everything with shiny new AI tools, we’re also stacking up fragile workflows and hidden dependencies, setting ourselves up for a quiet kind of chaos I’m calling Agent Rot. It’s like Link Rot, but for smart systems. The short answer is to just pay attention to this. For more, here’s why it’s happening, what could go wrong, and what we can do to keep our bots from forgetting what they were built to do.

---

Agent Rot will put Link Rot to shame. It’s not quite here yet, but I’m giving it a name. Because for now our burgeoning bright, shiny, new AI/GPT automation tools still have that “Fresh New Workflow Scent!” But what challenges will we face when they start to get creaky?

[Read more…]

A couple of weeks ago I deployed a small WordPress website in support of a particular medical community. My goal was to test using WordPress vs. A Gen AI website builder and see how well a moderately technical product manager type, (but not a developer), could use some tools to make digital things. I kind of joked at the end that “I’ll probably toss in a GPT Chat feature because, well, I mean… it’s 2025.” I wasn’t actually going to bother, but, really now… it is 2025 after all. The site should have a GPT. So let’s have at it! (Here’s the end result by the way: ACL-GPT.)

[Read more…]

tl;dr

• I wanted to test building a small website comparing WordPress to a custom AI generated site to support people with a medical issue.
• Bottom line is the generative site is easily competitive with WordPress, and very flexible.
• Final choice is to to go with the reliable WordPress option though; so I don’t have to worry about ongoing issues, but the generative site capabilities were somewhere between kind of cool and astounding.
• The end results include the ACL support site generated with a Gen AI tool, (which is not content complete at this point), and the go to market site, aclsupport.com, which is managed hosting WordPress. (No attempt to make them match visually or with content; just get the basics up.)

That’s it! if you want the full story, here you go…

[Read more…]

What are We Doing?

We’re going to look at a plaything I built in just a handful of hours while digging into agentic AI a bit just for fun. (Well, for career-related things as well, but mostly for fun.) The toy is an AI enabled Digital Product Manager assistant app where you can ask questions about product management. Which, as it turns out, could actually be a real product. I built it as a toy project for fun, but might actually soft launch this thing. Because, why not? Can you just do this with any GPT? Sure. But this one is tuned specifically to product management in general and digital product management in particular. (There are others like it though, so maybe just leave it as a personal tool. We’ll see.)

Why do this? Why bother. And why do you care? Since the vibe coding thing is so very in right now, (maybe for another few months), I figured it was time to jump in a bit. While I’ve built some agent workflows in the past and built a variety of apps, it’s usually been team based. This is one of those, “Hey, if I can do this… anyone can” type posts. The question, of course, is do you have a reason / use case? But the whole argument that used to exist about some things being “too hard” or “too technical” or “too much time” are fading away with some of these tools. Not completely. And some things absolutely – my opinion – require “real” developers. But others? Lower risk things? Personal productivity things? Not as much anymore. So I’m going to go through my process just at a super high level. My goal is to convince other product manager types to dive into this area more deeply than just watching a webinar and learning some of the lingo. Even for senior roles and beyond individual contributor roles, I personally think it’s useful to get a visceral feel for how things work. Doing so helps offer better context for what teams might be going through as well as understanding in what might be possible. And also, getting a sense of what budget implications might be if you’ve got P&L responsibility for a product.

Next we’ll get into the details, but if you want to see the end result, it’s here: (But note, the functionality won’t work as I’ve got the public webhooks turned off so as not to burn up my paid quotas on the services in use. To see it working, check the Loom Demo.)

Direct Link to the App Test Website

App Demo / Loom Video

[Read more…]

In GenAI and Search: Differences from a Product & UX Perspective, we started looking at differences between search and GenAI from a user needs perspective. We looked at the tools in terms of use cases.

Now we’re going to turn more towards how we present things. This skips over everything in the Machine Learning Ops (MLOps) flow. And that’s okay. Because maybe we should be starting with the goal. After all, before you spend what could be millions, whether it’s for a consumer facing startup or an internal Enterprise tool, it’s probably wise to do some prototyping and testing anyway. (For MLOps, See: What is MLOps? (Amazon), What is MLOps? (Google), Why You Need MLOps.) By the way, I’m focusing here on user facing products, whether consumer or business. (As opposed to internal tools for analysis or production, marketing tools, etc. Though these certainly can have UX concerns as well.)

Figuring out how to design products to better serve users is of course not entirely your job. Whether you’re an entry level product manager or senior leadership, you’re ideally living more in the customer problem space. You’re looking across all things. Yes, you’re looking at features, functions, benefits. But also the dozens of other things to do. Which is why you work with your talented Design Leads. Whether direct reports or as a shared service, your design partners need to be getting up to speed on GenAI if they haven’t already. So your job is likely more along the lines of figuring out where or if AI is useful for your business; either for internal production or your actual products. And if it is, you might be the one – or at least be among the several – advocating for the resources to build out capabilities.

The classic question for Product Managers remains… How much do you really have to know in a specific domain; code, design, whatever? If you go too deep in any one area, chances are you won’t be very effective at your cross-functional tasks. This topic treatment is intended for skimming and basic understanding so you can work well with your talented design and tech colleagues. Our goal here isn’t to get anyone in Product to a practitioner level. It’s more to give you the tools to contribute effectively and have customer focused conversations with your specialist colleagues and drive requirements of value. Depth here once again depends on the type of product person you are. If you’re solidly on the business side, your whole product might be a P&L exercise for you and most functions and other team members are ‘just’ a line item on a spreadsheet as far as you’re concerned. A Technical PM? OK, you’re in the weeds with everything from APIs to whatever. Most of us are somewhere in between. One thing generally seems true though… if something comes up in a new domain where no one has clear responsibility yet, chances are you’re going to own it while an organization gets up to speed with where to put the new thing.

[Read more…]

What is a Rubric for AI Products?

A rubric is about evaluation and quality control, but also standardization, consistency and more. The origin of rubrics is from education and assessment so the term may be new to a digital product person. The general idea is to have a highly structured way to evaluate qualitative judgments. This seemed to be somewhat parallel to what was needed to evaluate AI output, so the model was adapted for that purpose. Rubrics for AI evaluation are used in academia, by tech companies, and regulatory and standards bodies. For traditional development, we have a variety of QA standards. A lot of them involve unit and integration testing and in modern workflows is often part of a continuous development and deployment plan. Rubrics can also be used along a development path, during early evaluation and fine tuning, pre-deployment, and for ongoing testing. However, at least a rough model must be fully available.

In the case of AI model quality assessment, a rubric is a structured framework for evaluation.

[Read more…]

Note: This article is from a Product Manager and Information Architecture perspective. It’s not a consumer guide on how to search better using GenAI.

As Product people, the things we care about most deeply are in the problem spaces. What challenges are we trying to solve? In the fast-changing world of information retrieval, it’s useful to have an understanding of underlying motivations for customer behaviors.

Before we start scrambling to slap an AI prompt input field on top of whatever we’re already selling, we’re going to look at some of the “Why.” Why do people use some of these tools. What is it they really seek? As product managers, we come from diverse backgrounds. Not all have depth in basic information retrieval backgrounds. It’s going to be important to understand some of these concepts as you and your teams will likely be working on projects that will need them. And you may need to consider P&L or similar concerns in these areas.

We’re going to explore use case differences in search vs. some of the newer Large Language Model (LLMs) and Generative AI (GenAI) tools with a longer term goal of how we can do a solid job crafting product that makes use of GenAI experiences. (Including those that go beyond the search use cases.) To do this will take a few steps. The first is making sure we’re thinking about the problem spaces of users and the use cases of traditional search and now generative AI from customer use case perspective. There are many use cases beyond this. Various AI tools can be used for IoT needs, Agent inputs/triggers, Oracle data for blockchain Smart Contracts (arguably these are just agentic triggers as well), and more. (Not to mention multi-modal object types.) These situations offer good cause to evaluate architectures at a deep information architecture level. But for now, we’re going to focus on the day-to-day human interface and we’ll start with basic search. In upcoming articles, we’ll look at design patterns and additional resources for those who want to go deeper.

It’s useful to start with traditional search. Partly as a kind of warm up to get us thinking about how to solve new kinds of problems. Also because there’s overlap with GenAI and we can build on search towards better understanding of how to deploy GenAI.

[Read more…]