A couple of weeks ago I deployed a small WordPress website in support of a particular medical community. My goal was to test using WordPress vs. A Gen AI website builder and see how well a moderately technical product manager type, (but not a developer), could use some tools to make digital things. I kind of joked at the end that “I’ll probably toss in a GPT Chat feature because, well, I mean… it’s 2025.” I wasn’t actually going to bother, but, really now… it is 2025 after all. The site should have a GPT. So let’s have at it! (Here’s the end result by the way: ACL-GPT.)
[Read more…]Comparison Site Build: WordPress vs. AI Builder
tl;dr
- I wanted to test building a small website comparing WordPress to a custom AI generated site to support people with a medical issue.
- Bottom line is the generative site is easily competitive with WordPress, and very flexible.
- Final choice is to to go with the reliable WordPress option though; so I don’t have to worry about ongoing issues, but the generative site capabilities were somewhere between kind of cool and astounding. (I personally think it’s just going to be a relatively short time before I’ll be choosing to use them more.)
- The end results include the ACL support site generated with a Gen AI tool, (which is not content complete at this point), and the go to market site, aclsupport.com, which is managed hosting WordPress. (No attempt to make them match visually or with content; just get the basics up.)
That’s it! if you want the full story, here you go…
Building a PM Helper with AI
What are We Doing?
We’re going to look at a plaything I built in just a handful of hours while digging into agentic AI a bit just for fun. (Well, for career-related things as well, but mostly for fun.) The toy is an AI enabled Digital Product Manager assistant app where you can ask questions about product management. Which, as it turns out, could actually be a real product. I built it as a toy project for fun, but might actually soft launch this thing. Because, why not? Can you just do this with any GPT? Sure. But this one is tuned specifically to product management in general and digital product management in particular. (There are others like it though, so maybe just leave it as a personal tool. We’ll see.)
Why do this? Why bother. And why do you care? Since the vibe coding thing is so very in right now, (maybe for another few months), I figured it was time to jump in a bit. While I’ve built some agent workflows in the past and built a variety of apps, it’s usually been team based. This is one of those, “Hey, if I can do this… anyone can” type posts. The question, of course, is do you have a reason / use case? But the whole argument that used to exist about some things being “too hard” or “too technical” or “too much time” are fading away with some of these tools. Not completely. And some things absolutely – my opinion – require “real” developers. But others? Lower risk things? Personal productivity things? Not as much anymore. So I’m going to go through my process just at a super high level. My goal is to convince other product manager types to dive into this area more deeply than just watching a webinar and learning some of the lingo. Even for senior roles and beyond individual contributor roles, I personally think it’s useful to get a visceral feel for how things work. Doing so helps offer better context for what teams might be going through as well as understanding in what might be possible. And also, getting a sense of what budget implications might be if you’ve got P&L responsibility for a product.
Next we’ll get into the details, but if you want to see the end result, it’s here: (But note, the functionality won’t work as I’ve got the public webhooks turned off so as not to burn up my paid quotas on the services in use. To see it working, check the Loom Demo.)
Direct Link to the App Test Website
[Read more…]Intrinsic / Extrinsic Product Values Framework
In a prior post about Intrinsic / Extrinsic Product Value Dynamics, we looked at the basic differences between intrinsic and extrinsic values and consumer perceptions. Here, we’re going over some frameworks you can consider using to define and position your products along these dimensions. The idea is to make sure that everything from features to messaging align with intended values.
Please note that this framework is a suggested proposal; not something I’ve seen used or tested. It focuses on creating and expressing value deeper in product features and structures rather than surface level messaging. (Which is of course still critical.) There is no experimental data of which I’m aware that tests on varying such dimensions against each other. (Though in the Value Dynamics article, we did go over examples of successful and challenged products.) My goal is to offer a structure to explicitly consider these dimensions and use them as input to a larger scope product positioning strategy that would include the other usual elements of actual features, pricing, competition and so on.
[Read more…]GenAI UX Issues for Product Managers
In GenAI and Search: Differences from a Product & UX Perspective, we started looking at differences between search and GenAI from a user needs perspective. We looked at the tools in terms of use cases.
Now we’re going to turn more towards how we present things. This skips over everything in the Machine Learning Ops (MLOps) flow. And that’s okay. Because maybe we should be starting with the goal. After all, before you spend what could be millions, whether it’s for a consumer facing startup or an internal Enterprise tool, it’s probably wise to do some prototyping and testing anyway. (For MLOps, See: What is MLOps? (Amazon), What is MLOps? (Google), Why You Need MLOps.) By the way, I’m focusing here on user facing products, whether consumer or business. (As opposed to internal tools for analysis or production, marketing tools, etc. Though these certainly can have UX concerns as well.)
Figuring out how to design products to better serve users is of course not entirely your job. Whether you’re an entry level product manager or senior leadership, you’re ideally living more in the customer problem space. You’re looking across all things. Yes, you’re looking at features, functions, benefits. But also the dozens of other things to do. Which is why you work with your talented Design Leads. Whether direct reports or as a shared service, your design partners need to be getting up to speed on GenAI if they haven’t already. So your job is likely more along the lines of figuring out where or if AI is useful for your business; either for internal production or your actual products. And if it is, you might be the one – or at least be among the several – advocating for the resources to build out capabilities.
The classic question for Product Managers remains… How much do you really have to know in a specific domain; code, design, whatever? If you go too deep in any one area, chances are you won’t be very effective at your cross-functional tasks. This topic treatment is intended for skimming and basic understanding so you can work well with your talented design and tech colleagues. Our goal here isn’t to get anyone in Product to a practitioner level. It’s more to give you the tools to contribute effectively and have customer focused conversations with your specialist colleagues and drive requirements of value. Depth here once again depends on the type of product person you are. If you’re solidly on the business side, your whole product might be a P&L exercise for you and most functions and other team members are ‘just’ a line item on a spreadsheet as far as you’re concerned. A Technical PM? OK, you’re in the weeds with everything from APIs to whatever. Most of us are somewhere in between. One thing generally seems true though… if something comes up in a new domain where no one has clear responsibility yet, chances are you’re going to own it while an organization gets up to speed with where to put the new thing.
[Read more…]Intro to AI Rubrics for Product Managers
What is a Rubric for AI Products?
A rubric is about evaluation and quality control, but also standardization, consistency and more. The origin of rubrics is from education and assessment so the term may be new to a digital product person. The general idea is to have a highly structured way to evaluate qualitative judgments. This seemed to be somewhat parallel to what was needed to evaluate AI output, so the model was adapted for that purpose. Rubrics for AI evaluation are used in academia, by tech companies, and regulatory and standards bodies. For traditional development, we have a variety of QA standards. A lot of them involve unit and integration testing and in modern workflows is often part of a continuous development and deployment plan. Rubrics can also be used along a development path, during early evaluation and fine tuning, pre-deployment, and for ongoing testing. However, at least a rough model must be fully available.
In the case of AI model quality assessment, a rubric is a structured framework for evaluation.
[Read more…]GenAI and Search: Differences from a Product & UX Perspective
Note: This article is from a Product Manager and Information Architecture perspective. It’s not a consumer guide on how to search better using GenAI.
As Product people, the things we care about most deeply are in the problem spaces. What challenges are we trying to solve? In the fast-changing world of information retrieval, it’s useful to have an understanding of underlying motivations for customer behaviors.
Before we start scrambling to slap an AI prompt input field on top of whatever we’re already selling, we’re going to look at some of the “Why.” Why do people use some of these tools. What is it they really seek? As product managers, we come from diverse backgrounds. Not all have depth in basic information retrieval backgrounds. It’s going to be important to understand some of these concepts as you and your teams will likely be working on projects that will need them. And you may need to consider P&L or similar concerns in these areas.
We’re going to explore use case differences in search vs. some of the newer Large Language Model (LLMs) and Generative AI (GenAI) tools with a longer term goal of how we can do a solid job crafting product that makes use of GenAI experiences. (Including those that go beyond the search use cases.) To do this will take a few steps. The first is making sure we’re thinking about the problem spaces of users and the use cases of traditional search and now generative AI from customer use case perspective. There are many use cases beyond this. Various AI tools can be used for IoT needs, Agent inputs/triggers, Oracle data for blockchain Smart Contracts (arguably these are just agentic triggers as well), and more. (Not to mention multi-modal object types.) These situations offer good cause to evaluate architectures at a deep information architecture level. But for now, we’re going to focus on the day-to-day human interface and we’ll start with basic search. In upcoming articles, we’ll look at design patterns and additional resources for those who want to go deeper.
It’s useful to start with traditional search. Partly as a kind of warm up to get us thinking about how to solve new kinds of problems. Also because there’s overlap with GenAI and we can build on search towards better understanding of how to deploy GenAI.
[Read more…]Strategic Responses to GenAI from Search
I initially planned to write about how Generative AI (GenAI) might impact traditional search in a classic startup vs. incumbent scenario. Over time, as the landscape rapidly evolved, first I shared thoughts on “Search Tools in a GPT World” and then “Traditional Search vs. GPT Business Models.” This all leads to the obvious question… what are traditional search engines doing and what else might they do to respond?
Traditional Search vs. GPT Business Models
What About Business Models?
In the first part of this article series, “Search Tools in a GPT World,” we looked at Search Tools in a product environment where AI GPTs are clearly on a tear. Now we’ll look at how technology and consumer sentiment shifts are impacting economics and business models.
The rise of GPTs introduces significant shifts in business models underpinning search and information retrieval. Search engines operate primarily on ad-driven models, based on traffic, clicks, and ranking. This impacts income to both search engines and the publishers to whom they drive traffic. We’re going to focus on the search engines themselves. In contrast to traditional search, GPTs seem mostly out of the starting gate with pay-per-use, subscription, or freemium models. This may be a reflection of the resource-intensive nature of generating real-time responses. It’s a simpler business model than search, which depends on a complex ad services ecosystem. As well, ads alone might not be sensible from a P&L perspective. Let’s review some of the business models.
[Read more…]Search Tools in a GPT World
I’ve always enjoyed search, both as user and builder. So from a product perspective, I’ve been fascinated by its evolution and the recent fires lit under the traditional tools thanks to the ascendence of AIs. This will be a three part series. First, Search Tools in a GPT world, then business models, and lastly, how traditional search might respond.
So… How might the “traditional” Search industry evolve in the face of AI GPTs? Let’s take a historical tour to consider some customer pain points and values that various tools deal with and how these are morphing. It’s not as simple as GPTs are better search and it might be useful to consider other technology shifts. Did Video Kill the Radio Star? Maybe. But video didn’t kill radio. At least, not completely. Yet. OK, yes, perhaps the shift decimated revenues, but niche use cases survived through both television and even through more recent digital streaming. Even satellite radio was also able to find a place. Will the information retrieval industry experience something similar with what’s been billed as an even more disruptive technology? Or is this truly something radically different if we consider this shift on the level of industrial revolution?
Will the future of Search follow a similar path? Perhaps somewhat, but maybe not quite the death blow some have suggested given there seem to be a lot of niche values for Search. AI driven GPTs, (Generative Pre-trained Transformers), are already changing the search landscape. But their evolution is not as simply obvious as “this is a better search” for at least two related, but separate reasons. First, GPTs can likely excel past traditional search for a wide variety of use cases. But perhaps not all. And second, GPTs can and are used for significantly different use cases than search.
[Read more…]