What are We Doing?

We’re going to look at a plaything I built in just a handful of hours while digging into agentic AI a bit just for fun. (Well, for career-related things as well, but mostly for fun.) The toy is an AI enabled Digital Product Manager assistant app where you can ask questions about product management. Which, as it turns out, could actually be a real product. I built it as a toy project for fun, but might actually soft launch this thing. Because, why not? Can you just do this with any GPT? Sure. But this one is tuned specifically to product management in general and digital product management in particular. (There are others like it though, so maybe just leave it as a personal tool. We’ll see.)

Why do this? Why bother. And why do you care? Since the vibe coding thing is so very in right now, (maybe for another few months), I figured it was time to jump in a bit. While I’ve built some agent workflows in the past and built a variety of apps, it’s usually been team based. This is one of those, “Hey, if I can do this… anyone can” type posts. The question, of course, is do you have a reason / use case? But the whole argument that used to exist about some things being “too hard” or “too technical” or “too much time” are fading away with some of these tools. Not completely. And some things absolutely – my opinion – require “real” developers. But others? Lower risk things? Personal productivity things? Not as much anymore. So I’m going to go through my process just at a super high level. My goal is to convince other product manager types to dive into this area more deeply than just watching a webinar and learning some of the lingo. Even for senior roles and beyond individual contributor roles, I personally think it’s useful to get a visceral feel for how things work. Doing so helps offer better context for what teams might be going through as well as understanding in what might be possible. And also, getting a sense of what budget implications might be if you’ve got P&L responsibility for a product.

Next we’ll get into the details, but if you want to see the end result, it’s here: (But note, the functionality won’t work as I’ve got the public webhooks turned off so as not to burn up my paid quotas on the services in use. To see it working, check the Loom Demo.)

Direct Link to the App Test Website

App Demo / Loom Video

[Read more…]

In GenAI and Search: Differences from a Product & UX Perspective, we started looking at differences between search and GenAI from a user needs perspective. We looked at the tools in terms of use cases.

Now we’re going to turn more towards how we present things. This skips over everything in the Machine Learning Ops (MLOps) flow. And that’s okay. Because maybe we should be starting with the goal. After all, before you spend what could be millions, whether it’s for a consumer facing startup or an internal Enterprise tool, it’s probably wise to do some prototyping and testing anyway. (For MLOps, See: What is MLOps? (Amazon), What is MLOps? (Google), Why You Need MLOps.) By the way, I’m focusing here on user facing products, whether consumer or business. (As opposed to internal tools for analysis or production, marketing tools, etc. Though these certainly can have UX concerns as well.)

Figuring out how to design products to better serve users is of course not entirely your job. Whether you’re an entry level product manager or senior leadership, you’re ideally living more in the customer problem space. You’re looking across all things. Yes, you’re looking at features, functions, benefits. But also the dozens of other things to do. Which is why you work with your talented Design Leads. Whether direct reports or as a shared service, your design partners need to be getting up to speed on GenAI if they haven’t already. So your job is likely more along the lines of figuring out where or if AI is useful for your business; either for internal production or your actual products. And if it is, you might be the one – or at least be among the several – advocating for the resources to build out capabilities.

The classic question for Product Managers remains… How much do you really have to know in a specific domain; code, design, whatever? If you go too deep in any one area, chances are you won’t be very effective at your cross-functional tasks. This topic treatment is intended for skimming and basic understanding so you can work well with your talented design and tech colleagues. Our goal here isn’t to get anyone in Product to a practitioner level. It’s more to give you the tools to contribute effectively and have customer focused conversations with your specialist colleagues and drive requirements of value. Depth here once again depends on the type of product person you are. If you’re solidly on the business side, your whole product might be a P&L exercise for you and most functions and other team members are ‘just’ a line item on a spreadsheet as far as you’re concerned. A Technical PM? OK, you’re in the weeds with everything from APIs to whatever. Most of us are somewhere in between. One thing generally seems true though… if something comes up in a new domain where no one has clear responsibility yet, chances are you’re going to own it while an organization gets up to speed with where to put the new thing.

[Read more…]

What is a Rubric for AI Products?

A rubric is about evaluation and quality control, but also standardization, consistency and more. The origin of rubrics is from education and assessment so the term may be new to a digital product person. The general idea is to have a highly structured way to evaluate qualitative judgments. This seemed to be somewhat parallel to what was needed to evaluate AI output, so the model was adapted for that purpose. Rubrics for AI evaluation are used in academia, by tech companies, and regulatory and standards bodies. For traditional development, we have a variety of QA standards. A lot of them involve unit and integration testing and in modern workflows is often part of a continuous development and deployment plan. Rubrics can also be used along a development path, during early evaluation and fine tuning, pre-deployment, and for ongoing testing. However, at least a rough model must be fully available.

In the case of AI model quality assessment, a rubric is a structured framework for evaluation.

[Read more…]

Note: This article is from a Product Manager and Information Architecture perspective. It’s not a consumer guide on how to search better using GenAI.

As Product people, the things we care about most deeply are in the problem spaces. What challenges are we trying to solve? In the fast-changing world of information retrieval, it’s useful to have an understanding of underlying motivations for customer behaviors.

Before we start scrambling to slap an AI prompt input field on top of whatever we’re already selling, we’re going to look at some of the “Why.” Why do people use some of these tools. What is it they really seek? As product managers, we come from diverse backgrounds. Not all have depth in basic information retrieval backgrounds. It’s going to be important to understand some of these concepts as you and your teams will likely be working on projects that will need them. And you may need to consider P&L or similar concerns in these areas.

We’re going to explore use case differences in search vs. some of the newer Large Language Model (LLMs) and Generative AI (GenAI) tools with a longer term goal of how we can do a solid job crafting product that makes use of GenAI experiences. (Including those that go beyond the search use cases.) To do this will take a few steps. The first is making sure we’re thinking about the problem spaces of users and the use cases of traditional search and now generative AI from customer use case perspective. There are many use cases beyond this. Various AI tools can be used for IoT needs, Agent inputs/triggers, Oracle data for blockchain Smart Contracts (arguably these are just agentic triggers as well), and more. (Not to mention multi-modal object types.) These situations offer good cause to evaluate architectures at a deep information architecture level. But for now, we’re going to focus on the day-to-day human interface and we’ll start with basic search. In upcoming articles, we’ll look at design patterns and additional resources for those who want to go deeper.

It’s useful to start with traditional search. Partly as a kind of warm up to get us thinking about how to solve new kinds of problems. Also because there’s overlap with GenAI and we can build on search towards better understanding of how to deploy GenAI.

[Read more…]

Is Prompt Engineering just an overly fancy way of saying, “Here’s a better search query?” Maybe it should just be called Search 2.0? True enough, the output of an AI large language model is more than just a bunch of results, but the query itself is still ‘just’ an instruction of sorts, right?

In some cases, yes, it’s essentially the same as a fancy query. But mostly not. There are obvious differences in the use cases for prompts using AI Large Language Models (LLMs) vs how keywords get used in traditional search engines. And for all their potential faults and risks, LLMs can provide stunning new capabilities across a variety of use cases. At the same time, there seems to be some overblown expectations as to what prompts can do. For example, at least in some places, a misunderstanding that prompt engineering can make models better. While it may be true that prompts and responses can be iteratively honed and fed back into the fine-tuning of models to actually make models better, for the most part, they’re not used this way. I’d like to try to clear this up because I think it’s important we understand how we can use our tools and where they’re limited. Just to be clear, I’m not talking about the handful of folks who really are evaluating prompt output to adjust models. (If you’re one of those folks, you’re ideally operating more at the data science kind of level of prompt engineering.) For our purposes here, I’m talking about the typical consumer or business use that seems to have some people believing prompt input alone changes how the models themselves work.

[Read more…]