TetraMesa

Intro to AI Rubrics for Product Managers

By

What is a Rubric for AI Products?

A rubric is about evaluation and quality control, but also standardization, consistency and more. The origin of rubrics is from education and assessment so the term may be new to a digital product person. The general idea is to have a highly structured way to evaluate qualitative judgments. This seemed to be somewhat parallel to what was needed to evaluate AI output, so the model was adapted for that purpose. Rubrics for AI evaluation are used in academia, by tech companies, and regulatory and standards bodies. For traditional development, we have a variety of QA standards. A lot of them involve unit and integration testing and in modern workflows is often part of a continuous development and deployment plan. Rubrics can also be used along a development path, during early evaluation and fine tuning, pre-deployment, and for ongoing testing. However, at least a rough model must be fully available.

In the case of AI model quality assessment, a rubric is a structured framework for evaluation.

[Read more…]

GenAI and Search: Differences from a Product & UX Perspective

By

Note: This article is from a Product Manager and Information Architecture perspective. It’s not a consumer guide on how to search better using GenAI.

As Product people, the things we care about most deeply are in the problem spaces. What challenges are we trying to solve? In the fast-changing world of information retrieval, it’s useful to have an understanding of underlying motivations for customer behaviors.

Before we start scrambling to slap an AI prompt input field on top of whatever we’re already selling, we’re going to look at some of the “Why.” Why do people use some of these tools. What is it they really seek? As product managers, we come from diverse backgrounds. Not all have depth in basic information retrieval backgrounds. It’s going to be important to understand some of these concepts as you and your teams will likely be working on projects that will need them. And you may need to consider P&L or similar concerns in these areas.

We’re going to explore use case differences in search vs. some of the newer Large Language Model (LLMs) and Generative AI (GenAI) tools with a longer term goal of how we can do a solid job crafting product that makes use of GenAI experiences. (Including those that go beyond the search use cases.) To do this will take a few steps. The first is making sure we’re thinking about the problem spaces of users and the use cases of traditional search and now generative AI from customer use case perspective. There are many use cases beyond this. Various AI tools can be used for IoT needs, Agent inputs/triggers, Oracle data for blockchain Smart Contracts (arguably these are just agentic triggers as well), and more. (Not to mention multi-modal object types.) These situations offer good cause to evaluate architectures at a deep information architecture level. But for now, we’re going to focus on the day-to-day human interface and we’ll start with basic search. In upcoming articles, we’ll look at design patterns and additional resources for those who want to go deeper.

It’s useful to start with traditional search. Partly as a kind of warm up to get us thinking about how to solve new kinds of problems. Also because there’s overlap with GenAI and we can build on search towards better understanding of how to deploy GenAI.

[Read more…]

MVPs – Refresh, Reimagine or Retire?

By

TL;DR:

  • If there’s no market, then your methodology doesn’t matter, MVP or otherwise.
  • This article is going to be a bit about MVP history and where it’s gotten a bit tired. (Or really more about where the idea is misused.) If you’re just interested in some recent thinking on alternatives and extensions, skip this article entirely and go to the Part 2: MVPs – Something.Next.

How is MVP Feeling These Days?

You’ve seen the clickbait headlines… “MVP is Dead. Long Live MVP” Etc. I saw one recently and since I work with several startups that’s what motivated me to create these two posts.

The Minimum Viable Product idea was the new bright shiny object once upon a time, spawning all manner of adherents. It got going around 2001 with Frank Robinson and Steve Blank, but didn’t get popular until 2011 with Eric Ries’ book “The Lean Startup.” Like many management innovations, everyone likes to pile on. There are just two, maybe three issues…

  • It’s rare that one tool can solve every problem.
  • It’s hard to measure. (We see the winners, but does anyone really know how many fail?)
  • Time. We learn things with time, but don’t necessarily adapt well.

It’s time to look into this. Maybe make a few tweaks. For the record, I’m an Eric Ries, Marty Cagan, Dan Olsen fan. So this isn’t about MVP bashing. If you haven’t, read their books and blogs to realize the way some talk and go about MVP is not what they espoused. Ries’ ideas are about rapid experimentation and validated learning, Cagan about product leadership, empowered teams with discovery and understanding user needs. Olsen expands this with ideas of problem-solution fit and more. (If getting Cagan’s “Inspired” make sure to get the latest edition.)

None of them say just fire up your task management software and have at it.

Over time we’ve found holes in the framework, pushed against some edges, and maybe run into a wall here or there. Often perceived problems aren’t the model, but poor interpretation and use. Or perhaps it’s the wrong tool for a challenge. MVP should help validate ideas in the marketplace as early as practicable. What could be wrong with that? Well, if it’s just used as a shortcut to produce something minimal, but not at all viable, maybe you’ve wasted more time and development cycle costs than you would have with better research. Even if your building and iterating early tests super fast with AI prototyping tools, you might just be headed into a wall faster.

It’s time to take another look. First, we’ll go over some basics and where some aging, but still valid, ideas fall down somewhat. Then we’ll look at enhancements to fill in these gaps.

[Read more…]

Content Moderation: Management Checklist

By

Content Moderation for New Products

This is a follow up to “Content Moderation: Where We Are Now.“

If your product has user generated content (UGC), moderation is an essential burden. This is likely the last thing on your mind. You and your team have a vision of some great new thing. Dealing with the dark side here is a time suck distraction and cost. But deal with it you must. Products small to large haven’t fully solved this. The large scale social networks are the lightning rods, but all services contend with this. And you take pain from at least two sides; managing moderation itself, and the criticism from those who think you’re doing it wrong. Maybe too little. Maybe too much. Most likely just not “their” way. You need to have communications management in place to deal with this as well.

Spoiler Alert: At the end of this writeup is a link to a sample of how code classification for moderation can work.

[Read more…]

Content Moderation: Where We Are Now

By

This pair of articles is for product managers who might deal with content moderation issues. We’re not going too deeply into the hot debate about moderation at the major platforms. There’s no shortage of folks throwing in on that. Though we’ll dive in somewhat to take away lessons. We’re going to look more at what’s going on for those who are practitioners as managers and builders, not observers.

Rising Volume of Posts and Debate

Besides the top blogworthy targets of debate, there’s tens of millions of sites with some form of user generated content. (Estimate based on Netcraft’s 2024 Web Server survey. Over a billion websites across 270 million domains. If we look at categories of sites that likely have UGC, it’s maybe 40% Even just half of that is still 50 million.)

This means many of us who don’t happen to be Facebook, Twitter, YouTube, Reddit, etc. have the benefit of not being hauled before Congress to testify on our content moderation policies. However we still must deal with the issues. Especially since manipulation isn’t just about politics, it’s serious dollars. Among the statistics CrowdRiff reports, they include the GrandView Research estimate of 2023 UGC market at $5.36 billion, growing at a 29%+ CAGR through 2030.

[Read more…]

Content Moderation: Product History

By

This is mostly for product managers who might be involved with products that have content moderation involved. Given the increased pitch around this topic right now, having built or managed several products with these features, I thought I’d do a series on some of the issues involved. This first part will be historical. If you’re not into that, you can skip to an upcoming article, (once I write it), to go right to some suggestions.

Our collective societal debate over content moderation moving from a simmering debate to a boiling over mess was inevitable. As one of the first product people to be faced with managing consumer masses as they gained access to the open internet, I thought it might be interesting to take a look at how we got here and where we might go next. Especially because so many of us working in product management have some degree of this issue to deal with. You don’t have to be a Facebook or Twitter or Reddit to face issues in these areas.

[Read more…]

Strategic Responses to GenAI from Search

By


I initially planned to write about how Generative AI (GenAI) might impact traditional search in a classic startup vs. incumbent scenario. Over time, as the landscape rapidly evolved, first I shared thoughts on “Search Tools in a GPT World” and then “Traditional Search vs. GPT Business Models.” This all leads to the obvious question… what are traditional search engines doing and what else might they do to respond?

[Read more…]

Traditional Search vs. GPT Business Models

By

What About Business Models?

In the first part of this article series, “Search Tools in a GPT World,” we looked at Search Tools in a product environment where AI GPTs are clearly on a tear. Now we’ll look at how technology and consumer sentiment shifts are impacting economics and business models.

The rise of GPTs introduces significant shifts in business models underpinning search and information retrieval. Search engines operate primarily on ad-driven models, based on traffic, clicks, and ranking. This impacts income to both search engines and the publishers to whom they drive traffic. We’re going to focus on the search engines themselves. In contrast to traditional search, GPTs seem mostly out of the starting gate with pay-per-use, subscription, or freemium models. This may be a reflection of the resource-intensive nature of generating real-time responses. It’s a simpler business model than search, which depends on a complex ad services ecosystem. As well, ads alone might not be sensible from a P&L perspective. Let’s review some of the business models.

[Read more…]

Search Tools in a GPT World

By

I’ve always enjoyed search, both as user and builder. So from a product perspective, I’ve been fascinated by its evolution and the recent fires lit under the traditional tools thanks to the ascendence of AIs. This will be a three part series. First, Search Tools in a GPT world, then business models, and lastly, how traditional search might respond.

So… How might the “traditional” Search industry evolve in the face of AI GPTs? Let’s take a historical tour to consider some customer pain points and values that various tools deal with and how these are morphing. It’s not as simple as GPTs are better search and it might be useful to consider other technology shifts. Did Video Kill the Radio Star? Maybe. But video didn’t kill radio. At least, not completely. Yet. OK, yes, perhaps the shift decimated revenues, but niche use cases survived through both television and even through more recent digital streaming. Even satellite radio was also able to find a place. Will the information retrieval industry experience something similar with what’s been billed as an even more disruptive technology? Or is this truly something radically different if we consider this shift on the level of industrial revolution?

Will the future of Search follow a similar path? Perhaps somewhat, but maybe not quite the death blow some have suggested given there seem to be a lot of niche values for Search. AI driven GPTs, (Generative Pre-trained Transformers), are already changing the search landscape. But their evolution is not as simply obvious as “this is a better search” for at least two related, but separate reasons. First, GPTs can likely excel past traditional search for a wide variety of use cases. But perhaps not all. And second, GPTs can and are used for significantly different use cases than search.

[Read more…]

Location

We're located in Stamford, CT, "The City that Works." Most of our in person engagement Clients are located in the metro NYC area in either New York City, Westchester or Fairfield Counties, as well as Los Angeles and San Francisco. We do off site work for a variety of Clients as well.

Have a Project?

If you have a project you would like to discuss, just get in touch via our Contact Form.

Connect

As a small consultancy, we spend more time with our Clients' social media than our own. If you would like to keep up with us the rare times we have something important enough to say via social media, feel free to follow our accounts.