TetraMesa

Intro to AI Rubrics for Product Managers

By

What is a Rubric for AI Products?

A rubric is about evaluation and quality control, but also standardization, consistency and more. The origin of rubrics is from education and assessment so the term may be new to a digital product person. The general idea is to have a highly structured way to evaluate qualitative judgments. This seemed to be somewhat parallel to what was needed to evaluate AI output, so the model was adapted for that purpose. Rubrics for AI evaluation are used in academia, by tech companies, and regulatory and standards bodies. For traditional development, we have a variety of QA standards. A lot of them involve unit and integration testing and in modern workflows is often part of a continuous development and deployment plan. Rubrics can also be used along a development path, during early evaluation and fine tuning, pre-deployment, and for ongoing testing. However, at least a rough model must be fully available.

In the case of AI model quality assessment, a rubric is a structured framework for evaluation.

[Read more…]

GenAI and Search: Differences from a Product & UX Perspective

By

Note: This article is from a Product Manager and Information Architecture perspective. It’s not a consumer guide on how to search better using GenAI.

As Product people, the things we care about most deeply are in the problem spaces. What challenges are we trying to solve? In the fast-changing world of information retrieval, it’s useful to have an understanding of underlying motivations for customer behaviors.

Before we start scrambling to slap an AI prompt input field on top of whatever we’re already selling, we’re going to look at some of the “Why.” Why do people use some of these tools. What is it they really seek? As product managers, we come from diverse backgrounds. Not all have depth in basic information retrieval backgrounds. It’s going to be important to understand some of these concepts as you and your teams will likely be working on projects that will need them. And you may need to consider P&L or similar concerns in these areas.

We’re going to explore use case differences in search vs. some of the newer Large Language Model (LLMs) and Generative AI (GenAI) tools with a longer term goal of how we can do a solid job crafting product that makes use of GenAI experiences. (Including those that go beyond the search use cases.) To do this will take a few steps. The first is making sure we’re thinking about the problem spaces of users and the use cases of traditional search and now generative AI from customer use case perspective. There are many use cases beyond this. Various AI tools can be used for IoT needs, Agent inputs/triggers, Oracle data for blockchain Smart Contracts (arguably these are just agentic triggers as well), and more. (Not to mention multi-modal object types.) These situations offer good cause to evaluate architectures at a deep information architecture level. But for now, we’re going to focus on the day-to-day human interface and we’ll start with basic search. In upcoming articles, we’ll look at design patterns and additional resources for those who want to go deeper.

It’s useful to start with traditional search. Partly as a kind of warm up to get us thinking about how to solve new kinds of problems. Also because there’s overlap with GenAI and we can build on search towards better understanding of how to deploy GenAI.

[Read more…]

MVPs – Refresh, Reimagine or Retire?

By

TL;DR:

  • If there’s no market, then your methodology doesn’t matter, MVP or otherwise.
  • This article is going to be a bit about MVP history and where it’s gotten a bit tired. (Or really more about where the idea is misused.) If you’re just interested in some recent thinking on alternatives and extensions, skip this article entirely and go to the Part 2: MVPs – Something.Next.

How is MVP Feeling These Days?

You’ve seen the clickbait headlines… “MVP is Dead. Long Live MVP” Etc. I saw one recently and since I work with several startups that’s what motivated me to create these two posts.

The Minimum Viable Product idea was the new bright shiny object once upon a time, spawning all manner of adherents. It got going around 2001 with Frank Robinson and Steve Blank, but didn’t get popular until 2011 with Eric Ries’ book “The Lean Startup.” Like many management innovations, everyone likes to pile on. There are just two, maybe three issues…

  • It’s rare that one tool can solve every problem.
  • It’s hard to measure. (We see the winners, but does anyone really know how many fail?)
  • Time. We learn things with time, but don’t necessarily adapt well.

It’s time to look into this. Maybe make a few tweaks. For the record, I’m an Eric Ries, Marty Cagan, Dan Olsen fan. So this isn’t about MVP bashing. If you haven’t, read their books and blogs to realize the way some talk and go about MVP is not what they espoused. Ries’ ideas are about rapid experimentation and validated learning, Cagan about product leadership, empowered teams with discovery and understanding user needs. Olsen expands this with ideas of problem-solution fit and more. (If getting Cagan’s “Inspired” make sure to get the latest edition.)

None of them say just fire up your task management software and have at it.

Over time we’ve found holes in the framework, pushed against some edges, and maybe run into a wall here or there. Often perceived problems aren’t the model, but poor interpretation and use. Or perhaps it’s the wrong tool for a challenge. MVP should help validate ideas in the marketplace as early as practicable. What could be wrong with that? Well, if it’s just used as a shortcut to produce something minimal, but not at all viable, maybe you’ve wasted more time and development cycle costs than you would have with better research. Even if your building and iterating early tests super fast with AI prototyping tools, you might just be headed into a wall faster.

It’s time to take another look. First, we’ll go over some basics and where some aging, but still valid, ideas fall down somewhat. Then we’ll look at enhancements to fill in these gaps.

[Read more…]

DeepSeek and AI – Take a Breath

By

As a user of several GPTs and subscriber to several, of course I had to check out the new bright shiny object. There seems to have been a lot of very fast reaction to this thing. Sometimes it’s useful to just take a moment and gain some perspective.

Last week, when DeepSeek released its open source AI model, it seemed like all hell broke loose in AI world. The quality of this Chinese-led effort seemed to be a surprise, especially given the apparent cost differences and supposedly use of much lesser compute infrastructure to create it. Various stock prices took hits, concerns about information sharing based on use of the DeepSeek product, similar to TikTok issues, were surfaced, and so on.

Let’s survey the issues and put some of this in perspective.

[Read more…]

Content Moderation: Management Checklist

By

Content Moderation for New Products

This is a follow up to “Content Moderation: Where We Are Now.“

If your product has user generated content (UGC), moderation is an essential burden. This is likely the last thing on your mind. You and your team have a vision of some great new thing. Dealing with the dark side here is a time suck distraction and cost. But deal with it you must. Products small to large haven’t fully solved this. The large scale social networks are the lightning rods, but all services contend with this. And you take pain from at least two sides; managing moderation itself, and the criticism from those who think you’re doing it wrong. Maybe too little. Maybe too much. Most likely just not “their” way. You need to have communications management in place to deal with this as well.

Spoiler Alert: At the end of this writeup is a link to a sample of how code classification for moderation can work.

[Read more…]

Content Moderation: Where We Are Now

By

This pair of articles is for product managers who might deal with content moderation issues. We’re not going too deeply into the hot debate about moderation at the major platforms. There’s no shortage of folks throwing in on that. Though we’ll dive in somewhat to take away lessons. We’re going to look more at what’s going on for those who are practitioners as managers and builders, not observers.

Rising Volume of Posts and Debate

Besides the top blogworthy targets of debate, there’s tens of millions of sites with some form of user generated content. (Estimate based on Netcraft’s 2024 Web Server survey. Over a billion websites across 270 million domains. If we look at categories of sites that likely have UGC, it’s maybe 40% Even just half of that is still 50 million.)

This means many of us who don’t happen to be Facebook, Twitter, YouTube, Reddit, etc. have the benefit of not being hauled before Congress to testify on our content moderation policies. However we still must deal with the issues. Especially since manipulation isn’t just about politics, it’s serious dollars. Among the statistics CrowdRiff reports, they include the GrandView Research estimate of 2023 UGC market at $5.36 billion, growing at a 29%+ CAGR through 2030.

[Read more…]

Content Moderation: Product History

By

This is mostly for product managers who might be involved with products that have content moderation involved. Given the increased pitch around this topic right now, having built or managed several products with these features, I thought I’d do a series on some of the issues involved. This first part will be historical. If you’re not into that, you can skip to an upcoming article, (once I write it), to go right to some suggestions.

Our collective societal debate over content moderation moving from a simmering debate to a boiling over mess was inevitable. As one of the first product people to be faced with managing consumer masses as they gained access to the open internet, I thought it might be interesting to take a look at how we got here and where we might go next. Especially because so many of us working in product management have some degree of this issue to deal with. You don’t have to be a Facebook or Twitter or Reddit to face issues in these areas.

[Read more…]

Ironies of Safety vs. Risk for Personal Self-Sovereign Assets

By


Crypto is on a new tear again. With 40% of U.S. adults supposedly now owning some crypto, are we moving from early adoption of a couple of years ago into early majority in terms of the technology adoption curve?

Wherever we are in the larger scale hype cycle, I’ve had too many conversations recently indicating poor info out there. A lot comes from the influencer class on YouTube who often breathlessly extol the virtues of how if you don’t do or don’t do this or that your financial future will lay in tatters. If you were to somehow measure the emotional tenor of some of these folks, it’s similar to if you watch the manically effusive presentations of the gamers playing everything from Minecraft to other kids’ games. Most of them probably mean well and many provide great value, but there’s things they’re leaving out. And one major issue is the irony that having more control of your assets could incur more risk than one otherwise might.

The last thing the world needs is another “Intro to Crypto.” And this isn’t that. But I do want to go over some issues considering use of self-sovereign wallets vs. custodial type accounts. For those just starting to get exposure to crypto as an asset class, with Custodial wallets/accounts, a third party holds your crypto, manages any account keys and executes transactions on your behalf. (Such as Coinbase, Kraken, etc.) They’re like your traditional bank or brokerage. Alternatively, (or additionally), with Self-sovereign wallets you fully control your assets using private keys in your wallet and what it holds with no intermediaries involved. Whether you’re transferring funds, trading on decentralized exchanges, purchasing things, connecting and transacting with other applications, or whatever, you have to do everything in terms of managing transactions and you own all associated risks; of which there are several. My goal isn’t to dissuade you from this category. It’s just to shine a brighter light on some of the downside risk and offer some risk mitigation strategies.

[Read more…]

How Badly Will the GPTs Kick Google’s Teeth In

By


Notice I didn’t ask “if” newer Artificial Intelligence tools like Generative Pre-trained Transformers (GPTs) will impact Google. I asked “How badly.” While Google isn’t going anywhere, these new tools will chip away at its search market share and perhaps overall value. Google and parent company Alphabet develop properties from mobile operating systems to devices like Nest and Fitbit, Waymo autonomous driving, cloud services, and more. Still, advertising from search, YouTube and their network was almost 80% of their revenues in 2023. Google’s dominance remains and its advertising revenue is still its golden goose. However, GenAI tools introduce a serious competitive threat to the core search business. How badly will Google feel the impact?

Here’s a high level summary of the main points, after which I’ll try to defend each.

  1. Basic Search Needs: AI GPTs are increasingly satisfying basic search needs. While not perfect, they meet many use cases where users are seeking answers; not links to maybe answers.
  2. AI Stickiness: GPTs offer capabilities beyond search, encouraging users to stay in that space. Even with some of the challenges with AI accuracy, people may stick with a “good enough” solution.
  3. GPTs are Improving: These tools are rapidly evolving, fueled by intense investment and innovation.
  4. Google Is Somewhat Stuck: Google’s brand is so tied to traditional search that pivoting may alienate users or undermine its core business.
  5. Everyone is Attacking from Multiple Vectors: Yes. OpenAI’s ChatGPT may have been the alarm bell, but there’s a whole lot more coming; both consumer and business.
  6. Business Use Cases beyond advertising: People are paying premium prices.

That’s the tl;dr. Stop here. Or if you want the backup rationale behind these points, continue…

[Read more…]

Location

We're located in Stamford, CT, "The City that Works." Most of our in person engagement Clients are located in the metro NYC area in either New York City, Westchester or Fairfield Counties, as well as Los Angeles and San Francisco. We do off site work for a variety of Clients as well.

Have a Project?

If you have a project you would like to discuss, just get in touch via our Contact Form.

Connect

As a small consultancy, we spend more time with our Clients' social media than our own. If you would like to keep up with us the rare times we have something important enough to say via social media, feel free to follow our accounts.