Human-Centered Product Reviews That Catch Risk Early

A product can perform exactly as designed and still fail the people expected to use it. That is where a human-centered product review begins.

I think this is one of the harder things for product teams to see, especially when the product is technically working. The model performs well. The feature does what it was built to do. Users can complete the task. Nothing is obviously broken.

But then the product meets the real world.

A clinician creates a workaround because an AI recommendation does not fit the patient in front of them. A student technically has access to a learning platform but struggles to use it independently. A healthcare tool performs well during testing but creates a new barrier when it enters an actual care setting.

We tend to call these adoption problems, implementation challenges, or edge cases. Sometimes they are. But sometimes they are telling us something about the product itself.

The team evaluated whether the product worked. No one looked closely enough at what happened when the product met real people, real workflows, and conditions that were messier than the ones used to design it.

That is the purpose of a human-centered product review.

Key Takeaways

• A human-centered product review examines more than whether a product works as designed. It looks at assumptions, real-world workflows, lived experiences, and the systems surrounding the product.

• For AI-enabled products, teams should follow the decision from data and model development through deployment and monitoring, asking who may be missing, how errors affect people, and whether recommendations can be questioned.

• Inclusion and stakeholder engagement provide useful product information by revealing barriers, workarounds, trust problems, and unintended consequences that performance data alone may miss.

• Finding a problem is only the beginning. A useful review prioritizes action by clarifying what the evidence means, what remains uncertain, who owns the decision, and when the team will look again.

• Human-centered review is an ongoing design process that should continue before launch, during implementation, and after launch as people, data, workflows, and conditions of use change.

What are we actually reviewing?

Most product reviews begin with reasonable questions. Does the product meet its requirements? Does the model reach its performance target? Can users complete the intended task? Is it ready to launch?

I would ask those questions too.

I would also want to know whose customer needs are represented in those requirements. What has to be true for the product to work as the team expects?

That means reviewing product requirements alongside practical design methods, including an assumption audit.

Reviewing a healthcare tool is an extension of user-centered design. Ask whether it assumes reliable internet access, complete medical records, uninterrupted clinician attention, English fluency, or easy access to follow-up care.

Those assumptions may never appear together in a product requirements document. They may never be written down at all. Qualitative methods, such as interviews and observation, can help reveal them.

This is different from market validation or defining a value proposition. A product can attract a market and still create harm or exclusion when its assumptions fail.

Once you start looking for those assumptions, you see them everywhere.

A learning platform built around digital technology may assume reading fluency or digital independence. An automated screening system may rely on historical data without asking who had access to the opportunity represented in that data. A customer-facing AI tool may provide accurate information in language that some intended users cannot reasonably understand.

The question I keep coming back to is: What happens when the assumptions behind the product are wrong?

That question takes us beyond the product itself and into the system around it.

Test the decision, not only the model

This becomes especially important with artificial intelligence (AI).

We often talk about AI bias as a problem with training data or model performance. Both matter. But bias can enter much earlier and continue much later.

Research published in the Journal of Medical Internet Research describes potential sources of bias throughout the AI lifecycle, including data collection, annotation, model development, evaluation, deployment, monitoring, and feedback. The authors also make the case for involving a broader range of people throughout that lifecycle (Chen et al., 2023).

Human-centered design offers an even simpler way to think about this: follow the decision.

What decision is the product helping someone make? What information goes into that decision? What happens because of it?

Imagine an algorithm designed to identify patients who might benefit from additional care management. It produces a risk score as part of a larger digital technology system. That score may affect which patient receives outreach, what a clinician sees first, or where a health system directs limited resources.

At that point, I want to know more than whether the algorithm is accurate.

Who is missing or underrepresented in the data? What is the model actually using as a measure of need? Can the clinician understand and question the recommendation? What happens when it is wrong? Does that error have the same consequence for everyone?

Mixed methods can combine model-performance measures with qualitative methods, including interviews, observation, and case review. Design methods can also help teams examine data provenance, proxy variables, and downstream consequences.

A well-known healthcare algorithm provides a useful example. Researchers found that an algorithm used to identify patients who would benefit from additional care underestimated the needs of Black patients. The algorithm used healthcare spending as a proxy for health need. Because less money had historically been spent on Black patients with the same level of illness, lower spending was interpreted as lower need (Obermeyer et al., 2019).

The algorithm was doing what it had been designed to do.

The problem was further upstream. Healthcare spending and healthcare need are not the same thing. That distinction matters when developing health innovations.

The example also shows why artificial intelligence used across broader health systems needs evaluation beyond model metrics.

A review focused only on model performance can miss that distinction.

There is often a gap between intended use and lived use

Product teams know what they intended to build. I think that can actually make some problems harder to see.

If we know a feature was designed to simplify a workflow, we may interpret friction as a training problem. If an AI tool is supposed to assist rather than replace human judgment, we may assume everyone understands that distinction. If average performance looks strong, we may not immediately look for the people or settings where the product performs differently.

But users don't experience the intention behind a product. They experience the product.

That gap between intended use and lived use is one of the places I would look closely during a human-centered review.

In healthcare, that could mean watching what happens when a clinician receives an alert while managing several other demands at the same time.

In education, it could mean discovering that a feature designed to give students more independence actually creates additional work for a student, teacher, or parent.

With an AI-enabled service, I would want to know whether people understand what the system is doing, recognize the limits of its output, and have some way to question a recommendation when it doesn't make sense.

These are things that can be difficult to see from inside a product team.

The people closest to the consequences often see them much faster.

Inclusion gives us information we didn't have before

I have spent a lot of my career thinking about inclusion, and one thing I wish we would stop doing is treating it as separate from the "real" work of building a product.

Inclusion gives us information.

A patient may identify a barrier that never appears in a performance dashboard. A clinician can show us that a useful recommendation adds three steps to an already overloaded workflow. A student may explain why a feature intended to create flexibility actually makes something harder to navigate. Someone from the community a product is supposed to serve may recognize a trust problem that the product team has interpreted as resistance to adoption.

That is useful product information.

It may change the requirements. It may change testing. It may change the interface or the workflow. Sometimes it should make us reconsider whether we are solving the right problem in the first place.

This is also one reason NIST's AI Risk Management Framework calls for input from a broad range of perspectives across the AI lifecycle. NIST treats risk management as something that happens throughout the design, development, deployment, and evaluation of AI systems, rather than as a final compliance exercise (Tabassi, 2023).

We don't need every person affected by a product sitting in every meeting. We do need to know when our own perspective is too narrow to see what we need to see.

Sometimes the product isn't where the failure happens

This is another reason I like looking at products as part of larger systems.

Imagine a healthcare product makes the right recommendation, but following it requires transportation the patient doesn't have.

Or the software appropriately flags uncertainty, but workplace culture makes a junior employee reluctant to question how someone more senior interprets the result.

Or health innovations work beautifully during a pilot at an academic medical center and struggle when introduced somewhere with different staffing, patient populations, resources, or workflows.

Where is the product failure in those examples?

There may not be one in the traditional sense.

But there is still a problem worth understanding.

This is where my systems background probably influences the way I look at product work. I want to know what is happening around the technology, because the surrounding environment changes what digital technology does in practice.

That often requires mixed methods. We might combine workflow observation, stakeholder input, and outcome data.

A product also makes more sense when assessed as part of an innovation strategy, not as an isolated feature.

So a human-centered review might look at:

People: Who uses the product? Who is affected by decisions made with it, even if they never interact with it directly?

Purpose: What problem are we trying to solve, and whose problem is it?

Evidence: What information shaped the product? What information might be missing?

Workflow: Where does the product enter an existing process? What additional work or judgment does it create?

Impact: Who benefits? Who may have a harder user experience? Who carries the consequences when something goes wrong?

Governance: What stakeholder engagement is needed? Who has the authority to respond when we find a problem?

That last question matters more than it may seem.

Finding a problem isn't enough

A product review can generate a very long list of pain points and possible improvements.

That isn't especially useful on its own.

Some findings matter much more than others. Some require a product change. Others may call for additional testing, clearer communication, better monitoring, a workflow change, or an explicit acknowledgment that the product has a limitation.

And sometimes the evidence may tell us that the risk is small enough to accept.

A useful review is a problem-solving technique for prioritizing action, not just documenting defects.

The point is to see enough to make that decision intentionally.

This is particularly important for health innovations and other products that affect education, access, opportunity, or meaningful parts of people's lives.

The World Health Organization's guidance on AI for health includes human autonomy, safety, accountability, inclusiveness, and equity among its core principles. It also calls for AI applications to be evaluated during actual use, when organizations can see consequences that may not have been apparent during development (World Health Organization, 2021).

Qualitative methods can reveal those consequences in context. A mixed methods approach can combine those findings with performance data and monitoring signals, especially as digital technology changes data, workflows, and conditions of use.

For me, that means a useful review should eventually get us to some very practical questions:

What did we find?

How much does it matter?

What don't we know yet?

What are we going to do about it?

Who owns that decision?

And when are we going to look again?

Without those last few questions, it is very easy to produce an excellent report that changes absolutely nothing.

Some things don't become visible until after launch

Human-centered design should not treat product review as a one-time exercise.

There are questions we can ask very early, when a team is still deciding what problem to solve and for whom. That is often the best time to challenge assumptions in product development, because changing direction is still relatively easy.

There are different questions to ask before launch, using prototyping and testing. Does the product work across the people and environments where we expect it to be used? What happens when it fails? Can people recover? Are there differences in performance or experience that matter?

These design methods should continue after launch. People use products in unexpected ways. They create workarounds. Different populations adopt them differently. Data changes. Organizations change. A tool designed for one purpose gets used for another.

During the implementation stage, digital technology can reveal patterns that weren't visible earlier. We can learn through mixed methods, combining usage data with interviews or observation.

NIST and WHO both treat artificial intelligence risk management as ongoing for essentially this reason. What happens in actual use is evidence too.

I think product teams should treat it as an iterative design process. Look again, learn, and adjust.

Better products begin with seeing more clearly

I don't expect any product to work perfectly for every person under every possible condition. That isn't a realistic standard, and it isn't what this kind of review is trying to accomplish.

What we can do is become much better at seeing where our assumptions end, then make consequential product design decisions with greater care.

We can look beyond whether digital technology functions and ask what happens around it. We can bring in people who know things the product team doesn't know. We can examine who benefits, who struggles, and what those differences reveal about the user experience. And when we find something consequential, we can make an actual decision about what to do next.

That is what this kind of review makes possible, it gives us another vantage point on a product we may already know extremely well.

A product can work exactly as we designed it to work, and sometimes that is precisely why we need to look again.

Frequently Asked Questions

What is a human-centered product review?

A human-centered product review looks at what happens when a product interacts with the people and systems around it. Depending on the product, that might include assumptions about users, data, workflow, accessibility, product-supported decisions, potential harms, and what happens when the product doesn't work as expected.

How is this different from usability testing?

Usability testing is one important source of information. A human-centered review looks more broadly at the system around the product. A product may be easy to use and still rely on problematic assumptions, affect people differently, or create consequences elsewhere in a workflow.

How is this different from customer reviews?

Customer reviews capture individual reactions and experiences with a product. A human-centered review examines the assumptions, workflows, evidence, and consequences surrounding that product.

When should a human-centered review happen?

There are useful questions to ask throughout the product lifecycle. Early reviews can uncover assumptions while they are still relatively easy to change. Pre-launch reviews can examine performance and real-world workflow. After launch, teams can look for unexpected outcomes, workarounds, differences in adoption, and changes in how people actually use the product.

Is this only relevant to AI?

No. AI makes some of these questions especially important because AI systems increasingly influence consequential decisions. But the same approach can be useful for healthcare products, learning technologies, digital platforms, and other systems where design decisions affect people's experiences or opportunities.

Does a human-centered review eliminate product risk?

No. There will always be risk and uncertainty. A good review helps a team see important risks that might otherwise remain hidden and gives leaders better information for deciding what to do about them.

References

Chen, Y., Clayton, E. W., Novak, L. L., Anders, S., & Malin, B. (2023). Human-centered design to address biases in artificial intelligence. Journal of Medical Internet Research, 25, e43251. https://doi.org/10.2196/43251

Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447–453. https://doi.org/10.1126/science.aax2342

Tabassi, E. (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). National Institute of Standards and Technology. https://doi.org/10.6028/NIST.AI.100-1

World Health Organization. (2021). Ethics and governance of artificial intelligence for health: WHO guidance. World Health Organization. https://www.who.int/publications/i/item/9789240029200