What Regulated Environments Taught Me About AI Failure Design

Financial services has a design culture around failure that most consumer tech doesn’t. Not because financial services designers are better — in many respects the UX is worse — but because the regulatory and liability environment forces explicit engagement with failure modes that consumer products can ignore until they become PR problems.

In regulated environments you don’t get to say “we’ll iterate on the failure cases.” You have to characterize them in advance, design for them explicitly, and defend your design decisions to parties who are specifically looking for gaps. That discipline produces a specific kind of design rigor that I’ve found almost entirely absent in AI product design.

When I look at most AI product failure design, what I see is: generic disclaimers applied uniformly, confidence indicators that don’t vary based on actual model reliability, and failure states that are designed for the modal failure rather than the consequential one. The modal failure — model produces a slightly wrong output, user notices, user corrects — is handled adequately. The consequential failure — model produces a confidently wrong output in a high-stakes context, user doesn’t notice, user acts on it — is barely designed for.

In financial services, the consequential failure is the design target. You don’t get credit for handling the common case gracefully if you haven’t demonstrated you’ve thought carefully about the tail. The tail is where the liability lives, and more importantly, it’s where the actual harm lives.

AI products are increasingly operating in consequential domains. The design discipline needs to catch up. That means failure characterization before deployment, not after. It means differentiating between failure modes by consequence severity, not just by frequency. It means designing failure states with the same care as success states, and accepting that this will slow some things down.

The alternative is a series of public failures in high-stakes domains that were predictable and weren’t predicted because nobody was doing the work.