#130 - Theatre or Assurance?

5 Nov 26

A design review can look rigorous from the outside: the right people in the room, a full agenda and a unanimous decision at the end. It can also be pure theatre. The test is whether anyone genuinely established that the technical solution is safe, secure and in accordance with the customer's requirements. The Chair ultimately decides which kind of review it turns out to be.

Trial Brief

A design review exists to answer one question on behalf of the organisation and its customer: is this technical solution safe, secure and in accordance with the customer's requirements? Everything else in the review, from risk registers and mitigations to design maturity, is a means of answering it honestly. When a review does that, it finds what the design team couldn't see and gives the organisation a true picture of where it stands. When it doesn't, it provides a veneer of assurance with nothing behind it, which is worse than having no review at all, because it creates false confidence.

Trial Execution

Too often, the review is seen as a hurdle to get past. The design team wants the gate passed so the programme can move on. The reviewers have other priorities and limited time. The Chair is under pressure to keep to the schedule. Everyone goes through the motions, the risk register is noted, the box is ticked and the design proceeds.

Nobody in that room has to be acting in bad faith. The incentives point the wrong way. The design team is rewarded for passing, reviewers are rarely rewarded for finding problems and the programme is rewarded for staying on schedule. The path of least resistance is a review that approves what it is shown.

The difference between theatre and assurance is whether the three questions are actually tested. Is it safe? Is it secure? Does it meet what the customer asked for? It's easy to record answers to those. It's much harder to test them. A risk register full of entries marked "mitigated" proves nothing unless someone has checked that each mitigation exists, will work and is supported by evidence. A design described as "mature" proves nothing unless the evidence of its maturity has been examined.

The Chair is the person who makes that happen, by insisting on the depth, protecting the time for it and making it safe for others to challenge.

Trial Findings

Three questions the review has to answer

Is it safe?

• Have the hazards been identified, including those arising from failure, misuse and interaction with other systems?
• Are the safety requirements and standards that apply understood, and is there evidence the design meets them?
• Is the safety case coherent and credible, or is it a collection of statements?
• Who has accepted any residual safety risk, and do they have the authority to do so?

Is it secure?

• Have the threats and vulnerabilities been assessed, and do the controls address them?
• Are security requirements built into the design, or being added afterwards?
• Do the interfaces, data flows and dependencies, including the supply chain, introduce exposure?
• Is there any tension between safety and security measures, and has it been resolved deliberately?

Is it in accordance with the customer's requirements?

• Are the requirements understood and agreed, and are they stable or still changing? Can you eventually get off contract?
• Is every requirement traceable to the design and to a verification method?
• Where the design departs from a requirement, is the departure known, recorded and agreed with the customer?
• Is the review testing against what the customer actually asked for, or what the team believes they meant?

How the review tests those answers

A good Chair uses risk, mitigation and maturity as the tools for testing the three questions.

Technical risk and programme risk are two views of the same problem. An unresolved technical risk becomes a programme risk through delay, rework and cost, and a compressed programme creates technical risk through shortcuts and untested assumptions. A review that looks at only one will miss the other. For each significant risk, a Chair should be able to establish:

• Is the risk correctly identified and honestly rated? Is it described in terms of cause, event and consequence? What isn't on the register that should be? Be wary of risks that stay low in the face of unresolved technical questions.
• Is there a real mitigation? A specific action with an owner, a date and resource, and not a statement of intent such as "will be monitored".
• Will the mitigation work? What is the evidence? Analysis, test results, prior experience or a proven precedent are evidence. Confidence and assurances aren't.
• Does the mitigation create new risk? A fix that adds complexity, an interface or a dependency can move the problem rather than remove it.
• What is the residual risk, and who accepts it?
• Is the programme realistic? A mitigation that needs three months of testing in a plan that allows three weeks isn't a mitigation.

Maturity is the question that review packs are least likely to answer honestly. A design can have a complete set of drawings and still be immature, if the critical questions behind them are unresolved. A Chair can probe it through:

• Evidence versus assertion - How much of the design's performance is demonstrated through analysis, test or prior use, and how much is assumed? Where the claim is "we expect", the design isn't yet mature.
• Requirements and interfaces - Are they defined, agreed and controlled? Interfaces are where immature designs usually fail.
• Open issues - How many remain, how significant are they, and are they closing or accumulating from review to review?
• Verification and validation - Is there a credible plan, tested against the risks, and is it resourced?
• Readiness for the next stage - Does the design meet the agreed criteria to progress, or is it moving on because the date has arrived?

The most useful question is often the simplest: what would we have to be wrong about for this design to be unsafe, insecure or non-compliant, and how do we know we aren't?

The warning signs that lip service is being paid

In my experience, these are the signs that a review has drifted from assessment towards theatre.

Before the review:

• Late or thin material - Packs that arrive the night before, or are too large or poorly organised to be read.
• Claims without evidence - "Safe", "secure", "compliant" or "all risks mitigated" with no supporting analysis, test data or traceability.
• The outcome is assumed - Milestones and payments depend on a successful review, so a fail isn't a realistic option.
• Missing or token reviewers - Safety, security or the customer's interests aren't represented, or are represented by someone without the experience or authority to challenge.

In the room:

• Silence - A design of any complexity raises questions. Long stretches without any usually mean people haven't engaged or don't feel safe to speak.
• Presentation instead of examination - The design team talks through slides for the whole session, leaving little time for scrutiny.
• Easy questions - Challenge stays on formatting and labelling and avoids the hard questions about assumptions, interfaces, failure modes and compliance.
• Defensiveness and deflection - Questions treated as personal attacks, or "we'll take that offline" used to close a topic.
• Agreement that comes too easily - A review that finishes early with no disagreement deserves suspicion, not congratulations.

After the review:

• Vague or trivial actions - Actions with no owner or date, or "to be considered".
• Mitigations that never close - Risks that are closed without evidence, or because the date passed.
• The same findings recur - Issues that surface review after review point to a process that records problems without correcting them.
• Risks only go down - If the risk profile improves at every review without corresponding evidence, the process is probably being managed, not tested.

How a Chair can spot them

A Chair can look for these signs before and during the review, not afterwards, by asking:

• Who has read the pack, and how long did they have?
• What evidence sits behind the claim that this is safe, secure and compliant?
• What isn't in the pack that should be?
• What would change our decision? If nothing could, why are we holding the review?

Tracking the pattern over time helps too. How many reviews raised significant findings? How many ended with a fail or a conditional pass? How many mitigations were closed with evidence, and how many on a date? A process with no findings and no pushback isn't evidence of excellent design, it's evidence that the review isn't doing its job.

How to make challenge healthy and real

Challenge depends on psychological safety. People have to believe that raising a concern will be heard on its merits and not punished. A few things the Chair controls make the difference:

• Set the tone at the start - State that the purpose is to find problems while they are cheap to fix, and that challenge is expected and not personal.
• Protect the time - Limit the presentation, insist on adequate scrutiny and push back on schedule pressure.
• Draw people out - Ask quieter reviewers directly, particularly junior ones, and give them room to answer.
• Ask the questions others won't - Probe the claims that rest on the least evidence.
• Respond well to challenge - How the Chair treats the first difficult question determines whether there's a second one.
• Make a fail a real option - If a review can't fail, it isn't a review. Be prepared to withhold approval, or approve with conditions, even when it's uncomfortable.
• Hold the line on actions - Every finding needs an owner, a date and closure evidence.

How to fix the underlying attitude

If lip service has become the norm, one good review won't fix it. The attitude is cultural, and has to be addressed at that level.

• Fix the incentives - If the programme is rewarded only for passing, it will aim only to pass. Recognise teams that surface problems early and treat a hard review as a success for the process.
• Separate the review from schedule and cost pressure - Protect the independence of the engineering judgement so that the Chair isn't deciding under delivery pressure.
• Back the Chair - A Chair who fails a design needs support from senior leaders. If they are overruled or blamed, others will learn not to try.
• Make sure reviewers are competent and authorised - They need the right experience, the time to prepare and the authority to challenge.
• Share the learning - Show what reviews have caught and what it would have cost if they hadn't. That makes the value visible and changes how the process is seen.

The Common Thread

A design review is only as good as its answer to one question: is this solution safe, secure and in accordance with the customer's requirements? It has the form of assurance whenever the right people are in the room and the right process is followed, but it only delivers assurance when someone tests the evidence behind each of those three claims, along with the risks, mitigations and maturity that support them. The Chair is responsible for creating those conditions. When they do, the review becomes a deep dive. When they don't, it becomes theatre with a signature at the end.

Trial Sign-off

One practical action

Take your most recent design review and ask what evidence supported each of three claims: that the solution is safe, that it is secure, and that it meets the customer's requirements. If the answers are assurances rather than evidence, you've found where to dig.

One question for reflection

If an accident investigator, a regulator or the customer asked how your last design review satisfied itself that the solution was safe, secure and compliant, what would you show them?

Have a great week!

Steve

If you're interested in through-life Engineering Leadership development from first time Engineering leader/manager, through Head of Engineering up to Director, see the link here.

For short courses which target specific leadership issues, see this link.