The death of risk assessment

Image by lapping from Pixabay

Risk assessment, as currently practiced in clinical trials, is dead.

I’m not saying we need to eliminate risk management altogether as a concept, but it needs a complete overhaul to deal with clinical data risk in the 21st century.

Our concept of risk as a static condition must evolve.

Clinical data risk should be viewed as organic and perpetually changing.

Clinical data risk should be computed from the live data.

And, we cannot ignore the inherent bias in human risk assessment.

We saw the movie “Blood Diamond” last night.

The phrase TIA — “This Is Africa” expresses fatalism with the problems and frustrations of Africa.

The way drug companies practice clinical data risk management reminded me of TIA. How the industry fatalistically accepts CRO change orders and the ineffectiveness of SDV.

Risk is dynamic

I think that risk is dynamic — it always has been — it’s just that the current inferno in financial markets reminds all of us, rather brutally, how dynamic it can be.

And then there is the link-baiting aspect of the title…

I think that it is incorrect to suggest that digital transformation of clinical trials, whatever that means, makes a difference to clinical data risk management.

In any case — whether it is a digital asset, financial asset, physical asset, patient safety asset, patient compliance asset, or data integrity asset — threats cause damage to assets and create risk.

We need to assess risk in a common language of brick and mortar security no matter what the asset is.

Brick and mortar security

What do I mean by “brick and mortar security”?

I’ll illustrate with a short story.

Our neighbor in LA was a retired professional baseball player. He and his wife were California blonde, athletic and attractive with 2 little California blonde, athletic and attractive children.

One day — I see Rick with a dog, a nice-sized German shepherd.

‘What’s with the dog Rick?’

‘Well — my wife has some jewelry, and we thought the dog would be good to scare off burglars. But — he’s so friendly, if a burglar broke in, he would just wag his tail and show the burglar to the jewelry. The kids love him.’

Linda’s jewelry are assets. The windows in the ground floor home are vulnerabilities. The threat is a thief. The security countermeasures are an alarm system, a sign on the lawn, and a dog.

Clinical data is a core intellectual asset of a life science company

Modern clinical trials collect clinical data using an assortment of eClinical applications: EDC, ePRO, eCOA, RTSM, CTMS, connected devices and labs.

Integrity of clinical data, protection of patient safety, clinical outcomes, and patient compliance are critical for the business, not just for clinical operations managers, investigators and for patients.

Risk management rolls up all the way from a connected device pill dispenser to the patient to the site coordinator to the clinical operations manager to the VP R&D to the CEO.

Conventional clinical data risk assessment is dead because it is based on 4 erroneous assumptions:

1 — Your quality system can mitigate high-impact change

• June 2021, you launched your Phase 3. You did your Initial Cross-Functional Risk Assessment. You involved multiple stakeholders and identified critical-to-quality risks across the entire trial lifecycle as well as mitigation strategies using the CRO RBQM.
• The markets were high and you were happy — planning a vacation in Belize after FDA approval and cash from a licensing event.
• March 10, 2022 — Silicon Valley Bank collapsed. Funding dried up. You fired your big 3 CRO and brought clinops in-house. Something your Cross-Functional Risk Assessment didn’t anticipate. Now, you don’t have access to the CRO RBQM either.
• May 1, 2023 — You amend the protocol to use half the dose you had last year and added a visit. Was the change implemented properly in Rave? How do you handle the old data? What is the impact on safety, compliance and the statistical analysis? Was the eCOA application updated properly? Were all the sites trained on-time? Was manufacturing and the supply chain updated properly? Is labeling correct? Did all the sites get the update? Do the new CRA’s know how to handle the old dose? Is the new data manager up to speed on the old data?
• June 1, 2023 — After the change, what is your new risk profile?

2 — You and your patients are on the same side of the table

• No. You are not.
• Sponsors and sites are about executing predictable processes.
• Patients are about creating unpredictable events.
• This is why line managers must ask themselves what threats might result in damaging events and what processes are vulnerable and need fixing.

3 — Clinical data risk is an independent variable that can be assessed

• In fact, GCP compliance risk is a dependent variable that can be computed as a function of assets (patient safety, data integrity, protocol compliance), vulnerabilities, threats and controls that mitigate the threats.
• A misconfigured Schedule of Events in the EDC is a vulnerability shared by data integrity and protocol adherence.
• Lab results out of range and marked clinically significant are a possible threat to safety. The frequency of the threat on various dimensions such as age and gender can be computed from the data, in order to determine the severity of the threat and how to mitigate safety risk.

4 — You can assess risk and ignore human bias

The astrophysicist Neil deGrasse Tyson counts 14 common types of bias that influence our decision making. Not all are relevant to clinical data risk management. I found 6 that are relevant:

1. Confirmation bias — your risk estimate supports something you already believe. You remember the hits and forget the misses. You spend your time on things that matter to you.
2. The Dunning-Kruger Effect — perceiving a concept to be simple, because your knowledge is limited. The less you know about something, the less complicated it appears. This form of bias limits curiosity as to the root cause of risk, because the problem is simple.
3. In-group bias — people are more likely to support or believe someone within their own social group than an outsider. This is why the industry is still doing SDV
4. Information bias — thinking that more information improves decision-making, even if that extra information is irrelevant. This is a topic for an entire essay — how sponsors collect much more clinical data than they actually need. This is relevant for clinical data risk management, since collecting more variables and dimensions results in more things can go wrong.
5. Availability bias — this mental shortcut that weights the first things you remember as valid and ignores alternative solutions or opinions. Similar to anchoring bias — which weights the first piece of information you receive.
6. Hindsight bias — the “I knew-it-all-along effect”. You overestimate your ability to predict an outcome beforehand, even though the information you had at the time would not have led you to the correct outcome. This can lead to overconfidence in risk assessment.

Summary

It seems to me, that while quality management is closely related to risk management, it cannot replace careful and critical understanding of the risk in your clinical data.

Computing the risk using the data, will help eliminate the various types of bias in your risk assessment, deal with change and cope with unpredictable patient events.