The trolls are preventing our happiness

I can’t get no satisfaction. The Rolling Stones

Not coming from the world of clinical research, I still feel like an outsider. I’m a legal alien — a cyber guy in the New York of clinical research.

In 2013, I had my first opportunity to see a pivotal clinical trial data management operation. It was a mess. They had 185 patients at 15 sites in the US and over 6,000 queries. The CRO was spending 6 months cleaning data and still not at the finish line. I was a cyber consultant to the sponsor and I asked what was the problem. They were sad. They said — “well, you know, there is a lot of data”. I counted 185,000 data point and said to the CRO data manager “Look this is nothing. I was working with mobile operators in Central Europe with 10 million subscribers and 100,000 cyber events on their network every day. We detected issues in real-time using automated detection and response technology. Suspected data loss events were prioritized. If they warranted forensics investigation, a forensics team took over. If an event was detected inside the organization, a person from the security walked over to the suspect. If dumpster diving was needed, a person would put on galoshes and a rain-coat and dive into the dumpsters.”

It seemed that clinical operations people are not happy. I thought it might have something to do with all that paper, queries and data management loops which are antiquated, annoying and slow-moving. I was inspired to coin the following phrase by Scott Adams’ accounting trolls: Trolls in the data management dungeon.

COVID-19 and the sense of urgency

The COVID-19 pandemic that broke out in February 2020 created a sense of extreme urgency for vaccines, therapeutics and diagnostics. As of writing, NIH is trying out an experimental mRNA vaccine on human subjects. Requests are being made to FDA to drop requirements for proving efficacy and safety in large Phase 3 trials.

The constraint to delivering a safe, effective vaccine will be neither science, nor FDA. The constraint will be the data integrity of the Phase 2 and Phase 3 studies.

Phase 3 studies are meant to provide interpretable data for efficacy and safety in a statistically significant sample in order to enable FDA to take a decision to release the product to market.

Data integrity issues are created by variability in clinical protocol execution by the research sites conducting the trial. Badly collected outcomes include missing data, poorly specified outcomes and under-reporting of adverse events.

Authors of a systematic review of 235 RCTs found that 19% of the trials were no longer significant, based on assumptions that the losses to follow-up actually had the outcome of interest. This figure was 58% in a worst-case scenario, where all participants lost to follow-up in the intervention group and none in the control group had the event of interest.

A review of 11 studies reported that 43% to 100% (median 64%) of adverse events (including outcomes such as death or suicide) were missed.

In order to achieve interpretable data, life science companies contract with service organizations who use manual, batch-oriented processes. There is typically a 4–13 week delay to log protocol deviations dictated by the rhythm of CRA site-monitoring visits and centralized data management and monitoring activities. During this window, non-compliant patients go undetected and eventually must be replaced, requiring recruitment of replacements; incurring further delays. After the last patient visit, a data cleaning process performed by a data management team, adding another 3–6 months to study completion.

Cumulative delays due to increased recruitment load and data cleaning in a Phase 2–3 trial can add an additional 12–18 months to trial completion.

So what can we do about this?

More than I have time for tonight. Next post.