Last July, I blogged about an article published in BMJ Archives of Disease in Childhood that seemed to validate the so-called Lightning Process – a mishmash of “neuro-linguistic programming,” life coaching, and “emotional self-regulation” – as a potentially effective treatment for ME/CFS in children. David Tuller, of the University of California, documented glaring flaws in the study design, including recruitment of more than half the participants before registering the trial and swapping primary and secondary outcomes after the study was in progress.
These changes allowed the investigators to report a positive outcome – thus making the article more publishable – without including any acknowledgement in the article itself. An open letter to BMJ, signed by numerous scientists, physicians, and academics (including me), sought retraction of the article, which led to a lengthy “correction” by Archives of Disease in Childhood. Shockingly, however, the journal did not retract the article, following an “assurance from the authors that the change in primary outcome was not influenced by (positive) findings in the feasibility phase.” As I explained in my original post, basic conflict of interest principles – applicable in law and many other fields – should not have allowed the authors themselves to decide whether or not their results had been improperly influenced.
Many of the signatories to the original open letter have now written personal letters to Dr. Fiona Godlee, who is BMJ’s editor-in-chief, overseeing all of its journals (BMJ was previously British Medical Journal). The strongest among these, in my opinion, was written by Rebecca Goldin, professor of Mathematical Science at George Mason University and director of STATS, a collaborative project of the American Statistical Association.
I have included Prof. Goldin’s full letter after the jump. It is short and powerful, and an excellent example of persuasive writing that could serve as a model for lawyers and law students.
Dear Dr. Godlee,
Like many doctors and scientists, I signed a petition requesting that BMJ retract the article “Clinical and cost-effectiveness of the Lightning Process…” I am writing to express again my grave concern about BMJ’s publication practice as well as the results of the article.
As a mathematician and Director of STATS (a nonprofit involved with the accurate reporting of statistical aspects of medical research), I reviewed the article with great interest in its public face. Mainly, journalists and parents want to know whether the results “are true” and the methods “are valid”. A modest effort at detangling the research methods makes evident damning weaknesses of the article.
The experimental design of this study seems like a textbook case of why the public shouldn’t trust scientists. Post-facto decisions about outcome measurements (such as swapping primary and secondary outcomes), and pursuing explicitly forbidden scientific practices (such as recruiting participants trial registration through a feasibility study) increase the likelihood of false positives; when there are scientific and business motivations to do such switches, all credibility is lost.
I am frankly surprised that BMJ would be willing to risk its reputation to defend a study that looks on the outside like a corporate front. To title the piece a “randomized controlled trial” is misleading as the participants were not randomly selected. To suggest that there is a clinical benefit when the data were so easily manipulated in ways we know about implies that brokers of scientific knowledge are no longer able to stem the poor science with corporate motives from entering the scientific mainstream. To suggest that the method is “cost effective” while the paper advertises the company’s dubiously “proven” methods suggests to the public that the BMJ has motives that extend past the health of these children. Public suspicion of science is founded on poorly conceived “results" getting institutional endorsements such as a publication in BMJ.
Unfortunately for children who suffer from ME/CFE, BMJ still has a big impact on treatment protocols. You personally have the opportunity to stand for quality science, and BMJ should do so. Rescind the paper based on BMJ policies, admitted disinformation by the authors of the study, and data whose quality cannot be assessed due to collection methods. There is no harm to suggest that science can be done better by rescinding the paper, especially considering the data manipulation that has already been discovered. In contrast, the harm stemming from publishing results that are essentially cooked up will have a lasting impact both on children who need high quality research, on BMJ’s reputation, and on public trust of medical science.
—
Rebecca Goldin, Ph.D.
Professor, Mathematical Sciences
George Mason University
Director, STATS
Sense About Science
@rebegol