healthy_back_18 (![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png)
healthy_back_18) wrote2016-03-10 02:05 pm
healthy_back_18) wrote2016-03-10 02:05 pm
Entry tags:
Why Most Published Research Findings Are False
John P. A. Ioannidis
Published: August 30, 2005DOI: 10.1371/journal.pmed.0020124
http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124
Abstract
Summary
There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.
Published research findings are sometimes refuted by subsequent evidence, with ensuing confusion and disappointment. Refutation and controversy is seen across the range of research designs, from clinical trials and traditional epidemiological studies [1–3] to the most modern molecular research [4,5]. There is increasing concern that in modern research, false findings may be the majority or even the vast majority of published research claims [6–8]. However, this should not be surprising. It can be proven that most claimed research findings are false. Here I will examine the key factors that influence this problem and some corollaries thereof.
Modeling the Framework for False Positive Findings
Several methodologists have pointed out [9–11] that the high rate of nonreplication (lack of confirmation) of research discoveries is a consequence of the convenient, yet ill-founded strategy of claiming conclusive research findings solely on the basis of a single study assessed by formal statistical significance, typically for a p-value less than 0.05. Research is not most appropriately represented and summarized by p-values, but, unfortunately, there is a widespread notion that medical research articles should be interpreted based only on p-values. Research findings are defined here as any relationship reaching formal statistical significance, e.g., effective interventions, informative predictors, risk factors, or associations. “Negative” research is also very useful. “Negative” is actually a misnomer, and the misinterpretation is widespread. However, here we will target relationships that investigators claim exist, rather than null findings.
Bias
First, let us define bias as the combination of various design, data, analysis, and presentation factors that tend to produce research findings when they should not be produced. Let u be the proportion of probed analyses that would not have been “research findings,” but nevertheless end up presented and reported as such, because of bias. Bias should not be confused with chance variability that causes some findings to be false by chance even though the study design, data, analysis, and presentation are perfect. Bias can entail manipulation in the analysis or reporting of findings. Selective or distorted reporting is a typical form of such bias. We may assume that u does not depend on whether a true relationship exists or not. This is not an unreasonable assumption, since typically it is impossible to know which relationships are indeed true. In the presence of bias (Table 2), one gets PPV = ([1 - β]R + uβR)/(R + α − βR + u − uα + uβR), and PPV decreases with increasing u, unless 1 − β ≤ α, i.e., 1 − β ≤ 0.05 for most situations. Thus, with increasing bias, the chances that a research finding is true diminish considerably. This is shown for different levels of power and for different pre-study odds in Figure 1. Conversely, true research findings may occasionally be annulled because of reverse bias. For example, with large measurement errors relationships are lost in noise [12], or investigators use data inefficiently or fail to notice statistically significant relationships, or there may be conflicts of interest that tend to “bury” significant findings [13]. There is no good large-scale empirical evidence on how frequently such reverse bias may occur across diverse research fields. However, it is probably fair to say that reverse bias is not as common. Moreover measurement errors and inefficient use of data are probably becoming less frequent problems, since measurement error has decreased with technological advances in the molecular era and investigators are becoming increasingly sophisticated about their data. Regardless, reverse bias may be modeled in the same way as bias above. Also reverse bias should not be confused with chance variability that may lead to missing a true relationship because of chance.
Testing by Several Independent Teams
Several independent teams may be addressing the same sets of research questions. As research efforts are globalized, it is practically the rule that several research teams, often dozens of them, may probe the same or similar questions. Unfortunately, in some areas, the prevailing mentality until now has been to focus on isolated discoveries by single teams and interpret research experiments in isolation. An increasing number of questions have at least one study claiming a research finding, and this receives unilateral attention. The probability that at least one study, among several done on the same question, claims a statistically significant research finding is easy to estimate.
Corollaries
These corollaries consider each factor separately, but these factors often influence each other. For example, investigators working in fields where true effect sizes are perceived to be small may be more likely to perform large studies than investigators working in fields where true effect sizes are perceived to be large. Or prejudice may prevail in a hot scientific field, further undermining the predictive value of its research findings. Highly prejudiced stakeholders may even create a barrier that aborts efforts at obtaining and disseminating opposing results. Conversely, the fact that a field is hot or has strong invested interests may sometimes promote larger studies and improved standards of research, enhancing the predictive value of its research findings. Or massive discovery-oriented testing may result in such a large yield of significant relationships that investigators have enough to report and search further and thus refrain from data dredging and manipulation.
Claimed Research Findings May Often Be Simply Accurate Measures of the Prevailing Bias
As shown, the majority of modern biomedical research is operating in areas with very low pre- and post-study probability for true findings. Let us suppose that in a research field there are no true findings at all to be discovered. History of science teaches us that scientific endeavor has often in the past wasted effort in fields with absolutely no yield of true scientific information, at least based on our current understanding. In such a “null field,” one would ideally expect all observed effect sizes to vary by chance around the null in the absence of bias. The extent that observed findings deviate from what is expected by chance alone would be simply a pure measure of the prevailing bias.
http://prepareforchange.net/2016/02/18/meet-the-doctor-who-says-prescription-drugs-are-killing-us-and-hes-not-the-only-one/
Meet The Doctor Who Says Prescription Drugs Are Killing Us – And He’s Not The Only One
February 18, 2016 by Dane Arr
Dr. Peter Gotzsche, co-founder of the Cochrane Collaboration (the world’s most foremost body in assessing medical evidence), hopes to make clear that here is a reason why the most widely accessed article in the history of the Public Library of Science (PLoS) is entitled, Why Most Published Research Findings Are False. In the report, researchers stated that most current published research findings are false, and this was more than 10 years ago.
When it comes to our health, taking one person’s word as doctrine might not be the best idea, whether they are a doctor or not. What one person truly believes to be the best course of action in treating an illness may be the last thing someone else recommends, depending on a complex range of factors, including where and how they were educated, and, in particular, who funded that education. Indeed, many concerns have been raised about the use of industry-accepted pharmaceuticals, often by the very doctors who were told to use them. We now have, moreover, an overwhelming amount of evidence to corroborate what many of these professionals have been trying to tell us for decades:
The medical profession is being bought by the pharmaceutical industry, not only in terms of the practice of medicine, but also in terms of teaching and research. The academic institutions of this country are allowing themselves to be the paid agents of the pharmaceutical industry. I think it’s disgraceful.
– Arnold Seymour Relman (1923-2014), Harvard Professor of Medicine and Former Editor-in-Chief of the New England Medical Journal (source)(source)
There is a reason why the most widely accessed article in the history of the Public Library of Science (PLoS) is entitled, Why Most Published Research Findings Are False. In the report, researchers stated that most current published research findings are false, and this was more than 10 years.
Dr. Peter Gotzsche, co-founder of the Cochrane Collaboration (the world’s most foremost body in assessing medical evidence), hopes to make clear this very problem. He is currently working to inform the world about the dangers associated with several pharmaceutical grade drugs. Based on his research, he estimates that 100,000 people in the United States alone die each year from the side-effects of correctly used prescription drugs, noting that “it’s remarkable that nobody raises an eyebrow when we kill so many of our own citizens with drugs.” He published a paper last year in the Lancet arguing that our use of antidepressants is causing more harm than good, and taking into consideration the recent leaks regarding antidepressant drugs, it seems he is correct.
The most recent example of this kind of corruption in relation to antidepressants comes from a study that was published last week in the British Medical Journal by researchers at the Nordic Cochrane Center in Copenhagen. The study showed that pharmaceutical companies were not disclosing all information regarding the results of their drug trials:
[This study] confirms that the full degree of harm of antidepressants is not reported. They are not reported in the published literature, we know that – and it appears that they are not properly reported in clinical study reports that go to the regulators and from the basis of decisions about licensing. (http://www.scientificamerican.com/article/the-hidden-harm-of-antidepressants/)
Researchers looked at documents from 70 different double-blind, placebo-controlled trials of selective serotonin reuptake inhibitors (SSRI) and serotonin and norepinephrine reuptake inhibitors (SNRI) and found that the full extent of serious harm in clinical study reports went unreported. These are the reports sent to major health authorities like the U.S. Food and Drug Administration.
Tamang Sharma, a PhD student at Cochrane and lead author of the study, said:
We found that a lot of the appendices were often only available upon request to the authorities, and the authorities had never requested them. I’m actually kind of scared about how bad the actual situation would be if we had the complete data.
This is not the first time that pharmaceutical companies have been caught manipulating science in order to get antidepressants onto the shelves. It was only a couple of months ago that an independent review found that the commonly prescribed antidepressant drug Paxil (paroxetine) is not safe for teenagers, even though a large amount of literature had already suggested this previously. The 2001 drug trial that took place, funded by GlaxoSmithKline, found that these drugs were completely safe, and used that ‘science’ to market Paxil as safe for teenagers.
Gotzche’s two main areas of focus are antidepressants and “non-steroidal anti-inflammatory” painkillers like ibuprofen, tylenol, celecoxib, and diclofenac. Another is Vioxx, which was actually withdrawn after it was discovered that it caused more than 100,000 cases of serious heart disease in the United States during the five years that it was on the market.
According to Gotzche, these deaths are just the tip of the iceberg when it comes to the failure of the drug regulatory process to protect patients:
These terms for our drugs are invented by the drug industry. They had a huge financial interest in calling these things anti-inflammatory. It lured doctors into believing that these drugs somehow also had an effect on the disease process and reduced the joint damage.
In his paper he also notes that antidepressants have replaced drugs that were found to be harmful, like Valium and Xanax, but are just as addictive and their side effects just as dangerous.
According to Professor Gotzsche, here’s a list of things you want to avoid:
- Antidepressants for all, because they probably don’t work for severe cases of depression
- All brain-active drugs in children
- Anti-psychotics and other brain-active drugs for the elderly. Psychotropic drugs should be used as little as possible and mostly in very acute situations, as they are very harmful when used long term
- Non-steroidal anti-inflammatory drugs used for arthritis, muscle pain and headaches, including over-the-counter, low dose ibuprofen. These drugs should be used as little as possible
- Mammography screening, as it doesn’t prolong life whereas it makes many healthy women ill through over diagnosis and leads to the premature death for some because radiotherapy and chemotherapy increases mortality when used for harmless cancers detected at screening.
- Drugs for urinary incontinence, as they very likely don’t work
“The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue. Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance, science has taken a turn towards darkness.” – Dr. Richard Horton, the current Editor-In-Chief of the Lancet (source)
Here is a great video that I share in most of my articles that have to do with this topic. It’s a clip of Dr. Peter Rost, a former vice president of Pfizer and a whistleblower of the pharmaceutical industry. Author of “The Whistleblower, Confessions of a Healthcare Hitman,” Rost is an insider expert on big pharma marketing.
Source: Collective Evolution February 17, 2016 by Arjun Walia
Published: August 30, 2005DOI: 10.1371/journal.pmed.0020124
http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124
Abstract
Summary
There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.
Published research findings are sometimes refuted by subsequent evidence, with ensuing confusion and disappointment. Refutation and controversy is seen across the range of research designs, from clinical trials and traditional epidemiological studies [1–3] to the most modern molecular research [4,5]. There is increasing concern that in modern research, false findings may be the majority or even the vast majority of published research claims [6–8]. However, this should not be surprising. It can be proven that most claimed research findings are false. Here I will examine the key factors that influence this problem and some corollaries thereof.
Modeling the Framework for False Positive Findings
Several methodologists have pointed out [9–11] that the high rate of nonreplication (lack of confirmation) of research discoveries is a consequence of the convenient, yet ill-founded strategy of claiming conclusive research findings solely on the basis of a single study assessed by formal statistical significance, typically for a p-value less than 0.05. Research is not most appropriately represented and summarized by p-values, but, unfortunately, there is a widespread notion that medical research articles should be interpreted based only on p-values. Research findings are defined here as any relationship reaching formal statistical significance, e.g., effective interventions, informative predictors, risk factors, or associations. “Negative” research is also very useful. “Negative” is actually a misnomer, and the misinterpretation is widespread. However, here we will target relationships that investigators claim exist, rather than null findings.
Bias
First, let us define bias as the combination of various design, data, analysis, and presentation factors that tend to produce research findings when they should not be produced. Let u be the proportion of probed analyses that would not have been “research findings,” but nevertheless end up presented and reported as such, because of bias. Bias should not be confused with chance variability that causes some findings to be false by chance even though the study design, data, analysis, and presentation are perfect. Bias can entail manipulation in the analysis or reporting of findings. Selective or distorted reporting is a typical form of such bias. We may assume that u does not depend on whether a true relationship exists or not. This is not an unreasonable assumption, since typically it is impossible to know which relationships are indeed true. In the presence of bias (Table 2), one gets PPV = ([1 - β]R + uβR)/(R + α − βR + u − uα + uβR), and PPV decreases with increasing u, unless 1 − β ≤ α, i.e., 1 − β ≤ 0.05 for most situations. Thus, with increasing bias, the chances that a research finding is true diminish considerably. This is shown for different levels of power and for different pre-study odds in Figure 1. Conversely, true research findings may occasionally be annulled because of reverse bias. For example, with large measurement errors relationships are lost in noise [12], or investigators use data inefficiently or fail to notice statistically significant relationships, or there may be conflicts of interest that tend to “bury” significant findings [13]. There is no good large-scale empirical evidence on how frequently such reverse bias may occur across diverse research fields. However, it is probably fair to say that reverse bias is not as common. Moreover measurement errors and inefficient use of data are probably becoming less frequent problems, since measurement error has decreased with technological advances in the molecular era and investigators are becoming increasingly sophisticated about their data. Regardless, reverse bias may be modeled in the same way as bias above. Also reverse bias should not be confused with chance variability that may lead to missing a true relationship because of chance.
Testing by Several Independent Teams
Several independent teams may be addressing the same sets of research questions. As research efforts are globalized, it is practically the rule that several research teams, often dozens of them, may probe the same or similar questions. Unfortunately, in some areas, the prevailing mentality until now has been to focus on isolated discoveries by single teams and interpret research experiments in isolation. An increasing number of questions have at least one study claiming a research finding, and this receives unilateral attention. The probability that at least one study, among several done on the same question, claims a statistically significant research finding is easy to estimate.
Corollaries
These corollaries consider each factor separately, but these factors often influence each other. For example, investigators working in fields where true effect sizes are perceived to be small may be more likely to perform large studies than investigators working in fields where true effect sizes are perceived to be large. Or prejudice may prevail in a hot scientific field, further undermining the predictive value of its research findings. Highly prejudiced stakeholders may even create a barrier that aborts efforts at obtaining and disseminating opposing results. Conversely, the fact that a field is hot or has strong invested interests may sometimes promote larger studies and improved standards of research, enhancing the predictive value of its research findings. Or massive discovery-oriented testing may result in such a large yield of significant relationships that investigators have enough to report and search further and thus refrain from data dredging and manipulation.
Claimed Research Findings May Often Be Simply Accurate Measures of the Prevailing Bias
As shown, the majority of modern biomedical research is operating in areas with very low pre- and post-study probability for true findings. Let us suppose that in a research field there are no true findings at all to be discovered. History of science teaches us that scientific endeavor has often in the past wasted effort in fields with absolutely no yield of true scientific information, at least based on our current understanding. In such a “null field,” one would ideally expect all observed effect sizes to vary by chance around the null in the absence of bias. The extent that observed findings deviate from what is expected by chance alone would be simply a pure measure of the prevailing bias.
http://prepareforchange.net/2016/02/18/meet-the-doctor-who-says-prescription-drugs-are-killing-us-and-hes-not-the-only-one/
Meet The Doctor Who Says Prescription Drugs Are Killing Us – And He’s Not The Only One
February 18, 2016 by Dane Arr
Dr. Peter Gotzsche, co-founder of the Cochrane Collaboration (the world’s most foremost body in assessing medical evidence), hopes to make clear that here is a reason why the most widely accessed article in the history of the Public Library of Science (PLoS) is entitled, Why Most Published Research Findings Are False. In the report, researchers stated that most current published research findings are false, and this was more than 10 years ago.
When it comes to our health, taking one person’s word as doctrine might not be the best idea, whether they are a doctor or not. What one person truly believes to be the best course of action in treating an illness may be the last thing someone else recommends, depending on a complex range of factors, including where and how they were educated, and, in particular, who funded that education. Indeed, many concerns have been raised about the use of industry-accepted pharmaceuticals, often by the very doctors who were told to use them. We now have, moreover, an overwhelming amount of evidence to corroborate what many of these professionals have been trying to tell us for decades:
The medical profession is being bought by the pharmaceutical industry, not only in terms of the practice of medicine, but also in terms of teaching and research. The academic institutions of this country are allowing themselves to be the paid agents of the pharmaceutical industry. I think it’s disgraceful.
– Arnold Seymour Relman (1923-2014), Harvard Professor of Medicine and Former Editor-in-Chief of the New England Medical Journal (source)(source)
There is a reason why the most widely accessed article in the history of the Public Library of Science (PLoS) is entitled, Why Most Published Research Findings Are False. In the report, researchers stated that most current published research findings are false, and this was more than 10 years.
Dr. Peter Gotzsche, co-founder of the Cochrane Collaboration (the world’s most foremost body in assessing medical evidence), hopes to make clear this very problem. He is currently working to inform the world about the dangers associated with several pharmaceutical grade drugs. Based on his research, he estimates that 100,000 people in the United States alone die each year from the side-effects of correctly used prescription drugs, noting that “it’s remarkable that nobody raises an eyebrow when we kill so many of our own citizens with drugs.” He published a paper last year in the Lancet arguing that our use of antidepressants is causing more harm than good, and taking into consideration the recent leaks regarding antidepressant drugs, it seems he is correct.
The most recent example of this kind of corruption in relation to antidepressants comes from a study that was published last week in the British Medical Journal by researchers at the Nordic Cochrane Center in Copenhagen. The study showed that pharmaceutical companies were not disclosing all information regarding the results of their drug trials:
[This study] confirms that the full degree of harm of antidepressants is not reported. They are not reported in the published literature, we know that – and it appears that they are not properly reported in clinical study reports that go to the regulators and from the basis of decisions about licensing. (http://www.scientificamerican.com/article/the-hidden-harm-of-antidepressants/)
Researchers looked at documents from 70 different double-blind, placebo-controlled trials of selective serotonin reuptake inhibitors (SSRI) and serotonin and norepinephrine reuptake inhibitors (SNRI) and found that the full extent of serious harm in clinical study reports went unreported. These are the reports sent to major health authorities like the U.S. Food and Drug Administration.
Tamang Sharma, a PhD student at Cochrane and lead author of the study, said:
We found that a lot of the appendices were often only available upon request to the authorities, and the authorities had never requested them. I’m actually kind of scared about how bad the actual situation would be if we had the complete data.
This is not the first time that pharmaceutical companies have been caught manipulating science in order to get antidepressants onto the shelves. It was only a couple of months ago that an independent review found that the commonly prescribed antidepressant drug Paxil (paroxetine) is not safe for teenagers, even though a large amount of literature had already suggested this previously. The 2001 drug trial that took place, funded by GlaxoSmithKline, found that these drugs were completely safe, and used that ‘science’ to market Paxil as safe for teenagers.
Gotzche’s two main areas of focus are antidepressants and “non-steroidal anti-inflammatory” painkillers like ibuprofen, tylenol, celecoxib, and diclofenac. Another is Vioxx, which was actually withdrawn after it was discovered that it caused more than 100,000 cases of serious heart disease in the United States during the five years that it was on the market.
According to Gotzche, these deaths are just the tip of the iceberg when it comes to the failure of the drug regulatory process to protect patients:
These terms for our drugs are invented by the drug industry. They had a huge financial interest in calling these things anti-inflammatory. It lured doctors into believing that these drugs somehow also had an effect on the disease process and reduced the joint damage.
In his paper he also notes that antidepressants have replaced drugs that were found to be harmful, like Valium and Xanax, but are just as addictive and their side effects just as dangerous.
According to Professor Gotzsche, here’s a list of things you want to avoid:
- Antidepressants for all, because they probably don’t work for severe cases of depression
- All brain-active drugs in children
- Anti-psychotics and other brain-active drugs for the elderly. Psychotropic drugs should be used as little as possible and mostly in very acute situations, as they are very harmful when used long term
- Non-steroidal anti-inflammatory drugs used for arthritis, muscle pain and headaches, including over-the-counter, low dose ibuprofen. These drugs should be used as little as possible
- Mammography screening, as it doesn’t prolong life whereas it makes many healthy women ill through over diagnosis and leads to the premature death for some because radiotherapy and chemotherapy increases mortality when used for harmless cancers detected at screening.
- Drugs for urinary incontinence, as they very likely don’t work
“The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue. Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance, science has taken a turn towards darkness.” – Dr. Richard Horton, the current Editor-In-Chief of the Lancet (source)
Here is a great video that I share in most of my articles that have to do with this topic. It’s a clip of Dr. Peter Rost, a former vice president of Pfizer and a whistleblower of the pharmaceutical industry. Author of “The Whistleblower, Confessions of a Healthcare Hitman,” Rost is an insider expert on big pharma marketing.
Source: Collective Evolution February 17, 2016 by Arjun Walia