"Science Isn’t Broken"

"If we’re going to rely on science as a means for reaching the truth — and it’s still the best tool we have — it’s important that we understand and respect just how difficult it is to get a rigorous result. I could pontificate about all the reasons why science is arduous, but instead I’m going to let you experience one of them for yourself. Welcome to the wild world of p-hacking...

The data in our interactive tool can be narrowed and expanded (p-hacked) to make either hypothesis appear correct. That’s because answering even a simple scientific question — which party is correlated with economic success — requires lots of choices that can shape the results. This doesn’t mean that science is unreliable. It just means that it’s more challenging than we sometimes give it credit for.
Which political party is best for the economy seems like a pretty straightforward question. But as you saw, it’s much easier to get a result than it is to get an answer. The variables in the data sets you used to test your hypothesis had 1,800 possible combinations. Of these, 1,078 yielded a publishable p-value,¹ but that doesn’t mean they showed that which party was in office had a strong effect on the economy. Most of them didn’t...

P-hacking and similar types of manipulations often arise from human biases. “You can do it in unconscious ways — I’ve done it in unconscious ways,” Simonsohn said. “You really believe your hypothesis and you get the data and there’s ambiguity about how to analyze it.” When the first analysis you try doesn’t spit out the result you want, you keep trying until you find one that does. (And if that doesn’t work, you can always fall back on HARKing — hypothesizing after the results are known .)

Subtle (or not-so-subtle) manipulations like these plague so many studies that Stanford meta-science researcher John Ioannidis concluded, in a famous 2005 paper, that most published research findings are false. “It’s really difficult to perform a good study,” he told me, admitting that he has surely published incorrect findings too. “There are so many potential biases and errors and issues that can interfere with getting a reliable, credible result.”...

But the Internet is changing the way scientists distribute and discuss their ideas and data, which may make it harder to pass off shoddy papers as good science. Today when researchers publish a study, their peers are standing by online to discuss and critique it. Sometimes comments are posted on the journal’s own website in the form of “rapid responses,” and new projects such as PubMed Commons and PubPeer provide forums for rapid, post-publication peer review. Discussions about new publications also commonly take place on science blogs and social media, which can help spread information about disputed or corrected results...

But the Internet is changing the way scientists distribute and discuss their ideas and data, which may make it harder to pass off shoddy papers as good science. Today when researchers publish a study, their peers are standing by online to discuss and critique it. Sometimes comments are posted on the journal’s own website in the form of “rapid responses,” and new projects such as PubMed Commons and PubPeer provide forums for rapid, post-publication peer review. Discussions about new publications also commonly take place on science blogs and social media, which can help spread information about disputed or corrected results."... Media accounts of science tend to gloss over the nuance"
http://fivethirtyeight.com/features/science-isnt-broken/?ex_cid=538fb

Related: Yahoo Answers is not Research one

(Credit to KM and CT)