Picking a small thing that happened to be significant – Sometimes studies set out to see if intervention A has an effect on outcomes 1, 2 or 3. But they find no change in those outcomes, so they comb the data looking for SOMETHING, and come up with some small difference in some small subgroup. …
They are everywhere. Research articles, abstracts, press releases, and news articles touting that such and such will DOUBLE THE RISK of some dread happening. Or cut the risk of some dread happening in half, depending on what slant they want to take on it. Describing risk this way makes small differences seem large. Which is …
One thing that researchers sometimes do is to keep studies smaller in scope to minimize the scope and size. This is often a budgetary thing. So a study might be limited to first time parents with no complications. That’s normal and appropriate, but as a reader, we need to remember that those limited samples don’t …
Research Trouble Spot 8: Sample isn’t representative Read More »
Peer review is the process of having experts in the field carefully review studies before publication to ensure they meet a high standard of quality. While it’s not a perfect system, it is a key part of the process, and studies that do not go through the peer review process should be taken with a …
Blinding is when the researchers don’t tell which group people are in. It’s done to avoid any intentional or unconscious bias in how the study is carried out. There are two kinds of blinding: Participant Blinding where the people getting treatment in the study don’t know which treatment group they are in. Researcher Blinding where …
It’s pretty common for people just starting to read research to confuse correlation (when two things seem to change together) with causation (when one thing causes another). Correlation can be a clue towards finding causation, but it is not the same thing. There are lots of reasons why two data points might be correlated without …
Research Trouble Spot 5: Correlation – Causation Confusion Read More »
How big does a study need to be in order to be a valid study? This is a question that actually has an answer! There’s a way to calculate that from the outset of the study. It’s called a power analysis. it involves all kinds of factors like the number of variables being studied, the …
Experimental studies that show something is safe or improves outcomes should, by definition, have something to compare the new treatment against. If they don’t, it’s hard to know if the new treatment is a better option or not. Control groups can come in several varieties: No Treatment Group – Researchers would compare the outcomes with …
Sometimes the conflict is really easy to see. The stereotype of the used car salesman telling you that he knows the little old lady who used to own this car, and she only ever used it to drive to the grocery store and church, and followed the maintenance schedule religiously. The restaurants who all have …
The Trouble Spot here is not necessarily in the study itself. Plenty of excellent, well done studies have been promoted with sensational headlines. The trouble spot here is when you let yourself fall for the sensationalism. Remember that in this day and age, web sites and news outlets are competing like crazy for the attention …