Many people have trouble understanding outcomes. Obviously, the outcome is whatever a study is measuring — the results. But what’s the difference between primary and secondary outcomes? Why is it important to specify them before the study is performed, rather than after? Why is it more rigorous to have one or two outcomes instead of fifty?

The simple answer: the more times you spin the wheel, the more likely you’re going to eventually hit the jackpot… but the less impressed we’ll be when it does.

 

Accurate rifles don’t need ten shots to hit the target

Suppose I told you that I’ve built a miraculous device — the Dice-Guesser 1000. I wave it over a six-sided die, it beeps and whirrs, and after a few moments, it flashes a number: 4. Now you roll the die, and it comes up with a 4.

Pretty impressive, right? Cool trick. It’s cool because the likelihood of guessing that roll by pure chance is only 1 in 6, which isn’t very high… so you think maybe the Dice-Guesser 1000 actually works. Normally I’d assume it was guessing randomly, but the odds were low that it would’ve guessed correctly the first time without any help, so maybe it really can predict dice rolls. If I did this trick many times, you’d be even more convinced, because it’s less and less likely that I’d keep guessing right if my device didn’t have some magic. Buy stock in my company.

Now, let’s do things a little differently. I have another device, the Dice-Guesser 2000. The only difference is this: instead of flashing one number, it flashes six numbers. 1, 2, 3, 4, 5, and 6. You roll a 4. Behold! One of my guesses was correct!

You’re not impressed this time? Why not? Isn’t the chance of correctly guessing the roll still just 1/6?

Sure it is. But I guessed six times, and that meant the chance that any one of my guesses would be correct was much higher (100%, in fact). The original feat was only impressive because I was picking one outcome out of many possibilities, and it was therefore improbable that I’d be right unless my device had some real magic. But if I pick more than one guess, it’s increasingly likely that random chance does explain my right answer — and if I guess enough times, it approaches statistical certainty that I’ll get it right. Doesn’t mean my device has any predictive power. Just means I tried until I got lucky.

So if you want to know something about my device, you’d be smart to limit me to only one guess. Maybe a couple. But not six.

 

My outcomes overfloweth

Now let’s apply this to a clinical study.

We’ll make it the strongest kind of study, a randomized controlled trial. Let’s say it’s a trial of Strokesallbetter, a pill that we think might help people after an ischemic stroke.

Before we run the trial, we specify a primary outcome: mortality. We’re guessing that patients who take Strokesallbetter will be less likely to die than patients who just take a placebo. We run the trial, and behold — patients who took Strokesallbetter were less likely to die! Are you impressed? You should be. Probably the drug works.

Now let’s take a different drug, Strokesjustfine. And let’s change things up. We’ll designate an outcome of mortality. But we’ll also designate another outcome: neurological deficits after 30 days. And we’ll designate another outcome too: duration of hospital stay. In fact, let’s designate one hundred different outcome measures, all unrelated. More data’s always a good thing, right?

We run the trial, and unfortunately, Strokesjustfine produces no difference in mortality. Or neurological deficits. Or hospital stays. In fact, it showed no difference for almost anything we measure, but one outcome was different: patients who took Strokesjustfine had a slightly lower chance of receiving a parking ticket in the 41 days following their stroke.

Are you impressed? Should we market Strokesallfine as a miracle parking-ticket cure?

Probably not. Why? Because this is just like the case where we kept guessing at the dice roll. It’s probably not random luck if you hit the target with the only shot you take. But if you shoot fifty times and one of them makes it, it probably is random luck. Nice try, no cigar.

Let’s put it a little more simply. In most studies, the statistical threshold for significance — how strong a correlation must be for us to assume it’s real, and not just chance — is 95% (a p value of .05). That translates to a 1 in 20 chance of the result happening randomly. Pretty good numbers… if you only guess once. What if you guess 20 times — in other words, you have 20 different outcomes? Now if one outcome shows a statistically significant correlation, that’s not an impressive result, because it had a 1 in 20 chance of happening by random probability, and you tried 20 times!

That’s why legitimate studies usually specify only one primary outcome. One hypothesis, one finish line; that’s where you’re putting your money, what you really think and hope might happen. At the most, there might be two or three, particularly if one is a safety outcome (i.e. not whether the drug worked, but whether it hurt anyone, which is a different kettle of fish). Best practice is to limit it to one primary outcome and perhaps two or three secondary outcomes (like the Pips to Gladys Knight, they’re not as central, but they’re something), all of them defined in advance. But that’s all you get. If you start listing dozens and dozens of outcomes, it no longer means anything when one of them happens to show a signal — statistically, that’s expected to happen, and you have no basis to believe it’s anything except chance.

That doesn’t mean you can’t measure or calculate plenty of different datapoints, of course. But when one of them has an interesting result, you don’t get to pretend that was your goal all along. If your primary outcome was a failure, your trial was a failure, period. If you had some secondary outcome that was positive, and there’s a plausible reason it might be a real finding (i.e. not the parking ticket situation above), then you can run another trial using that as the primary outcome. If you find a significant result this time, then great — you win! But merely noticing that result within the mountain of secondary data isn’t the same as designating the outcome ahead of time, for reasons that are hopefully now clear.

 

Hanging out the data nets

The ultimate example of this phenomenon is the practice of data mining.

Also known as data dredging, this is a lot like the big ocean-going fishing vessels that hang out nets and trawl along until they’re full. They’re not looking for anything in particular, they’re just picking up whatever — tuna, dolphins, old boots — until they notice something they like.

The way it works in research is like this: find a big bank of data. It could be from a study you performed, or it could be somebody else’s; doesn’t matter. Now, put it into a computer, and analyze it in every way possible. By our analogy above, you roll the dice over and over and over until you finally find a statistically-significant result. It doesn’t have to be something that was designated in advance. It doesn’t even have to be particularly plausible or logical. It could be the number of patients who lost between 63 and 66 hairs during the trial period. If it’s statistically significant, then you publish it.

And when you publish it, you highlight its statistical significance. Only a 1/20 chance of this happening randomly! What you certainly don’t highlight, or even mention, is that you ran twenty different analyses until you found this result. So nobody knows how many times you rolled the dice. Presto! We’ve created an impressive-looking association out of thin air.

The grim reality is that many published trials, even the big, impressive RCTs, are performed in this same way. They may designate a primary outcome beforehand. But there’s a lot of time and money invested in the study, so if it reaches its conclusion and the primary outcome is negative, do they publish it as a failure? No way. They dig through their data until they find some way of parsing the results that reaches statistical significance. Then they redact their study design and pretend that was their primary outcome all along. It’s called the Texas sharpshooter fallacy: fire your rifle at a blank wall, then draw a bulls-eye around wherever the bullets land, and you’ll never miss.

It’s hard to detect this nonsense, because when a study is published, you mostly have to trust that the authors haven’t changed their study design after peeking at the data. The only practical defense is pre-trial registration (ClinicalTrials.gov is the biggest registry), where researchers publish the design of their trials before they run them, and they can’t modify it later without the registry recording the change. That way, if you read their published write-up and it says the primary outcome was X, you can check the registry to confirm that their primary outcome was always X. Researchers are caught red-handed all the time by this, presumably because they hoped nobody would look.

 

Publication bias: the invisible dice

We talked before about publication bias, the phenomenon where negative trials are less likely to be published than positive ones, often disappearing into a desk drawer instead of appearing in the peer-reviewed literature and hence reaching your eyeballs. It happens all the time, particularly in studies funded by the industry that has a vested interest in their outcomes.

Guess what? Publication bias is another great way to roll extra, unseen dice. A significant result may have a 1/20 chance of occurring randomly, and maybe the trial only had a single outcome. But what if there are 19 other trials of the same drug, and every one of them was negative?

If you knew that, you’d recognize this positive result was probably spurious, despite its internal validity. But due to publication bias, you might never know about all those negatives. You search the literature, you find only the lucky positives, and you never know if they’re surrounded by a backdrop of failures that you simply don’t see.

 

So what’s the answer?

The take-away message is this:

1. If it intends to show a significant association, a study should define a very small number of outcomes (preferably just one).
2. Those outcomes should be designated before the study is conducted, and must not change at any point.
3. Any findings beyond those pre-defined outcomes may generate hypotheses for future trials, but signify nothing on their own.
4. Pre-trial registration can help readers verify that the above process was followed.
5. Publication bias can hide how many attempts lie behind a positive result. Its presence is ubiquitous, and is not easily accounted for.