>"Imagine that you begin a study of 200 people, measuring some variable. You see promising results, so you decide to extend the study and add more people; you test another 100 people.
By adding 100 more people to your study, have you increased or decreased the likelihood that your results were due to statistical chance? Counter-intuitively, the answer is that you have increased the odds that the result was due to chance."
I'll assume you are talking about using a t-test to see if two groups are samples from the same population. The problem you point out has nothing to do with chance. It is that you started with a null hypothesis that your groups were independent samples from the same distribution, but by having the second 100 people sampled conditional on the results of the first 200, you have ensured this is not true.
Such research designs are just a way of making sure the null hypothesis is false. If you get rid of the incorrect attribution to "chance", I don't see what is counter-intuitive about it.
> but by having the second 100 people sampled conditional on the results of the first 200, you have ensured this is not true.
I still think this is counter intuitive, and you must have a lot of practice in the field to get an intuitive feel for such cases.
The next 100 people are sampled just as randomly as the first 200. So by adding 100 people, we are essentially re-doing the experiment with a larger sample.
So how could the original experiment be valid if it had had 300 samples to begin with, when the now-augmented experiment, which for all intend and purposes is the same experiment, isn't valid?
I am not defending the validity of the above argument, but I am defending that it sounds pretty damn obvious.
You have to think about the hypothesis you are testing. This would be that groups A and B (including both the first set of 200 and second set of 100 people) have been independently sampled from the same distribution. This hypothesis is used to calculate a prediction of the expected results.
Instead your data is forced to consist of a sample where mean(A)-mean(B)=delta, where delta>0, for the first 200 people. Knowing that, would you make the same prediction about the final result (after the data from all 300 people is in)? It would be the same as getting data from all 300 people at once and adding delta to the first 200 in Group A. Clearly you have specifically created a deviation from the original hypothesis by design.
Also, let me note I said nothing about validity. I don't consider testing a hypothesis different than the research hypothesis to be a valid scientific activity. Just because the null hypothesis is false does not mean your research hypothesis is accurate or useful, so it is pointless from a scientific perspective. To me, this is on the level of arguing whether the Holy Spirit proceeds from the Father, or the Father and the Son. The entire premise behind the discussion is flawed, but we can still discuss the proper application of reason/logic to the arguments flowing from this flawed premise.
It is counter intuitive because for the vast majority of people statistics and probability are misunderstood. Casinos make money, people play the lottery, people are nervous about flying but not motorway driving and people think smoking or eating too much cake won't harm them.
By adding 100 more people to your study, have you increased or decreased the likelihood that your results were due to statistical chance? Counter-intuitively, the answer is that you have increased the odds that the result was due to chance."
I'll assume you are talking about using a t-test to see if two groups are samples from the same population. The problem you point out has nothing to do with chance. It is that you started with a null hypothesis that your groups were independent samples from the same distribution, but by having the second 100 people sampled conditional on the results of the first 200, you have ensured this is not true.
Such research designs are just a way of making sure the null hypothesis is false. If you get rid of the incorrect attribution to "chance", I don't see what is counter-intuitive about it.