The thing is, hard core frequentists don't really exist. You have Bayesians, and you have people who happily use Bayes theorem as appropriate.
Take this article as an example. Reading it, I got the sense that the author at some point in life discovered Bayesian statistics, and is now on a crusade to twist everything into a pro and anti-Bayesian stance.
His example of the base rate fallacy (breat cancer diagnosis) is probably in every "frequentist" text book out there. Frequentists are well aware of it, and have no aversion to using Bayes Theorem. You will not find a frequentist objecting to taking into account the base rate when applying statistics. The difference, as the GP mentioned, is that the base rate is fairly well known and not subject to much debate. Whereas this:
>Maybe we’re not so dogmatic as to rule out “The Thinker” hypothesis altogether, but a prior probability of 1 in 1,000, somewhere between the chance of being dealt a full house and four-of-a-kind in a poker hand, could be around the right order of magnitude.
Is a number he pulled out of his rear end, and his subsequent calculation is not meaningful to people who don't agree with the prior. Sure, anyone can manufacture a prior if they wanted. And part of me is merely wondering: If he wanted a prior of 1 in 1000, why not simply require a p value of 0.001 instead of the 0.03 the paper used? The problem with the paper is that the sample size is small (n=57), and small samples are a lot more likely to give extreme results. I'll be OK with a p=0.03 if n=10000, but not if n < 100.
> why not simply require a p value of 0.001 instead of the 0.03 the paper used?
That is the proper response to a low prior probability, in general, yes.
More to the point, in an ideal world the p value one picks as the significance criterion should somehow capture both the state of prior knowledge and the consequences of reaching the wrong conclusion. If it really doesn't matter what you conclude, p=0.5 (not a typo: 1/2) is fine. If the conclusion really matters for something important, p=0.03 is likely too high.
Most published research that does significance testing seems to have no particular discipline for picking their threshold p values other than cargo-culting, unfortunately.
> the sample size is small (n=57), and small samples are a lot more likely to give extreme results.
That's already captured in the p value, no? That is, the sample size is already part of the computation of the p value. If you come out with p=0.03, then that means that if the null hypothesis holds in 3% of cases you'd see your observed results, whatever size your sample is. I'd genuinely like to understand why you feel there is a qualitative difference between n=1e4, p=0.03 and n=1e2, p=0.03, because I feel like I'm missing something there.
(Now it's a lot easier to get p=0.001 with n=10000 if your effect is real than it is with n=57. So in that sense, having larger samples helps. Having a larger sample _might_ also help with the "I tried a bunch of experiments until I got one that tested significant" problem, if it's genuinely harder to do a larger-sample experiment. Of course people could also apply a Bonferroni correction, but most practitioners of statistical testing don't seem to realize it exists or might be needed...)
I don't know your age, but from my experience, at least in the 1980s and 1990s, there really were real warring camps of Bayesians and Frequentists. People working on the same scientific topic who were Frequentists wouldn't even cite the papers of their Bayesian colleagues. Frequentist textbooks like A.W.F Edward's "Likelihood" would spend pages disparaging Bayesian methods. But I agree that things are much calmer these days with most people being pragmatists that don't care about being "pure" but use a mixture of methods from both camps.
Take this article as an example. Reading it, I got the sense that the author at some point in life discovered Bayesian statistics, and is now on a crusade to twist everything into a pro and anti-Bayesian stance.
His example of the base rate fallacy (breat cancer diagnosis) is probably in every "frequentist" text book out there. Frequentists are well aware of it, and have no aversion to using Bayes Theorem. You will not find a frequentist objecting to taking into account the base rate when applying statistics. The difference, as the GP mentioned, is that the base rate is fairly well known and not subject to much debate. Whereas this:
>Maybe we’re not so dogmatic as to rule out “The Thinker” hypothesis altogether, but a prior probability of 1 in 1,000, somewhere between the chance of being dealt a full house and four-of-a-kind in a poker hand, could be around the right order of magnitude.
Is a number he pulled out of his rear end, and his subsequent calculation is not meaningful to people who don't agree with the prior. Sure, anyone can manufacture a prior if they wanted. And part of me is merely wondering: If he wanted a prior of 1 in 1000, why not simply require a p value of 0.001 instead of the 0.03 the paper used? The problem with the paper is that the sample size is small (n=57), and small samples are a lot more likely to give extreme results. I'll be OK with a p=0.03 if n=10000, but not if n < 100.