On the view of at least some Bayesians, "frequentist" is just the special case of "Bayesian" that you get when you are computing credences based on a large number of identically prepared, independent trials over a known sample space. So on this view (with which I tend to agree), the two are certainly not incompatible.
It is of course possible to do both frequentist and Bayesian statistics badly. I would say bad frequentism comes when one fails to realize that standard frequentist methods tell you the probability of the data given a hypothesis, when what you really need to know is the probability of the hypothesis given the data. Bayesianism at least starts right out with the latter approach, so it avoids the former (unforfunately all too common) error.
Bad Bayesianism, OTOH, I would say comes when one fails to realize that Bayes' rule is not a drop-in replacement for your brain. You still need to exercise judgment and common sense, and you still need to make an honest evaluation of the information you have. You can't just blindly plug numbers into Bayes' rule and expect to get useful answers.
Long ago when I was a physics grad student, I distinctly remember that when someone introduced Bayesian statistics in a talk, it was because they were trying to justify weeding outliers from their data by hand. And they always got called to task on it.
I think that it’s right to call out scientists who think that math is reality.
We made math up, end of story. The “Unreasonable Effectiveness of Mathematics“ is obvious selection bias.
Math, statistics, is a tool. I don’t expect my shovel to be a dowsing rod, and I don’t expect my bayesian methods to predict the future. But, shovels do dig wells and probably does, on average, work out; neither is worthy of being disregarded. But, there have literally been folks since Pythagoras‘s time who believe that logic, and math, are The Truth. Like, God: The Truth. Like, it works because it’s the way that nature is, and we understand and control it and itis math… a “Natural Law”.
A better scientific mind does not fall for such folly. The “outliers” are the very phenomena that science wants to study. If we can explain the outliers through an error in method, fine. But, if the outliers are not able to be explained then we would never want to gloss over them because they don’t fit our expectations of a mathematical model of reality.
There is less of a conflict than many would have you believe. In many situations, both approaches yield the same answer. There are some edge cases. For example, in A/B testing, is early peeking bad? From a frequentist perspective the answer is "yes, either use a sequential method, or don't early peek at all". From a Bayesian perspective the answer is "early peeking is fine".
It boils down to what properties you want your analysis to have. Cox and Hinkley's "Theoretical Statistics" has a great discussion (section 2.4). Basically, you might want your analysis to have a certain kind of internal consistency. But you might also want your analysis to be replicable either by yourself or by another researcher. Those both seem like pretty important things! But there are edge cases (like the early peeking example) where you can't have it both ways. So you have to pick which one you want, and use the corresponding methods.
The likelihood principle actually supports the Bayesian perspective on these issues of experiment design, and is regarded as foundational by many frequentists.
Agreed. But as Cox and Hinkley discuss, the likelihood principle is sometimes at odds with the repeated sampling principle, so in any particular application, you need to identify if there is a conflict, and if so, which principle is more important. In my domain (simple A/B tests), you can claw the repeated sampling principle from my cold, dead hands.
There's no conflict. Nowadays whoever takes some courses in statistics will be exposed to Bayesian statistics. The universal reaction when exposed to Bayesian statistics is to fall in love with it, to think it's the answer to all of the world's problems (with the possible exception of war and hunger). But most people get to the point where they realize that Bayesian statistics is just a set of tools. Frequentist statistics can often times get you essentially the same results, with less effort.
And by the way, what do you call MLE (maximum likelihood estimation)? Is it frequentist? Because it looks awfully close to Bayesian.
Nothing prevents you from "going Bayesian" in figuring out how compatible a hypothesis is with reality. Bayesians have no issues with probabilities representing frequencies even though frequentists cannot understand probabilities representing uncertainty.
My point is, in Many-Words the parallel universes aren’t just a thought experiment, they are the actual real future of your current self. The computed probabilities do not describe potentialities then, but actualities.
Probabilities can represent uncertainty about actual things - things that are hardly going to be different even if you strongly imagine your current self branching out into a myriad of parallel universes.
To me, they seemed like complementary tools, each with their own strengths and weaknesses.
You can use one or the other depending on what your goal is. Do you want to figure out how much you should believe something?
Or are you trying to figure out how compatible a hypothesis is with reality? In the first case, go bayesian, in the second, frequentist.