Probability is a weird concept. It's much more philosophical than it appears at first glance, but much less philosophical than something like the interpretations of quantum mechanics.
Ultimately, what is a "probability"? It's a number that can be used for making predictions about the future. Neither Bayesian methods nor frequentist methods are "ideal" in the sense of predicting (computing) the future — in fact, results from algorithmic information theory put the best bounds on what we can "hope to predict". Very loosely speaking, the best predictor is the shortest program that reproduces the data. But this is uncomputable in general, which means we use something like minimum message length (MML) or minimum description length (MDL) — which are also sometimes uncomputable, but a bit more manageable at least.
In some situations, Bayesian methods can be shown to be equivalent to MDL, in the sense that sizeof(model) + sizeof(parameters) + sizeof(residual data) is a log reformulation of Bayes theorem.
Ultimately, what is a "probability"? It's a number that can be used for making predictions about the future. Neither Bayesian methods nor frequentist methods are "ideal" in the sense of predicting (computing) the future — in fact, results from algorithmic information theory put the best bounds on what we can "hope to predict". Very loosely speaking, the best predictor is the shortest program that reproduces the data. But this is uncomputable in general, which means we use something like minimum message length (MML) or minimum description length (MDL) — which are also sometimes uncomputable, but a bit more manageable at least.
In some situations, Bayesian methods can be shown to be equivalent to MDL, in the sense that sizeof(model) + sizeof(parameters) + sizeof(residual data) is a log reformulation of Bayes theorem.
See this paper for more details: http://homepages.cwi.nl/~paulv/papers/mdlindbayeskolmcompl.p...