Friday 27 November 2015

Probability/ Randomness doesn't *really* exist

Physicist David Deutsch explains why probability is an abstract model that may be useful for some practical purposes - but is not found in real life, and does not explain anything.

The awful secret at the heart of probability theory is that physical events either happen or they don’t: there’s no such thing in nature as probably happening. Probability statements aren’t factual assertions at all.

The theory of probability as a whole is irretrievably “normative”: it says what ought to happen in certain circumstances and then presents us with a set of instructions. It is normative because it commands that very high probabilities, such as “the probability of x is near 1″, should be treated almost as if they were “x will happen”. But such a normative rule has no place in a scientific theory, especially not in physics. “There was a 99 per cent chance of sunny weather yesterday” does not mean “It was sunny”.

… Probability and associated ideas such as randomness didn’t originally have any deep scientific purpose. They were invented in the 16th and 17th centuries by people who wanted to win money at games of chance. To discover the best strategies for playing such games, they modelled them mathematically. True games of chance are driven by chancy physical processes such as throwing dice or shuffling cards. These have to be unpredictable (having no known pattern) yet equitable (not favouring any player over another).

…Before game theory, mathematics could not yet accommodate an unpredictable, equitable sequence of numbers, so game theorists had to invent mathematical randomness and probability. They analysed games as if the chancy elements were generated by “randomisers”: abstract devices generating random sequences, with uniform probability.

[But...] no finite sequence can be truly random. To expect fairly tossed dice to be less likely to come up with a double after a long sequence of doubles is a falsehood known as the gambler’s fallacy. But if you know that a finite sequence is equitable – it has an equal number of 1s and 0s, say – then towards the end, knowing what came before does make it easier to predict what must come next.

A second objection is that because classical physics is deterministic, no classical mechanism can generate a truly random sequence. So why did game theory work?

…The key is that in all of these applications, randomness is a very large sledgehammer used to crack the egg of modelling fair dice, or Brownian jiggling with no particular pattern, or mutations with no intentional design. The conditions that are required to model these situations are awkward to express mathematically, whereas the condition of randomness is easy, given probability theory. It is unphysical and far too strong, but no matter.
[However…], you could conceive of Earth as being literally flat, as people once did, and that falsehood might never adversely affect you. But it would also be quite capable of destroying our entire species, because it is incompatible with developing technology to avert, say, asteroid strikes.

Similarly, conceiving of the world as being literally probabilistic may not prevent you from developing quantum technology. But because the world isn’t probabilistic, it could well prevent you from developing a successor to quantum theory…
It is easy to accept that probability is part of the world, just as it’s easy to imagine Earth as flat when in your garden. But this is no guide to what the world is really like, and what the laws of nature actually are.

From New Scientist September 2015

< A fuller version is at

This means that the old chestnut used to check understanding of probability theory - of asking whether having thrown twelve 'heads' with a coin, the next throw of the dice is therefore more likely to come-up heads - is therefore misleading in practice and only true axiomatically. The supposed answer is that even after twelve heads the next throw is equally likely to come-up tails is only correct in terms of a model that this is true-by-assumption.

In real life, a dice that came up heads twelve times in a row, should usually be assumed to be non-random - so (unless the dice is being controlled specifically in order to trick you!) it is wise to assume that the next throw is more likely to come up heads than tails...

In general terms, randomness is just (part of) a model - and whether that model is true-to-life is a question of science - not of mathematics.

Some models can reasonably be termed an 'explanation' of a phenomenon - but models containing probability statements cannot.

I think it is fair to say that many or most statisticians fail to understand this; and that it undercuts many of the assumptions governing modern 'evidence-based' policy.


Atoning Unifex said...

Much of modern science is like media then, where an impression is created based on false abstractions.

In modern media the underlying equivalent to theoretical randomness is theoretical biological sameness, as if humans were interchangeable robot drones. Therefore any judgment of differences must be screeched at to be 'racist', as if the underlying false model being challenged is sacrosanct.

On a spiritual level the fact that all unfolds as it does amid supposed randomness suggests a moderator of chance! I would suggest this is an excellent place to look for God!

Andrew said...

I think you just ruined the first act of "Rosenkrantz and Guildenstern Are Dead" for me. ;)

Nicholas Fulford said...

Models can really only predict absolutely when the outcome is known, but then the same is also true of thinking.

Predicting future outcomes consistently and reliably is only possible in trivial systems, (see Godel's Theorem of Incompleteness.) In the complex world, be it human behaviour, weather, et cetera, there are distinct limits about how well the future can be predicted. Weather is a classic example, as is predicting stock market movements. (Economics - the dismal science fails so frequently because of the simple assumptions made about human motives. By assuming a rational actor economics fails to consider the many other factors that shape human behaviour.)

Does randomness exist in nature? Quantum Mechanics illustrates that there are limits with respect to certain types of measurement, and such things as Quantum Foam in the universe suggest not merely a lack of an ability to measure but that the universe bubbles / percolates at the micro level. This does not support a clockwork universe, which was a deeply disturbing way of thinking when it first appeared. (It is still mind-bending when you start reflecting upon it.)

Philosophically and theologically - if randomness exists - the idea of omniscience goes flying out the window, along with omnipotence. In fact as Godel points out, in any non-trivial system the axioms upon which it is built become unable to be proven from within the system. (Bertrand Russell and Alfred North Whitehead attempted and were unable to create a complete formalism for mathematics in Principia Mathematica.)

Total coherence based on axioms is not possible for any non-trivial system. While this grants some degrees of freedom from a dead dogmatic formalism it also means that absolute trust in any axiom that a man may apply to model the universe is incomplete at the very least. I suppose this should bring with it some humility and some flexibility to recast a model or view by substituting, altering or exchanging axioms. I actually don't mind this as uncertainty is something to which I must adapt my thinking. To attempt to conform the universe to my view is projection and hubris, but to play with, shift and alter the axioms is a creative exercise that opens my mind to ways of thinking and experiencing existence in ways which were formerly not possible. And that is a profoundly creative and meaningful exercise. Rather than being bound in the nutshell of a fixed view to which my ego is fearfully attached, there is a liberation to play and to think.

To be creative is to me the most vital thing about existence. The universe is continuously unfolding in extraordinary ways that invite my mind to playfully engage in the dance. Why be bounded within a nutshell of static and fixed axioms when I can participate in the creative exercise.

In many ways Dr. John C. Lilly was more than a bit out there, but in this statement I think he was spot on the mark:

In the province of the mind what one believes to be true, either is true or becomes true within certain limits. These limits are to be found experimentally and experientially. When so found these limits turn out to be further beliefs to be transcended. In the province of the mind there are no limits.

How big is your nutshell? And would you rather transcend its current limits to be as a king of infinite space, or be imprisoned by bad dreams. (Thanks to William Shakespeare's Hamlet for the famous nutshell statement from "Hamlet")

David said...

"But because the world isn’t probabilistic, it could well prevent you from developing a successor to quantum theory… "

It us interesting to speculate what randomness may be remodelled as if we change the underlying metaphysical assumptions. I have often wondered if we assumed that love and it's negation were conceived as fundamental aspects or 'forces' of reality, as well as free will, and the existences of multiple independent intelligences including degrees of 'consciousness' one might (someone far more capable than I and crucially skilled in mathematics, a modern James Clark Maxwell if you will) be able to produce a coherent and insightfully paradigm - shifted model of the physical world but crucially integrated with placing consciousness as a central feature (permanently alienated from current 'serious' scientific models as far as I can see, at least in mainstream). For example, if the coin is 'alive' as you have often supposed within a different context, would this alter it's behaviour in relation to another conscious agent acting harmony with it through a non - physical communication? (Maybe Uri Geller was onto something haha)

It sounds like the stuff of a best - selling sci-fi novel but I have often been fascinated by the possibilities of telekinesis representing some kind of harmonisation of one's 'will' the 'will' of so called 'inanimate' objects. After all, love is harmonisation of the will of two or more human beings, to a different extent to a dog or cat, a tree, why not also a gem stone or a pencil?:

hope the link works, I am a bit of a knights - move thinker and enjoy speculating...In a similar way, with regard to biblical miracles it seems that if they are believed, this means a belief in randomness must be over-ruled by divine intervention part of the time, all of the time or God does play dice for the rest of time.

If I think of God as not omniscient or omnipresent and in some sense parochial or constrained in his influence by say, the free will of other conscious/Intelligent entities chosing to oppose him then this seems to make some sense of other supposed randomness e.g. The tyranny of inanimate objects - usually experienced by angry people out of harmony or love with their surroundings. It also helps to understand how if a person's body is in the domain of their own conscious will then the state of the mind, negative affect and our deeper relationships with our own physicality can deeply effect our tendency to generate illness (e.g. Link between cancer and stress), whereas a sage/magician can pass through the world seemingly with the 'co-operation' of the inanimate world, which to him/her are recognised as alive and potential allies/recipients of loving - relationships and not the dead mechanicalism and the results of another way of relating to the world, such as solipsist or nihilistic conceptualisations that result in a purified state of alienation from the world in which they no longer resonate with love for themselves or the very clothes they wear, the air they breathe or the things they 'own' rather than love, value or express gratitude towards their very being.

David said...

I don't think the last link worked:

Watch "Phenomenon Theory" on YouTube