This follows on from my post last week, "To Be or Not To Be (a Local Realist)," which discusses Joy Christian's "Disproof of Bell's Theorem." Joy's response to my post is here.
As I was stating at the end of part one, I am grateful to Joy Christian for give such a detailed answer in "Quantum Music from a Classical Sphere." I was puzzled by his arguments and I was preparing several short replies to various points, when I realized this was getting out of hand and it was better to have a single coherent reply in a new post.
Let me start by saying that the topics we are presenting here are hard and there is no absolute agreement even between experts on the very definitions of what one means by locality, realism or contextuality, or a no-go theorem for hidden variable theories.聽 Some people consider quantum mechanics as local and non-realistic, while others insist on non-locality. Contextual hidden variable theories encompass a wide and ill-defined domain that is probably best defined by their opposition to the much sharper-defined non-contextual hidden variable theories. Some people think that non-contextual hidden variable theories were rejected by the theorems of Gleason and Kochen- Specker and Bell's theorem is important only to reject some additional contextual hidden variable theories. Other people point out that the Kochen- Specker theorem cannot be put to any experimental test, and the importance of Bell's theorem is to reject non-contextual hidden variable theories. Finally, some people think contextual hidden variable theories are unphysical, while others think they are the future of physics.
To cut through this confusing mess, the best approach is to rely on accurate historical narratives. This would provide a stable setting for the meanings of the terms used, which would allow a common discussion ground where we can compare apples to apples. Also it is best to seek experimental results to settle disputes instead of debates, however interesting. Last, speculation about future results is best not to be discussed at all. If those tenets are followed, charges of "alarming confusions" and a fruitless dialogue can be avoided.
With this preparation, let me attempt to explain what contextuality for hidden variable is, why Joy's result is contextual, and what it all means.
First some preliminaries. Earlier I discussed the EPR paper and then I jumped to Bell's contribution. However, historically Bohr also made an early critical observation in his attempt to refute EPR's argument. Bohr was using the good old fashioned complementarity principle: in some experiments the particles behaves like waves, while in others like point particles. However, one cannot see them both in the same experiment. Therefore this invalidates EPR's argument where on one arm of the experiment one behavior is measured, while in the other arm the complementary behavior is measured. To justify this experimentally, Bohr noted that observations depend not only on the state of the system but also on the disposition of the apparatus. And indeed, no physicist is ever going to deny this. Joy mentioned general relativity, and I can also point out for example electrostatics, and any other theories requiring boundary value problems. Joy considers this "contextuality," but I would argue that this is clearly an overreaching usage of the term, which in hidden variable theories has a very different meaning which I will define later in its historical context. For the sake of the clarity, let's call this "Bohr's contextuality".
As a supporting argument for his position, Joy mentioned Bell's classical paper "On the problem of hidden variables in quantum mechanics." However, Bell's paper has a slightly different intention than Joy presented. Bell did not have an abstract for this paper and please allow me to explain how I understand this famous paper.
Einstein once said: "God does not play dice" and Bohr replied: "Do not tell God what to do."聽 In a similar vein, I would say Bell's main idea of the paper is: "Do not tell me how to construct my hidden variable theories." Requirements which are obeyed by quantum mechanics are not necessarily obeyed by hidden variable theories, which should only reconstruct quantum mechanics' predictions.
For the interest of time and space let me only discuss Bell's criticism of von Neumann and Gleason in this paper. Von Neumann had enormous influence in quantum mechanics due to his seminal work on axiomatizing it. He produced a "no-go" theorem on hidden variable theories which Bell found unjustified in its assumptions. In particular, von Neumann required that the linear combination of expectation values is the expectation value of the combination. While true in quantum mechanics, this is too strong to be demanded for any particular value of the hidden variable, as an elementary example can show (from Reflections on Relativity by Kevin Brown):
Suppose we have two variables X and Y and two hidden variables 1 and 2. Suppose for hidden variable 1: X=2, Y=5 and X+Y=5 while for hidden variable 2: X=4, Y=3 and X+Y=9. Averaging for hidden variables 1 and 2, avg(x) = 3, avg(Y) = 4, and avg(X+Y)=7 and the sum of the averages is the average of the sums. On each hidden variable however, this property is not valid. (If you are worried about the funny math X+Y = 5 when X = 2 and Y = 5, "X", "Y", and "X+Y" are three separate experiments requiring separate experimental setups and the number appearing to the right hand side of the equal sign is the experimental outcome. The "+" in "X+Y" is only part of the label for an experimental setup and not a genuine plus operator.)
Gleason's theorem improves on von Neumann's result because it makes it a mathematical necessity to have a particular form of the average (the trace formalism) which does obey von Neumann's condition for compatible (commuting) observables. Bell's criticism on Gleason's result is much more subtle and only here "Bohr's contextuality" becomes a mandatory part of the argument. One can argue that von Neumann's condition from above can be dismissed even without the toy example (which is not part of the paper, but I use it for the clarity of the argument), by pointing out that it is nonsensical for non-commutative variables which requires a different experimental setup. However, for compatible commutative variables it is a natural requirement (and this is what is used by Gleason).聽 Here Bell's analysis becomes subtle.聽 To rule out hidden variables Gleason uses this requirement for commuting observables on spaces of dimensionality higher than two, meaning it has to be applied at least twice: on variables X and Y as well as X and Z, and while X and Y are compatible (commute), and X and Z are compatible as well, X and Z may not be (and the funny math from above can happen). It is only at this point where Bell says: "the result of an observation may reasonably depend not only on the state of the system (including hidden variables) but also on the disposition of the apparatus."
So, Bell's main argument was a defense of the freedom to construct a hidden variable theory. Unjustified requirements were also proven unjustified by showing they violate Bohr's contextuality (but this is only a secondary supporting idea in the paper).
But why bother splitting hairs over the meaning of contextuality? Because it critically appears in many proposals of hidden variable theories aimed at recovering the predictions of quantum mechanics. Everyone agrees that quantum mechanics accurately describes nature, but not everyone is satisfied with quantum mechanics' interpretation. Why is this so? In the traditional quantum world, there are no safe harbors for quantum objects (pure states are not immune to "collapse") and you get epistemology (a statistical description) without ontology (objective reality) which generates a sense that something is fundamentally missing in this scheme. Hidden variables are supposed to fill this missing ontological gap and restore the "ignorance interpretation" for the statistical nature of quantum mechanical predictions. First and foremost, in my opinion, hidden variables have to have a rock solid ontological value (ontological definiteness - typically known as non-contextuality). (Ontological definiteness is a stronger property than mere objective definiteness, which can be claimed for example by some parts of Bohm's theory. Still, all hidden variable theories corresponding to Hilbert spaces of a dimension higher than two suffer in one form or another of ontological indefiniteness. In the case of Bohm's theory--a contextual algebraic hidden variable theory--momentum is ill defined but still has objective definiteness. However spin is treated just like in standard quantum mechanics with ontological indefiniteness. I would challenge anyone to name a single hidden variable theory with dimensions higher than two which does not have ANY ontological problems.)
For measurement results, a dependency on the experimental context is completely acceptable. However, if hidden variables or their meaning changes with the experimental context, we are back to square one in terms of the strangeness of quantum mechanics. Not only do hidden variable theories suffer from strangeness, but all interpretations of quantum mechanics aimed at making it palatable to our classical intuition employ a bag of tricks: transactional interpretation is wonderful if you can swallow messages from the future; Bohm's theory restores classicality at the expense of action at a distance; Everrett's approach transforms "OR" into "AND" and deprives even classical physics of counterfactual definiteness, etc, etc.
So what do we call this rock solid (ontological) foundation? (Ontological) non-contextuality. If you are unhappy with standard quantum mechanics and its wave/particle complementarity, then you should not be happy with any other ontological flip-flopping explanation regardless of the packaging and other genuinely intuitive but ultimately non-essential features tied to the particular mathematical formalism used (lack of superposition, lack of entanglement, etc). The big pink elephant of epistemology without ontology remains in the middle of the room.
Ontological non-contextuality represents the golden standard for any physical theory aimed at restoring classical intuition. (As a side remark, general relativity and all other classical physical theories obey ontological non-contextuality.) Local realism obeys ontological non-contextuality too. We can now drop the ontological qualifier and simply state either contextual or non-contextual in the remainder of the text. This is the standard meaning of contextuality or non-contextuality when discussing hidden variables (and in this meaning general relativity is non-contextual). All non-contextual hidden variable theories are ruled out by many no-go theorems, and the majority of physicists consider contextual hidden variables unphysical with no experimental evidence ever backing their existence. There are also many flavors of contextual hidden variable theories and we already encountered the algebraic type in Bohm's theory.
Now we are in a position to put everything together.
What does Bell's theorem assert (regardless of any fine mathematical print)? Any locally realistic theory obeys certain inequalities which are violated by quantum mechanics and ultimately by nature in experiments. What is Joy's challenge to Bell's theorem? Non-commutative beables allows him to go over Bell's inequality precisely like quantum mechanics. Is Joy's theory noncontextual as he claims? No. Non-commutative beables are ruled out by Clifton's theorem for non-contextual theories, meaning Joy's theory cannot be non-contextual (and I clarified some more what kind of contextual theory it is in my preprint: http://arxiv.org/abs/1107.1007).
But let's assume that both Clifton and I are mistaken. There is a simple way to put this to the test: if Joy's claim of non-contextuality is correct, then one can write a computer program modeling his theory and obtain Tsirelson's bounds. (This is not my idea; I got it from researching the web for reactions to Joy's results.) I claim this cannot be done. If anyone manages to actually write such a program which goes above Bell's limit then I admit I am completely wrong and Joy is right in his non-contextuality claim. And if he is right on his theory's non-contextuality claim, then he did manage to disprove Bell's theorem as usually understood by the physics community.
In the meantime, let me explain why I think Joy's theory cannot be modeled on a computer and why it is not a strictly locally realistic theory.
Recently Joy published http://arxiv.org/abs/1106.0748. It is relevant to note that the state space (encoding the complete information) for spin 1/2 is SU(2) which is isomorphic with SO(3) where geometric algebra can naturally be used. There is an additional degree of freedom coming from the double cover property of the isomorphism and this allows for the possibility of a hidden variable which encodes the information on which one-to-two map you are located. It is not hard to see that Joy's formalism when done consistently in the geometric algebra formalism is an equivalent representation of the standard quantum mechanics and his agreement with quantum mechanics comes as no surprise. However, Joy's formalism lacks a "Born rule" which translates his formalism into the actual experimental outcomes (real numbers). If in http://arxiv.org/abs/1106.0748 one applies at any intermediate steps a map from geometric algebra objects to real numbers, it is easy to see that one only recovers Bell's limit. (In standard complex quantum mechanics formalism this would correspond to applying Born's rule too early and adding probabilities instead of amplitudes resulting in incorrect classical behavior predictions). But not applying any translation mechanism until the end in Joy's formulation means that this step is delayed until the interaction taking place during the measurement process (which has to somehow implement/model the geometric algebra operations).Tung Ten Yong (http://arxiv.org/abs/0712.1637) has criticized Joy's model because the final results are expressed as geometric algebra objects. I do not think this is a decisive criticism of Joy's theory as Yong claims because if nature is quantum mechanical at heart, and if Joy's theory reproduces all quantum mechanics results, this means that geometric algebra should be the proper ontology of nature. And then it is very natural to expect interaction during measurement to realize geometric algebra operations.
But geometric algebra objects are non-local and while Joy can call his model local due to factorizability, it is non-local due to the critical usage of a geometric algebra product (during measurement). Locality does not mean only separability between Alice and Bob, but must include a notion of physical distance as well. This additional non-locality would prevent any faithful simulation on a computer program because one cannot program a way to compute the experimental outcome for Alice at one end without accessing all information encoded in SO(3), meaning the inclusion of the information at Bob's end as well (which is unavailable due to physical separation).
Again, if you disagree with the analysis above, Clifton's theorems, or my preprint http://arxiv.org/abs/1107.1007, write a computer program modeling Joy's theory for EPR-Bohm and show you get Tsirelson's bound. If you succeed, I am wrong, this post is essentially incorrect, Joy's theory is non-contextual, and Joy did disprove Bell's theorem by counterexample.
Joy's theory can be considered both local due to factorizability, and non-local due to geometric algebra. It can also be considered both realistic as it employs classical three dimensional geometrical objects, and non-realistic as it is (ontologically) contextual (the local beables critically depend on the experimental setting).
Finally, the same conclusions from my prior post and preprint stand: Joy does not disprove Bell's theorem mathematically because he starts with a different assumption, that of a non-commutative beable algebra. I really did not understand Joy's remark: "In particular, contrary to what Florin asserts, I do not start with mathematical assumptions different from those of Bell," Quoting Joy (http://arxiv.org/abs/quant-ph/0703244v12): "In fact, apart from the multiplicative non-commutativity, they satisfy exactly the same algebra as do the local beables of Bell. [...] Bell's algebra, by contrast, is a commutative and associative normed division algebra over the field of real numbers." Also: "Finally, Grangier expresses his unhappiness with the choice of my running title: "Disproof of Bell's Theorem." He rightly points out that as a mathematical theorem Bell's theorem cannot be disproved, since its conclusions follow from well defined premises in a mathematically impeccable manner."
Contrary to his claims and paper titles, Joy does not disprove the importance of Bell's theorem as the key result validated by experiments rejecting local realism. If Clifton's theorem is right or if my spin one analysis is correct, Joy's theory is a contextual theory not classifiable as locally realistic. This is a scientific falsifiable statement, as any computer simulation of Joy's theory exceeding Bell's limit would render both Clifton's theorems and my preprint wrong.
Last, Joy's theory does expose a weakness in Bell's theorem. Clifton's theorems fill this weakness and Bell and Clifton's theorem combined do rule out all hidden variable theories obeying local realism because Bell requires commutativity of beable algebra and Clifton shows that beable algebra must be commutative for all non-contextual theories and for any relativistic quantum field theories with bounded energies. As a counterexample, Joy's theory does limit Bell's importance in Shimony's interpretation for some contextual hidden variable theories. However, other physicists including myself disagree with Shimony and place Bell's theorem's importance solely for rejecting non-contextual hidden variable theories in an experimentally verifiable fashion (in my preprint's abstract I presented both points of view).
Scientific conclusions aside, I did enjoy learning about Joy's result, and I do think they represent very valuable contributions in our understanding of quantum mechanics and hidden variable theories. More interesting work remains to be done and I would be very interested to see any proofs for his "Theorema Egregium" or (Eq. 10 in my preprint). I wish Joy success in further developing his ideas.