Seeing is Believing

You are walking across a vast grass field on a misty morning when you suddenly spot a lake in the distance. As you get closer, you notice a lone duck swimming peacefully across it. But this image doesn’t last. After walking another ten yards, you realise your mistake—it isn’t a duck but a plastic bottle. Your perfectly smooth conscious image changes rather abruptly, flipping seamlessly from duck to bottle.

Conscious visual perception—what we are conscious of seeing—is typically explained by light bouncing off the various items around us and hitting our eye’s retina. The signal then travels to the back of our brain for visual processing and is fed forward through more complex areas until the brain can generate a perfectly coherent conscious image. A similar process takes place for other types of perception. But this raises a puzzle: if true, why is our conscious perception seemingly so perfect, lacking gaps even while we struggle to discern something?

This is one of many questions that has led some researchers to suspect that our conscious perception isn’t actually down to our brains processing what we experience with our senses. Instead, they suggest, we experience with our senses what our brain believes there to be; our brain helpfully fills in any gaps in perception. It is incredibly hard to prove this idea that what we see when we perceive the world is ultimately an internal projection. But Doris Tsao and Janis Hesse, neuroscientists at Caltech, in Pasadena, California, are tackling the problem with a series of experiments, aided by a $100,000 grant from FQXi.

Tsao’s interest in consciousness started at college, when she read Julian Jaynes’ 1976 book, The Origin of Consciousness in the Breakdown of the Bicameral Mind. "It was the first time I became curious about anything," she explains. "The cause of consciousness just seemed like the only truly interesting question in science."

Hesse has a similar attitude: "Consciousness is the most familiar thing to you—being the first thing that is with you when you wake up and the last thing to leave when you fall asleep—but at the same time its relation to the physical world and the brain is the biggest mystery in science," he says. "It is almost magical in that sense."

Obama or Taco?

The duo is using a particular experimental technique to probe what happens in the brain when we suddenly become conscious of something. This technique, which is called ’binocular rivarly,’ involves placing two distinctly different images—such as a picture of Barack Obama and a picture of a taco—in front of each eye respectively. You can try this now and see that what you become conscious of isn’t a mix of the two images. Rather, we perceive only one image at a time—either the taco or Obama—while being unconscious of the other. It is similar to the famous face-vase illusion, in which one can either see a vase or two faces—but not both at the same time (image above, right). What changes when we flip from one image to the other is therefore our conscious perception—our visual input remains exactly the same.

To investigate what happens in the brain during such changes in consciousness, the team is working with macaque monkeys, which have brains similar to those of humans. Specifically, they are zoning in on a network in the brain, previously discovered by Tsao, called the face-patch system. This consists of six distinct areas, most of which are located in two finger-shaped regions known as "the inferior temporal cortex" (IT cortex) towards the back of the brain. By inserting tiny, sensitive electrodes into these areas of the macaque brain, they can record the activity of thousands of individual neurons.

In 2017, this technique helped Tsao and her colleague Steven Le Chang to crack the exact code that these cells use to decipher a face (Chang, L. & Tsao, D. Y. Cell 169(6):1013-1028.e14 (2017)). While showing the monkeys images of faces, they discovered that cells in different patches code for specific things, such as some aspect of dimension or appearance. A specific cell can also code for several different things, such as distance between the eyes and length of the nose, with a unique firing pattern in each case. Amazingly, the team could even reconstruct an image based purely on a neuronal pattern.

It is because neuroscientists now know the face patch system so well that Tsao and Hesse are choosing to do their current research on faces. The face patches form a hierarchy in which lower level patches code for faces at specific views—more camera-like representation—while higher level patches code for identity of faces, invariant to view. "It’s a mini hierarchy of the brain," explains Tsao, adding that there are other patches coding for other objects in a similar way.

The approach has already yielded some results. Neuroscientists know that the cells in these patches fire differently depending on whether a monkey is looking at a picture of Obama or a picture of a taco. What Tsao and Hesse were interested in, however, was what would happen during binocular rivalry. Would the team see an instant swap from the highly active Obama brain pattern to the more silent taco pattern? If so, that would be a direct neural correlate of that conscious perception.

But nothing is ever simple when it comes to the brain. Rather than seeing a switch between two distinct patterns, they found that the cells encoded for both images simultaneously (preprint: bioRxiv 2020.04.22.047522 (2020)). "Previously people in the field believed that these face cells in the IT cortex basically represent your consciousness—that what they encode is exactly what you are conscious of," explains Hesse. But, assuming that the monkeys, like humans, are conscious of only one image at a time, "this data leaves open the question of how consciousness really is represented in the brain," Hesse says.

Mechanics of Consciousness

The duo is now digging deeper. Rather than just looking at how consciousness is represented in the brain, they want to discover its underlying mechanics. In their next experiment, they therefore plan to record cells from different areas in the hierarchy. While there are bottom-up pathways from simpler to more complex areas, there are also "feedback" connections going the other way. "We don’t really know what the feedback is for," says Hesse. "But there are as many backwards connections as forwards connections in the brain."

Their hypothesis is that conscious perception arises entirely through top-down signals rather than the other way around. "The brain has this high level model of the world which will feed back to predict and recreate the input in lower order sensory regions—it’s kind of like a computer graphics engine," explains Hesse. "So what we are perceiving is not the brain processing input but what the brain believes there to be, shaped by our prior experience and beliefs."

Tsao thinks this would explain why vision is so strangely coherent. "We think all of vision is due to this, whether you are perceiving a visual stimulus or visualising in the mind’s eye." In fact, while the top-down process that generates our conscious awareness may often be triggered by bottom-up signals from our senses—updating the brain’s model of reality by telling it there is a mismatch between the model and the visual system—this doesn’t seem to be necessary for conscious perception. Experiments dating back decades have already shown that you can create vivid hallucinations (both visual and other), a type of conscious experience, by electrically stimulating a person’s temporal lobe, which contains the IT cortex. This suggests the bottom-up signals from our senses aren’t necessary for conscious awareness, but may just help to continually tweak our conscious perception.

Excitingly, a recent independent study in mice, by neuroscientists in Germany, seems to support this hypothesis. In the brain’s most basic cells, called pyramidal cells, specific ’dendrites’—extensions that receive communication from other cells—get forward input while others get feedback. The study showed that when mice become unconscious due to anaesthesia, it is the dendrites that receive the feedback that are affected, suggesting there is a deep connection between top-down processing and consciousness at the cellular level.

Hesse and Tsao don’t just want to show that feedback is at work when we become conscious of something new. They also want to track the exact chain: where does the feedback start and where does it end? Does conscious perception require any involvement from the prefrontal cortex, which is normally involved in cognitively demanding tasks, or is it somehow generated in the IT cortex or elsewhere, from which it feeds back to simpler visual processing areas?

Matthias Michel, a philosopher at the Center for Mind, Brain, and Consciousness at New York University, New York, says it is a "good approach," but warns that you have to work very hard to make sure that the difference in signal you get during binocular rivalry is purely down to alternation in consciousness, rather than other confounding factors, such as thinking about an item. "Tsao is really an expert on face processing, so if anyone can do it correctly it is her," he adds.

Christof Koch, a neuroscientist at the Allen Institute for Brain Science, in Seattle, Washington, is also impressed. "It is a beautiful experiment," he says. Koch is hoping the approach can one day give some deep insights into how to best approach consciousness in a scientific way. "It speaks to the debate about whether consciousness is created within, in the back of the brain, or whether it requires critical input from the front of the brain."