Figure 1. Picture which some people find difficult to decipher. In case of difficulty, turn it upside down.
J. Kevin O'Regan
Laboratoire de Psychologie Expérimentale
Centre National de Recherche Scientifique,
Université René Descartes,
92774 Boulogne Billancourt, France
oregan@ext.jussieu.fr
http://nivea.psycho.univ-paris5.fr
Alva Noë
Department of Philosophy
University of California, Santa Cruz
Santa Cruz, CA 95064
anoe@cats.ucsc.edu
http://www2.ucsc.edu/people/anoe/
Manuscript submitted to Synthese, Feb 2000.
Abstract
The paper proposes a way of bridging the gap between physical processes
in the brain and the "felt" aspect of sensory experience. The approach
is based on the idea that experience is not generated by brain processes
themselves, but rather is constituted by the way these brain processes
enable a particular form of "give-and-take" between the perceiver and the
environment. From this starting-point we are able to characterize the phenomenological
differences between the different sensory modalities in a more principled
way than has been done in the past. We are also able to approach the issues
of visual awareness and consciousness in a satisfactory way. Finally we
consider a number of testable empirical consequences, one of which is the
striking prediction of the phenomenon of "change blindness".
The experience of driving a Porsche would seem to consist not so much in the occurrence, as it were all at once, of a certain sensation (something like a twinge or a wave of dizziness), but rather in the characteristic pattern of integrated activity in which the actual driving of the car consists. The particular experience of Porsche-driving comes from the typically Porsche-like give-and-take between you and the car and the things going on around you when you drive the Porsche.
This analysis of the experience of driving a Porsche in terms of a temporally extended pattern of activity has two consequences. First, it suggests the possibility that to investigate the nature of experience we must direct our investigations not to some ineffable inner event, but rather to the temporally extended activity itself, to the laws that govern this activity. We shall see that on this approach, the problem of understanding experience may become more tractable.
Second, it comes to seem doubtful that the Porsche-driving experience is the sort of thing that can be generated by activity in the brain. It seems clear that there is no special circuit that itself underlies the Porsche-driving experience. There are of course brain mechanisms that participate in all the various Porsche-driving behaviors, but none that can be thought of as sufficient to produce the Porsche-driving experience.
In the present paper we take an analogous approach to vision. Seeing, we argue, like Porsche-driving, is a temporally extended pattern of activity. To see is to be skilled in this activity. Visual experience, like that of Porsche-driving, does not consist in the occurrence of "qualia" or such like. Rather, it is a kind of give-and-take between you and the environment.
Moreover, we claim, there are no states or processes in the brain that generate the experience of seeing. Brain processes participate in seeing, but none deserves to be thought of as "the locus of seeing in the brain". Seeing is something we do, not something that happens in our brains (even though, of course, a lot goes on in the brain when we see).
A first advantange of this activity-based approach is that it enables us to overcome the problem of what has been called the "explanatory gap" (c.f. Levine [1983]; also e.g. Chalmers [1996]) -- that is, the problem of understanding how something physical like the brain, can generate something non-physical, namely experience. We have solved the problem by noting that experience is not generated in the brain at all.
Second, the approach allows us to actually characterize the quality of visual phenomenology, and to explain why nervous influx coming into the visual cortex appears to give rise to qualitatively different experiences than those arising from nervous influx coming into the auditory, or any other sensory cortex. Just as driving is different from fishing or cooking, seeing is different from hearing because it involves doing different things. This explanation of the difference in phenomenology between the different senses has the advantage of not requiring invocation of any kind of "specific nerve energy" (Müller [1838]) or other essence or mechanism that somehow endows visual cortex or visual pathways with the ability to generate visual-like experiences.
A third advantage of taking this approach to seeing, is that it sheds a new light on a number of classic problems in visual science, sometimes causing them simply to evaporate. A problem which evaporates is the problem of why the world seems stable despite eye movements and the problem of why we do not notice the blind spot in our visual field. Another problem which becomes a non-problem is the so-called binding problem (that is, the problem of how information about different visual attributes like color, texture, depth, form, movement, etc., are combined into a single spatially and temporally unified percept). The theory also makes empirical predictions, among them the phenomenon of change blindness (in which large portions of a visual scene can be modified without this being noticed), and the phenomenon of sensory subsitution (in which blind persons can obtain a form of vision through tactile stimulation).
In the next sections we shall define in greater detail what seeing consists in. We shall distinguish one aspect of the activity of seeing which can be related to what psychologists call "sensation", and one aspect which can be related to what they call "perception". We will then define a specific sense of the notion of visual "awareness" and go on to consider the problem of accounting for what philosophers and psychologists call visual consciousness or experience.
Note that the sensorimotor contingency laws that the missile guidance system has mastered are laws which depend on the nature of three-dimensional space and how the missile's apparatus senses things in that space: the type of optics and the type of projection used, the way the image is formed in the camera, the way it is sampled, etc., will all play a role in determining the laws that link the missile's possible behaviors to the resulting sensory inputs. If the system were using radar or sonar instead of visual information to track airplanes, the laws would be quite different.
Having mastery of the sensorimotor contingency laws governing airplane tracking does not mean that the missile guidance system is currently tracking an airplane -- it might currently be cruising with no airplane in view, or it might not be flying at all. Only when the laws of airplane tracking are currently being put to use, that is, only if the missile guidance system is exercising its mastery of airplane tracking is it true to say that the system is visually tracking the airplane.
But note also that visually tracking the airplane does not actually require any input to be coming into the missile guidance system. For example, suppose that the airplane has momentarily slipped out of the missile's view. Then the system's airplane detector circuits will not currently be active. Yet so long as the missile guidance system is, for example, tuned to the fact that it can turn to bring the airplane back into the camera's sights, we still would say that the missile guidance system is currently visually tracking the airplane. To say that it is "tuned to the fact that it can turn, etc.", is to say something about what the tracking system could do if it were presented with particular types of input.
We wish to make the parallel between airplane tracking by the missile guidance system, and sensation in humans. We shall say that perceivers have sensations in a particular sense modality, when they exercise their mastery of the sensorimotor laws that govern the relation between possible actions and the resulting changes in incoming information in that sense modality. Note that this use of the word sensation is somewhat unconventional, because it treats sensations as something that might be construed as unconscious. After all, on this view, the missile guidance system has sensations of some sort. We shall take this very limited definition of sensation at this stage, and show later how it can be used within a more complete framework to deal with the specifically human, "experiential" aspects of sensation.
Under the classical view, the sensation of red occurs when a particular ratio of excitation occurs in the long, medium and short wavelength sensitive mechanisms in the visual system. There are additionally some color constancy mechanisms which compensate for the effect of ambient lighting conditions. The output of these processes then is read by some other cortical process which then presumably generates the sensation of red. But how does this come about? What mechanism in the brain could take neural excitation from some cortical process, and convert it to experience? This is an outstanding and hotly-debated topic in recent philosophy and cognitive science.
Under the view espoused here, the sensation of seeing a red patch of color does not arise from activation of some cortical mechanism (even if that activity is necessary for the experience). On the contrary, the sensation of red occurs when the brain is tuned to certain very particular things that will happen to the neural influx if we do certain things with respect to the red patch of color. For example, one thing we can do is close our eyes, at which point there should be a very dramatic change in the neural influx. This particular sensorimotor contingency is an indication that we are seeing the red patch of color, and not hearing or tasting it, since eye-closing has no effect in the auditory or tactile modalities. Another thing we can do is move our eyes off the patch and back onto it. As the red patch moves from fovea into periphery, retinal sampling becomes more sparse and color information changes its nature, with influx from monochromatically tuned rod photoreceptors taking over as compared to influx from the three different cone types present in central vision. Another difference between sampling the red patch with central and peripheral vision derives from the fact that the central retina is covered with a yellow macular pigment, which absorbs light in the short wavelengths.
As a result of such differences, lawful changes in the neural influx occur as a function of the eyes' position. The laws underlying these changes, that is, the sensorimotor contingencies, are indicative of the fact that the patch is being sampled by the visual apparatus, and not via, say, the olfactory or tactile modalities.
Another kind of sensorimotor contingency is related not to the effect of eye movements, but to the effect of moving the red patch itself, or moving ourselves with respect to it. Yellowish sunlight or reddish lamplight have different spectra. Depending on how we move or how we move the patch, the proportion of sunlight or lamplight reflected off the patch will change, and the spectrum of the light coming into our eyes will change in a way that is typical of red, and not of other colors. Broackes [1992], in a theory of color which could be considered a special case of our theory, says that "color is the way surfaces change the light". D'Zmura & Iverson [1994] have developed a mathematical approach to color constancy related to this idea.
Exercising our mastery of all these possible kinds of sensorimotor contingency constitutes the sensation of red. The sensation of red is the exercise of our mastery of the way red behaves as we do things. The sensation of red is a way of doing things, not something that emerges from neural excitation.
But note that though we identify the sensation with a pattern of skillful activity, one can have the sensation even when one is, at the moment, inactive. (Just as the the missile guidance system can be tracking the airplane even though no information about the airplane is currently being received.) We can exercise our mastery of the sensorimotor contingencies that signal the presence of redness without actually blinking, or moving our eyes or our bodies with respect to the red patch. This exercise consists in our practical understanding that if we were to move our eyes or bodies or blink, the resulting changes would be those that are typical of red, and not of green patches of light. Indeed, psychologists over the last century have extensively studied vision under tachistoscopic presentation conditions, where presentation durations are so short that observers cannot properly exercise their mastery of visual contingencies. Because visual stimuli are relatively redundant it is nevertheless possible to deduce from such a momentary stimulation what the consequences of one's actions would be if one were allowed to move. Under the view taken here, visual sensation that occurs in tachistoscopic experiments is constituted by this knowledge, and does not derive in a direct fashion from the neural influx that activates the visual pathways.
Note also that to have the sensation of red, there need actually be no excitation currently coming into the brain which by itself might correspond to red: this is the situation when we blink. Thus, unless we pay attention to the fact that we are blinking, the sensation of redness (and for that matter the sensation of seeing) does not go away during the blink. This is also the situation when we see a red object first in central vision and then in peripheral vision: for a given object, the information about color available in peripheral vision is dramatically reduced as compared to what is available in central vision -- and yet the perception of redness persists. This is because what provides the sensation of redness is not the neural influx, but the knowledge of how the influx would change if you were to move your eyes.
Note that the ability of this extremely simple mechanism to "sense light" does not consist just in the fact of activity in the sensor. Light causally affects the sensors, to be sure. But the vehicle's ability to sense the light consists in the ways in which it makes use of the raw stimulation of the light on the sensor. If there were no linkage between the activity of the sensor and the driving mechanism, then activation in the sensor would have no behavioral or ecological implications for the vehicle and it would be as good as blind. Having sensations, as we are using the term, consists in the use the system (whether a simple Braitenberg vehicle or a complex organism) makes of input. Any system capable of making use of input is a system that is attuned to patterns of sensorimotor contingencies, and so to one that can be regarded as having practical knowledge or mastery of these contingencies.
Note one final consequence of this account of sensation. The sensation
does not occur in the brain anymore than it occurs in the sensors of the
vehicle. The occurrence of sensation is a system or creature-wide phenomenon.
Mastery of sensorimotor contingencies may be neurally encoded, but this
mastery does not itself reside in the brain. To the extent that it makes
sense to speak of mastery residing anywhere, then it resides in the creature
as a whole, in the whole neurally enlivened body.
We have seen from the example of the sensation of red that one aspect of the sensorimotor contingencies is related to the structure of the visual apparatus: blinks cause a drastic change in the sensory input; moving the eye causes lawful changes in the color information available from a stimulus. Other laws concern the spatial sampling of the image: foveal resolution is much finer than peripheral resolution, so as the eye moves, the resolution of the sampled information changes in a lawful way which is typical of the visual modality. Another lawful relationship is determined by the optics of the eye. Owing to the spherical shape of the eyeball, there are certain distortions that occur in the image when it shifts -- for example the curvature of the arc formed on the inside of the ocular sphere by looking at a straight line changes as the eye moves on and off the line (this point has been discussed by Platt [1960], and in a more general fashion by Koenderink [1984]). These changes are also affected by the eyes' state of accomodation, and by spherical and chromatic aberrations in the eyes' optics.
While the sensorimotor contingencies we have just discussed are predominantly a consequence of the structure of the visual apparatus, another somewhat different class of sensorimotor contingency is related to the 3D structure of space. Because the visual apparatus is sampling a two-dimensional projection of three-dimensional space, certain laws apply. For example, when you move forwards or backwards, there is an expanding or contracting flow-field on the retina; the amount of light impinging on the eye is governed by the inverse square law. There are also more complicated laws which, as pointed out by Piaget [1937/1977] in his studies on how the notion of space is acquired, are related to the (mathematical) group structure of three-dimensional space (see also Poincaré [1905] and Husserl [1907/1991] for related ideas). Thus for example, if you move one step forward and one step sideways, there is a diagonal direction you can move back along such that the nervous influx comes to be the same as it was to begin with.
We have distinguished two classes of sensorimotor contingencies that are typical of vision: one that is a consequence of the structure of the visual apparatus, and one that is a consequence of the nature of 3D space. What should be noted is that both kinds of sensorimotor contingencies are independent of the nature of the objects which are being seen. The laws that govern the changes that are created by blinks, eye movements and body movements are global laws that apply to whatever objects are in the visual scene. Thus, these particular types of sensorimotor contingency, because they are independent of objects but particular to the visual modality, could be considered to constitute the defining characteristics of visual sensation, and they are what distinguish vision from sensation in other modalities.
Take the tactile modality as a further example. Here the sensorimotor contingencies are quite different from the visual modality. Moving one's hand over a surface causes the texture elements to move in a way analogous to how the image shifts when one moves one's eye. But there are a number of important differences. For example, the way tactile acuity varies over the surface of the hand is nothing like the way spatial acuity varies over the retina. Furthermore, bringing one's hand nearer and farther from an object does not create an expanding or contracting tactile flow-field like that created on the retina. On the contrary there are laws of contact and pressure which have no equivalent among the sensorimotor laws of vision.
It is a very important aspect of our approach to sensation that we claim that what determines the particular, visual, tactile, auditory, olfactory, etc. nature of a stimulation is nothing directly to do with the sensory pathways or brain areas which carry the nervous influx. Rather, what determines the experienced sensory modality of a stimulation are the sensorimotor laws governing that stimulation. This important concept opens the possibility for sensory substitution. If we can, through tactile stimulation for example, create sensorimotor contingencies similar to those usually obtained via visual pathways, we should be able to create sensations through touch which will be perceived of as having a visual nature. There is indeed evidence that this is possible.
Thus, Bach-y-Rita [1972](see also Bach-y-Rita [1967]; Lechelt [1986]) mounted a 20 x 20 array of tactile vibrators on the back or abdomen of a blind or blindfolded person, and found that with surprisingly little practise, the subjects report that they no longer feel the tickling stimulation on the skin, but "see" objects in front of them. Furthermore, suddenly increasing the magnification of the video camera by jogging its zoom adjustment can cause users to involuntarily jump back, as though they saw that objects were going to collide with them (Bach-y-Rita [1972]) -- and this is true irrespective of whether the stimulator array is mounted on the observer's back, abdomen, or other cutaneous area.
The discussion remains lively of course as to what extent we should say people really "see" with such a device (Morgan [1977]), and it is interesting to note that blind people seem not as interested in using the devices as the engineers who construct them had hoped (Easton [1992]; Bach-y-Rita [1983]). Nonetheless, first-person reports by people who have learnt to use the devices certainly favor the notion that some kind of vision is involved (Guarniero [1974]; Guarniero [1977]; Apkarian [1983]).
Contrary to the sensorimotor contingencies which we considered in the previous section, therefore, there are other sensorimotor contingencies which are related, not to the nature of the visual apparatus or to the 3D space in which it is embedded, but rather to the nature of objects themselves. A pitcher, for example, is a thing that has the property that, depending on how you turn it, a protuberance (the handle) appears and disappears. A glass, in contrast, is a thing that does not have this property. We suggest that perception could be considered to be the exercise of mastery of this kind of object-related sensorimotor contingency.
Note that, just as was the case for our definition of sensation, there is nothing about our definition of perception which prevents a machine from having perception. We shall come to the specifically human aspects of perception later.
As a concrete illustration of what perception is under this approach, consider first an example from the domain of tactile perception: the children's game in which a household object like a cork, a potato, or a pencil sharpener, for example, is put in an opaque bag, and the child must attempt to identify the object by feeling it with his hand in the bag. The striking aspect of this game is that at first, when you have not yet identified the object, you are aware of local bits of texture, protuberances, edges, etc., but not of holding a particular object. Suddenly however, the "veil falls", and the previously unrelated parts come together into a whole. You no longer have the impression of a collection of incomprehensible protuberances, smoothnesses, edges, but of holding, say, a swiss army knife. It is worth playing this game in order to understand this sudden feeling of recognition, like an illumination. Once the illumination has occurred, you no longer feel the local sensations that you were feeling before, but you feel the object as a whole object. Even parts of the object that your fingers are not currently in contact with, somehow are perceived as being present.
We suggest that this feeling of presence derives from the fact that once the object has been recognized, you "have tabs" on it, you can exercise your mastery of the way it "behaves" under your grasp. You know that if you move your fingers upwards, you will encounter the ring attached to one end of the knife, and if you move the other way, you will encounter the smoothness of the plastic surface, and the roughness of the corkscrew , etc. It is this knowledge which constitutes the haptic perception of the object. In fact you need not do anything at all, and yet you have the acute feeling of holding a swiss army knife.
Notice that the fact that the sensitivity of different parts of your hand (fingertips, palm, finger nails, etc.) is quite different, and the fact that your hand is composed of fingers of different shapes and lengths and having spaces between them, has no impact on your ability to recognize the object. You know the difference that will occur if you use your fingernails rather than your fingertips to touch the surfaces, and if you lodge the object in the palm of your hand or put it inbetween finger and thumb. In fact your fingernails and the spaces between your fingers, far from being useless non-sensitive zones of the hand, can be used actively as a way of assessing the texture or the size of parts of an object.
The same analysis can be applied to visual perception -- indeed the idea that vision might consist in "palpation" by the eyes was already mentioned by Merleau-Ponty [1968] and developed by MacKay [1962; MacKay [1967; MacKay [1973], who suggested considering vision like a giant hand that samples the environment.
Suppose you are looking at an apple. In central vision you have stimulation from your retina corresponding to redness, and above it in peripheral vision you have stimulation of lesser resolution corresponding to the green leaves. You know this is an apple because you know that if you move your eyes up to the green bits, the change in stimulation will be typical of the change that occurs when green things move from periphery into central vision. You perceive this as an apple because you additionally know that if you turn it in a certain way, you can make the green leaves disappear and re-appear, and because you know that, being round, the apple's profile will not change very much when it rotates about itself.
Having the knowledge of the set of all possible things that we can do and the associated changes that occur is of course something, which in the case of vision, is generally so automatic and immediate, that we do not have the impression of "illumination" or the "veil falling" that we referred to in the case of the hand-in-the-bag game. However the impression can sometimes be obtained with pictures that are difficult to interpret, such as the well-known figure of the Dalmation, or the example in Figure 1. Until you suddenly realize what it is, you do not have the impression of the presence of a seen whole.
Figure 1. Picture which some people find difficult to decipher. In case of difficulty, turn it upside down.
Notice in the case of vision, just as for tactile perception, the local sensitivities of different parts of the retina are quite different, with the central region of the eye having high spatial and color resolution, whereas the peripheral regions have lower resolution. There are other non-uniformities due to the blind spot and vascular scotoma, which obscure large portions of the retina (see below). But, just as the non-homogeneities of the hand do not prevent the nature of objects from being assessed - on the contrary, they are sometimes an aid in determining the nature of an object -- the imperfections of the retina and of the eyes' optics are no hindrance and may actually be an aid to the identification of objects. The fact that the centrally and peripherally fixated parts of an object are, at any moment, being sampled with dramatically different spatial and color resolutions is irrelevant to the feeling of recognition that we get.
We have the impression of seeing a whole object, and not simply the part which is currently being sampled by the high resolution fovea. The reason is that recognition of an object occurs, not when neural excitation due to the object arrives in some cortical area, but when we are exercising our mastery of the way the object behaves under exploratory manipulation, when we have "tabs" on the behavior of the object as we move our eyes over it, or as we move the object before our eyes. What we mean by seeing an object is having knowledge of the set of all the possible things that we can do with respect to the object and the associated changes that will occur in the sensory influx when we do them.
Figure 2. Scotoma in the visual field present in normal people's eyes, caused by blood vessels which cover the anterior surface of the retina. The blind spot is seen on the right, covering a region subtending approximately 3-5 degrees horizontally and 8-10 degrees vertically. From Weekers & Humblet [1945]
However under the approach we are putting forward, and also as suggested by Durgin, Tripathy, & Levi [1995] and Pessoa, Thompson, & Noë [1998], there is no need for a filling-in mechanism to fill in the blind spot, any more than there should be a tactile filling-in mechanism to compensate for the spaces between our fingers when we do tactile recognition. Since visual recognition of objects occurs when we are exercising our mastery of the changes that occur as we move with respect to an object, the presence of "imperfections" such as the blind spot poses no problem. Just as people with dirty spectacles do not have the impression that the things they are looking at are dirty, the fact that we have a blind spot does not give us the impression that there is a "black hole" in our visual fields. Brain mechanisms that some authors have located and which appear to be doing filling-in type operations are, in our view, not actually filling in mechanisms, but simply part of the normal sampling machinery that the visual system uses to explore the environment at different spatial scales (O'Regan [1998]).
Similar arguments can be made in relation to the problem of saccadic suppression and the stability of the visual world. A large literature exists whose purpose it is to try to understand how it could be that we do not notice the tremendous purturbations that are caused on the retinal image every time the eye makes a saccade -- which is between three to five times a second, all the waking day (e.g. Matin [1974]; Shebilske [1977]; Li & Matin [1997]; Irwin [1992]; [Irwin [1991]; Grüsser [1986]; Bridgeman, van der Heijden, & Velichkovsky [1994]; McConkie & Currie [1996]). In our view this literature is misguided. Just as it would be curious to propose a mechanism whose purpose it is to compensate for hand movements in haptic object recognition (for example, a mechanism that ensures that the object is not perceived as disappearing when the hand is briefly lifted off it), there is no need for a mechanism to compensate for eye movements in visual recognition. Indeed, as stressed by (Neisser [1976]), vision does not consist in the reception and sticking-together of successive images captured by the eye. On the contrary, vision is constituted by knowledge of the changes that occur through eye movements and body movements. Vision occurs through movement, not despite movement. On the other hand, we obviously do not wish to deny the vast literature on saccadic suppression and the extraretinal signal showing that there are mechanisms in the brain which modify neural activity as a function of eye position and when saccades occur. But we claim that the purpose of these mechanisms is not to compensate for eye movements. On the contrary, their purpose must be part of the normal sampling of correlated visual and motor activity which constitutes visual perception.
It is certain that neural convergence does in fact exist in the brain: the convergence is certainly a computational mechanism used to route information in such a way that object attributes like "red" and "triangle" and their spatial locations can be disambiguated and made available to brain mechanisms that determine behavior. Synchronous firing may also be a brain mechanism that is used for similar computational purposes.
But we claim that what has been called the binding problem is actually a pseudo-problem: a by-product of the, in our opinion erroneous, conception that visual experience is generated by activation of a brain mechanism. Only if we assume this erroneous idea, are we obliged to postulate a way for the individual aspects of a stimulus to somehow "come together" so as to activate the experience-providing brain mechanism.
Let us abandon the experience-as-a-brain-mechanism view, and consider the analogy with Porsche-driving again. As noted earlier, the experience of driving a Porsche may depend causally on disparate neural processes. There is no need to suppose that these underlying components must be bound together by some unifying mechanism. The unity of the experience, we believe, can be explained by the unity of the activity itself, that is, by the fact that there is a coherent pattern of coordinated behavior that is responsive to external circumstances. And just the same holds for perceptual experience. The unity of perceptual experience can be explained by the fact that there is unity at the level of the activity in which perceptual experience consists, by the fact that there is an organized pattern of exploration and interaction with the environment. There's no need, therefore, to appeal to a unity at the level of underlying neural mechanisms.
A related example is that of life. Each of the processes underlying life: respiration, reproduction, ingestion, digestion, movement, maturation, cell division, to name but a few, has its own more or less independent physico-chemical mechanisms. Nothing extra is needed to endow an organism with life, once each of the mechanisms that underlie life is in place. No extra "life essence" is needed to bind together these different processes so that life has unity. On the contrary, life has unity because it is a coordinated way of behaving within an environment. It is not something that is generated from the underlying mechanisms. It is something that is constituted by a way of behaving.
A driver, for example, would be said to be aware of a red traffic light if, in addition to the mastery of sensorimotor contingencies associated with the red light, his attunement to these sensorimotor contingencies is integrated into his planning, rational thought or linguistic behavior.
Depending on the extent to which the seeing of the red light is incorporated into his planning or thought, the driver would be said to be aware of the red light to varying degrees. For example if he is driving while at the same time engaged in animated conversation with a friend, he may not be aware of the red light at all. Or he may be indirectly aware of it, because he realizes that stopping is going to make him late for an appointment. If the driver is an artist interested in shades of red, on the other hand, he may notice that this traffic light has a different hue than usual, and therefore be particularly aware of the color of the light, though he may not notice its shape.
Notice that there is nothing about this characterization of awareness that prevents its attribution to a machine -- it is conceivable that a sufficiently complex machine, with a sufficiently wide range of possible behaviors, in a sufficiently rich environment, could reasonably be said to be aware in this sense[1].
This is a hotly contested area, one where it is difficult to tread without stepping on the discarded remains of unsuccessful gambits. We venture forward, therefore, with a light step.
First, let us note that we have accounted for what it is like to see something red. After all, we have presented, in admittedly rough outline, the factors that differentiate the seeing of something red from the haptic feel of something red, or the seeing of something blue. What more is there to be accounted for?
One might respond: the further hitherto-not-yet-explained element is that distinctive feeling of the presence of redness. But the explanation of this "feeling of the presence of redness" is ready to hand. Suppose you look at a red colored wall. The redness is on the wall, there to be appreciated. Because we have continuous access to the present redness, it is as if you are continuously in contact with it. This would explain the fact that the redness would seem to be continuously present in experience. This point can be sharpened. The "feeling of present redness" that would seem to accompany the seeing of something red is to be explained by the fact that we understand (in a practical sense) that at any moment we can direct our attention to the red wall. That is to say, our feeling of present redness consists in our awareness of our perceptual access to environmental redness.
It would seem, then, that the "raw feel" of redness can be explained by our theory in terms of our second-order awareness of our own visual capacities. This account gains in credibility when we recall the feeling of "illumination" that occurs when a stimulus is recognized in the child's hand-in-the-bag game: one suddenly really "feels" the presence of a whole object. This feeling is clearly nothing more than the coming together of the knowledge associated with the different parts of the object: a kind of "being at home" with the various things that one can do in regard to the object. This example suggests how it might be possible that a state of knowledge about existing contingencies could actually constitute a feeling.
Another illustration which may help in rendering our approach more plausible is to consider cases where the word "feeling" is used outside what is normally taken as the domain of sensory experience. Consider what is meant when a person says he "feels rich". What he means is that he can, if he wishes, buy a yacht or an expensive car, take cash out of the bank or go on a cruise around the world, etc. His feeling of richness actually consists in knowledge of things he can do (other examples might be feeling British, feeling lonely, virtuous, etc.).
Obviously the "feeling" of richness does not have the same acute ongoing quality as the experience of a red light or a bell ringing or a pain. This difference could reside in the fact that, unlike the case of richness, the knowledge associated with perceptual experience is intimately linked to motor activity: the minutest eye-muscle twitch creates an associated change in the neural influx from the retina, but no muscle twitch changes the availability of cash at the bank. Another fact that gives perceptual feelings a more "real" quality could be that the associated sensorimotor contingencies are highly ingrained, learnt during maturation, and known perhaps only tacitly. Finally, we have already mentioned the very important difference with other mental activities: the fact that low-level perceptual systems possess genetically determined orienting responses which cause the perceptual apparatus to exogenously orient to sudden changes in stimulation. Thus, when a light turns on, the eyes are attracted to the change, but when my bank account goes empty, there is no immediate, automatic reaction of my body. Sensory stimulation thus provides sensorimotor contingencies which are profoundly driven by local external events, whereas feelings like those of "being rich" or "being virtuous" are not so tightly sensorimotor. This difference is so clear-cut that it may be better to say that the feelings of richness or virtuousness do not really provide any kind of "raw feel" at all.
The subjective impression people have of their visual world is one of great richness, with the feeling of almost infinite detail and color spread out before their eyes. The most natural explanation for this would appear to be the idea that in the brain, there is an internal representation of the world whose activation provides this experience. In order to provide the perceived richness of the experience, the internal representation must presumably also be very rich.
The theory we have developed here rejects the notion that experience is caused by activation of any brain mechanism -- for if it were, then we would have to explain how activation of some brain mechanism (something material) provides experience (something immaterial). On the contrary, our theory states that the character of the visual experience of the external world depends on the fact that the visual world is immediately accessible to our exploration. We see something, not because its image is impinging on our retinas, but because we know we can, by moving our eyes, our bodies, or the object itself, change the retinal stimulation that the object creates in certain known ways.
A way of thinking about this view of what seeing consists of, is to adopt the analogy of what might be called the "the world as an outside memory" (O'Regan [1992]; Minsky [1988]). Instead of supposing that we have visual experience because an internal representation of the world is activated, we claim that the outside world functions in some sense as an external memory store. Consider what happens with normal memory: The slightest flick of your thoughts to something you want to remember -- say the color of your grandmother's eyes -- suffices to bring that thing to your attention, and you can immediately recall it. Having memory of your grandmother's eyes thus consists in the fact that you know you can, by a mental effort, bring that bit of information into your mind. Similarly, with vision, the slightest flick of your eyes or of attention can bring up details about something before you. Thus, seeing something consists in the fact that you know you can, by the appropriate eye, body, or object movement, cause changes that provide information about that thing.
But the view we are taking is rather curious, because it says that to have the ongoing, occurrent, perception of richness of the visual world, the richness does not actually have to be continually impinging on the retina or activating some internal representation. It just has to be potentially obtainable by the appropriate movement.
From the point of view of brain storage and efficiency of calculation, it is of course a great advantage not to have to store the contents of the entire visual field, but rather to use the world itself as its own storage buffer.
On the other hand, this point of view makes a rather counter-intuitive prediction. It predicts that if something in the visual world were to change, unless at the moment of the change you happened to be attending to it, you would not notice the change. Since attentional capacity is extremely limited -- we can only consciously think about one thing at a time -- the question then arises of why we do not normally miss all sorts of things that change in our visual field?
The answer undoubtedly lies in the fact that, as we have pointed out before, the low-level visual system possesses automatic transient-detection mechanisms which cause attention to be exogenously oriented towards any change in luminance, color or position in the visual field. The best known example of such mechanisms might be motion detectors. When anything changes in the visual field, the eye, or attention, is immediately directed to the location of the change, and the changing element, because it is being attended to, is seen. This low-level "watchdog" mechanism therefore provides a way of accounting for perceptual awareness of details in the environment without supposing that everything is actually stored in the brain.
These considerations lead to the possibility of some interesting empirical verifications. If somehow we could render the normal attention-grabbing transient-detection mechanisms inoperative, then changes in a scene should not be noticed unless they were being directly attended to. We tested this prediction in a series of experiments in which the attention-grabbing transient was swamped in a variety of ways by the occurrence of other, irrelevant transients.
In one set of experiments we used a brief flicker that covered the whole image at the moment that the "true" scene change occurred (Rensink, O'Regan, & Clark [1997]; Rensink, O'Regan, & Clark [2000]). In another experiment the scene change was synchronized with the occurrence of a blink (O'Regan, Rensink, & Clark [1997]). Other workers have synchronized scene changes with the occurrence of eye saccades (Grimes [1996];McConkie & Currie [1996]; Irwin & Gordon [1998]). The principle of these experiments is that in all cases, the "true" change occurs simultaneously with a brief global change in the retinal image. This global change produces a large transient that overloads the local transient detectors which would normally signal the change location. Attention is therefore prevented from orienting to the location of the change. Another experimental paradigm that we have used involved what we called "mudsplashes": like mud splattered briefly on a car windshield, these created brief diversions which, when synchronized with the occurrence of a change in the visual field, acted as decoys, and attracted attention away from the location of the "true" change (O'Regan, Rensink, & Clark [1996]).
As predicted from the theory, in all these experiments very large changes could be made without observers noticing them. The changes could be so large that they occupied a significant portion of the picture, and were flagrantly obvious when not synchronized with the flicker, blink, saccade, or mudsplash (examples can be seen on the website http://nivea.psycho.univ-paris5.fr). The phenomenon was particularly striking in the "blink" experiment, where, it was shown by measuring eye movements, that even when people were looking directly at the change, in almost 50% of the cases they did not notice it (O'Regan et al. [1997]). Of course the fact that a person can be directly looking at something and yet not see it is quite compatible with our theory, since, as we have said, what determines whether you are aware of seeing something is not the fact that it impinges on your retina, but rather the fact that you are making use of it in your planning, thought, decisional or linguistic behavior. Even when fixating something directly, there may be some aspects of what you are fixating which you are, in this sense, not making use of, and so which you will not notice if it changes. For example, fixating in the middle of a word, you may be recognizing the word and not notice that the shape of the very letter you are fixating has changed, or that its color or the background color visible behind the letter has changed.
On our view, experience is not something that happens in us but is something we do. Perceptual experience consists in ways of exploring the environment. We have decomposed these ways of exploring the environment into several parts.
The first two parts correspond to what is usually called sensation and perception and occur when a person exercises mastery of the laws of sensorimotor contingency that govern how actions affect sensory inputs. Sensation occurs when a person exercises mastery of those sensorimotor contingencies which are typical of a sensory modality (in general those contingencies that are related to the way the modality samples the space in which it is embedded).
Perception occurs when a person has mastery of those sensorimotor contingencies which are typical of the way the attributes of objects are sampled by the particular sensory modality.
When one's sensation and perception are integrated into current planning, rational thought and speech behavior, we say that the perceiver is perceptually aware of that which is perceived.
We have also shown that what could be called the "raw feel" of sensory experience can be accounted for under our theory. This is because what people mean by such talk can be understood in terms of differences in the patterns of sensorimotor contingencies governing exploration of the world in different modalities and also with reference to the possibility of becoming aware of qualites (such as colors) that are a stable feature of one's immediate environment.
As a whole, the theory provides a unifying framework in which
to consider the relations between sensory experience in different sensory
modalities (and this includes unusual cases such synesthesia and tactile
visual sensory substitution), and the relation between sensory experience
and other human activities. The theory also provides insights into the
so-called "binding problem", and, within the domain of vision, into the
question of how vision can seem perfect despite what would seem to be the
glaring imperfections of the visual system (perturbations due to eye saccades,
retinal scotomas, non-homongeities in spatial and color sampling). Among
other empirical results that the theory relates to, and that have been
discussed in more detail elsewhere (O'Regan & Noë [under revision]),
the theory also speaks to the issue of the nature of the internal representation
of the visual world and makes predictions that have been verified concerning
the phenomenon of "change blindness".
Acknowledgments
We thank Ken Knoblauch for discussion on the draft of this paper. A.N.
gratefully acknowledges the support of faculty research funds granted by
the University of California, Santa Cruz.
[1] Certainly in order for the notion to be applicable however, the machine would have to have quite a sophisticated reasoning ability and would have to reside in quite a complicated environment. A medical diagnosis system or chess-playing machine could be said to evaluate possible plans before deciding on what actions to undertake. But one would not want to say that the machine is "aware" of the decisions it is making. Awareness seems to require broader social capabilities than simply making a diagnostic or advancing pieces on a chessboard. If the machines could additionally evaluate the impact that they were going to have on the patients or chess opponents (say, by not telling a patient he has cancer, or purposefully losing a game in order not to upset a child), then one would be more willing to use the word "aware" in describing the machine's behavior.
References
Abeles, M., & Prut, Y. (1996). Spatio-temporal firing patterns in the frontal cortex of behaving monkeys. J Physiol Paris, 90(3-4), 249-250.
Apkarian, P. A. (1983). Visual training after long term deprivation: a case report. Int J Neurosci, 19(1-4), 65-83.
Bach-y-Rita, P. (1967). Sensory plasticity. Applications to a vision substitution system. Acta Neurol Scand, 43(4), 417-426.
Bach-y-Rita, P. (1972). Brain mechanisms in sensory substitution. New York: Academic Press.
Bach-y-Rita, P. (1983). Tactile vision substitution: past and future. Int J Neurosci, 19(1-4), 29-36.
Barlow, H. B. (1972). Single units and sensation: a neuron doctrine for perceptual psychology? Perception, 1(4), 371-394.
Block, N. (1995). On a confusion about a function of consciousness. Behavioral and Brain Sciences, 18(2), 227-247.
Braitenberg, V. (1984). Vehicles. Experiments in synthetic psychology. Cambridge, Mass.: Bradford, MIT Press.
Brecht, M., Singer, W., & Engel, A. K. (1998). Correlation analysis of corticotectal interactions in the cat visual system. J Neurophysiol, 79(5), 2394-2407.
Bridgeman, B., van der Heijden, A. H. C., & Velichkovsky, B. M. (1994). A theory of visual stability across saccadic eye movements. Behavioral and Brain Sciences, 17, 247-292.
Broackes, J. (1992). The autonomy of colour. In K. Lennon & D. Charles (Eds.), Reduction, Explanation, and Realism (pp. 421-465). Oxford: Oxford University Press.
Castelo-Branco, M., Neuenschwander, S., & Singer, W. (1998). Synchronization of visual responses between the cortex, lateral geniculate nucleus, and retina in the anesthetized cat. J Neurosci, 18(16), 6395-6410.
Chalmers, D. J. (1996). The conscious mind: In search of a fundamental theory. New York, NY, USA: Oxford University Press.
D'Zmura, M., & Iverson, G. (1994). Color constancy. III. General linear recovery of spectral descriptions for lights and surfaces. J Opt Soc Am A, 11(9), 2398-2400.
Durgin, F. H., Tripathy, S. P., & Levi, D. M. (1995). On the filling in of the visual blind spot: Some rules of thumb. Perception, 24(7), 827-840.
Easton, R. D. (1992). Inherent problems of attempts to apply sonar and vibrotactile sensory aid technology to the perceptual needs of the blind. Optom Vis Sci, 69(1), 3-14.
Gray, C. M., & Singer, W. (1989). Stimulus-specific neuronal oscillations in orientation columns of cat visual cortex. Proc Natl Acad Sci U S A, 86(5), 1698-1702.
Grimes, J. (1996). On the failure to detect changes in scenes across saccades. In K. Akins (Ed.), Perception (pp. 89-110). New York & Oxford: Oxford University Press.
Grüsser, O. J. (1986). Interaction of efferent and afferent signals in visual perception. A history of ideas and experimental paradigms. Acta Psychol (Amst), 63(1-3), 3-21.
Guarniero, G. (1974). Experience of tactile vision. Perception, 3(1), 101-104.
Guarniero, G. (1977). Tactile Vision: a personal view. Visual Impairment and Blindness, 125-130.
Husserl, E. (1907/1991). Ding und Raum. Vorlesungen 1907. Hamburg: Meiner.
Irwin, D. E. (1991). Information integration across saccadic eye movements. Cognitive Psychology, 23, 420-456.
Irwin, D. E. (1992). Perceiving an integrated visual world. In D. E. Meyer & S. Kornblum (Eds.), Attention and performance XIV: Synergies in experimental psychology, artificial intelligence, and cognitive neuroscience. (pp. 121-142). Cambridge, Mass.: MIT Press.
Irwin, D. E., & Gordon, R. D. (1998). Eye movements, attention, and transsaccadic memory. Visual Cognition, 5(1-2), 127-155.
Koenderink, J. J. (1984). The concept of local sign. In A. J. van Doorn, W. A. van de Grind, & J. J. Koenderink (Eds.), Limits in perception (pp. 495-547). Zeist, Netherlands: VNU Science Press.
Lechelt, E. C. (1986). Sensory-substitution systems for the sensorily impaired: the case for the use of tactile-vibratory stimulation. Percept Mot Skills, 62(2), 356-358.
Levine, J. (1983). Materialism and qualia: The explanatory gap. Pacific Philosophical Quarterly, 64(354-361).
Li, W., & Matin, L. (1997). Saccadic suppression of displacement: Separate influences of saccade size and of target retinal eccentricity. Vision Research, 37(13), 1779-1797.
Llinas, R., & Ribary, U. (1993). Coherent 40-Hz oscillation characterizes dream state in humans. Proc Natl Acad Sci U S A, 90(5), 2078-2081.
MacKay, D. M. (1962). Theoretical models of space perception. In C. A. Muses (Ed.), Aspects of the theory of artificial intelligence (pp. 83-104). New York: Plenum Press.
MacKay, D. M. (1967). Ways of looking at perception. In W. Wathen-Dunn (Ed.), Mopdels for the perception of speech and visual form (pp. 25-43). Cambridge, Mass.: MIT Press.
MacKay, D. M. (1973). Visual stability and voluntary eye movements. In R. Jung (Ed.), Handbook of sensory physiology, Vol. VII/3A (pp. 307-331). Berlin: Springer.
Matin, E. (1974). Saccadic suppression: a review and an analysis. Psychol Bull, 81(12), 899-917.
McConkie, G. W., & Currie, C. B. (1996). Visual stability across saccades while viewing complex pictures. Journal of Experimental Psychology: Human Perception & Performance, 22(3), 563-581.
Merleau-Ponty, M. (1968). Résumés de cours au Collège de France. Paris: Gallimard.
Minsky, M. (1988). The society of mind: Simon & Schuster.
Morgan, M. J. (1977). Molyneux's question. Vision, touch and the philosophy of perception. Cambridge: Cambridge University Press.
Müller, J. (1838). Handbuch der Physiologie des Menschen. (Vol. V). Coblenz: Hölscher.
Nagel, T. (1974). What is it like to be a bat? Philosophical Review, 83(435-456).
Neisser, U. (1976). Cognition and reality. Principles and implications of cognitive psychology. San Francisco: W.H. Freeman.
O'Regan, J. K. (1992). Solving the "real" mysteries of visual perception: the world as an outside memory. Canadian Journal of Psychology, 46, 461-488.
O'Regan, J. K. (1998). No evidence for neural filling in -- Vision as an illusion -- Pinning down "enaction". Behavioral and Brain Sciences, 21(6).
O'Regan, J. K., & Noë, A. (under revision). The sensorimotor contingency theory: toward a new account of vision and visual consciousness. Behavioral and Brain Sciences.
O'Regan, J. K., Rensink, J. A., & Clark, J. J. (1996). "Mud splashes" render picture changes invisible. Invest. Ophthalmol. Vis. Sci., 37, S213.
O'Regan, J. K., Rensink, R. A., & Clark, J. J. (1997). Picture changes during blinks: not seeing where you are looking and seeing where you don't look. Invest. Ophthalmol. Vis., 38, S707.
Pessoa, L., Thompson, E., & Noë, A. (1998). Finding out about filling in: a guide to perceptual completion for visual science and the philosophy of perception. Behavioral and Brain Sciences, 21(6), 723-802.
Piaget, J. (1937/1977). La construction du réel chez l'enfant. (6th ed.). Neuchatel: Delachaux & Niestlé.
Platt, J. R. (1960). How we see straight lines. Scientific American, 202(6), 121-129.
Poincaré, H. (1905). La valeur de la science. Paris: Flammarion.
Ramachandran, V. S. (1992). Filling in gaps in perception: I. Current Directions in Psychological Science, 1(6), 199-205.
Ramachandran, V. S. (1995). Filling in gaps in logic: reply to Durgin et al. Perception, 24(7), 841-845.
Ramachandran, V. S., & Gregory, R. L. (1991). Perceptual filling in of artificially induced scotomas in human vision. Nature, 350(6320), 699-702.
Rensink, R. A., O'Regan, J. K., & Clark, J. J. (1997). To see or not to see: The need for attention to perceive changes in scenes. Psychological Science, 8(5), 368-373.
Rensink, R. A., O'Regan, J. K., & Clark, J. J. (2000). On the failure to detect changes in scenes across brief interruptions. Visual Cognition.
Shebilske, W. L. (1977). Visuomotor coordination, visual direction and position constancies. In W. Epstein (Ed.), Stability and constancy in visual perception: Mechanisms and processes : Wiley.
Weekers, R., & Humblet, M. (1945). L'Angioscotome physiologique.
Ophthalmologica,
110, 43-158.