Alva Noë
Department of Philosophy
University of California, Santa Cruz
Santa Cruz, CA 95064
anoe@cats.ucsc.edu
http://www2.ucsc.edu/people/anoe/
submitted 14 December 1999 as revision of manuscript under review
in Behavioral and Brain Sciences.
ABSTRACT
Many current neurophysiological, psychophysical and psychological approaches to vision rest on the idea that when we see, the brain produces an internal representation of the world. The activation of this internal representation is supposed to give rise to the experience of seeing. The problem with this kind of approach is that it leaves unexplained how the existence of such a detailed internal representation is supposed to produce experience.
An alternative proposal is made here in which the idea that seeing derives from the activation of an internal representation is rejected. Instead, it is proposed that seeing is, essentially, a way of acting. It is a particular way of exploring the environment . There is no internal representation whose activity generates the experience of seeing. The outside world serves as its own, external, representation. The experience of seeing occurs when the organism masters what we call the governing laws of sensorimotor contingency.
Several lines of empirical evidence are brought forward in support
of the theory, in particular: evidence from experiments in sensorimotor
adaptation, visual "filling in", visual stability despite eye movements,
change blindness, sensory substitution, and color perception.
It is generally thought that somewhere in the brain an internal representation of the outside world must be set up which, when it is activated, gives us the experience that we all share of the rich, three-dimensional, colorful world. Cortical maps -- those cortical areas where visual information seems to be retinotopically organized-- might appear to be good candidates for the locus of perception.
Cortical maps undoubtedly exist, and they contain information about the visual world. But the presence of these maps and the retinotopic nature of their organization cannot in itself explain the metric quality of visual phenomenology. Nor can it explain why activation of cortical maps should lead us to have the impression of seeing anything at all. Something extra would appear to be needed in order to make excitation in cortical maps provide, in addition, the subjective impression of seeing.
A number of proposals have come forth in recent years to suggest how this might come about, either specifically in the domain of visual awareness, or more generally in relation to the question of consciousness.
It has for example been suggested, from work with blindsight patients, that awareness in vision may derive from a "commentary " system situated somewhere in the fronto-limbic complex (taken to include prefrontal cortex, insula and claustrum (c.f. Weisskrantz [1997] p. 226). Crick & Koch [1990], Llinas & Ribary [1993] and Singer [1993; 1995] suggest that awareness might be correlated with particular states of the brain involving coherent oscillations in the 40-70 Hz range, and which would serve to bind together the percepts pertaining to a particular conscious moment[1]. Penrose [1994] and Hameroff [1994] suggest that the locus of consciousness might be a quantum process in neurons' microtubules. Edelman [1989] holds that re-entrant signaling between cortical maps might give rise to awareness. A variety of other possibilities that might constitute the "neural correlate of consciousness" has been compiled by, for example, Chalmers [1996b].
A problem with proposals of this kind is that they do little to elucidate the mystery of visual consciousness (as pointed out by, for example, Chalmers [1996b]). For even if one particular mechanism -- for example coherent oscillations in a particular brain area -- were proven to correlate perfectly with behavioral measures of awareness, the problem of awareness would simply be pushed back into a deeper hiding place: the question would now become, why and how should coherent oscillations ever generate awareness? After all, coherent oscillations are observed in many other branches of science, where they do not generate awareness. And even if awareness is assumed to arise from some new, previously unknown mechanism, such as quantum-gravity processes in tubules, then the puzzle remains as to what exactly it is about tubules that allows them to generate awareness, whereas other physical mechanisms do not.
It is tempting to think that seeing red is like seeing pink because the neural stimulation going on when we see something red is similar to that underlying our perception of pink: almost the same ratios of long, medium and short wavelength photoreceptors will be stimulated by red and pink. But note that though this seems reasonable, it does not suffice: there is no a priori reason why similar neural processes should generate similar percepts. To suppose otherwise is to assume a particular account of psychoneural correspondence.[2 ] But surely, how neural processes underwrite perceptual experiences is precisely what requires explanation. For example, one might think that neural activity is just a code. But then, as such, it is potentially arbitrary. In that case, an explanation is needed of what sensory experience will be associated with each element of the code. Why, for example, should more intense neural activity provoke more intense experiences? And what exactly is the mapping function: is it, for example, linear, logarithmic, or a power function? And why is it one of these rather than another? Even these questions leave open the more fundamental question of how a neural code could ever give rise to experience at all.
Not very much scientific investigation has addressed this kind of question. Most scientists seem satisfied with some variant of Müller's [1838] classic concept of "specific nerve energy". Müller's idea, in its modern form[3] amounts to the claim that what determines the particularly visual aspect of visual sensations is the fact that visual sensations are transmitted by particular nerve pathways (namely those originating in the retina and not in the cochlea) that project to particular cerebral regions (essentially cortical area V1). It is certainly true that retinal influx comes together in relatively circumscribed areas of the brain, and this may facilitate extraction of those types of information that are the substrate of visual sensation. But what is it about these pathways that generates the different sensations? Surely the choice of a particular subset of neurons or particular cortical regions cannot, in itself explain why we attribute visual rather than auditory qualities to this influx. We could suppose that the neurons involved are of a different kind, with, say, different neurotransmitters, but then why and how do different neurotransmitters give rise to different experiences? We could say that the type of calculation done in the different cortical areas is different, but then we must ask, how could calculations ever give rise to experience? Something still needs to be explained.
A first characteristic of the visual mode of exploration of the world is the fact that it is carried out with the visual apparatus, that is, with the eye with its characteristic optical properties, capacities for movement, retina, etc. These "tools" have certain properties, and the nervous inflow and outflow that occurs when they are used obeys certain laws. The associated laws of sensorimotor contingency are specific to exploration by the visual apparatus, and are different from the laws that are involved in tactile, auditory or other types of sensory experience. Thus, what we mean when we say we have visual rather than auditory sensations (for example) is that the particular contingencies typical of exploration with visual apparatus are being obeyed.
The notion of sensorimotor contingency is fundamental in the theory to be presented here. It is also used to characterize the second important aspect of vision we shall consider: When objects are explored with the visual apparatus, owing to the fact that they are embedded in a three-dimensional world and are being apprehended from a particular distance and point of view, certain laws are always obeyed. In particular, moving the object makes parts of the object appear and disappear and makes its apparent size or the amount of light reflected from it change in predictable ways. These laws are examples of the sensorimotor contingencies typical of the way three-dimensional objects are apprehended by visual exploration. These laws provide the visual (rather than say tactile) aspects of shape, color, brightness and texture, and could be considered to be the basis for the visual mode of identification and categorization of visual stimuli. These kinds of sensorimotor contingencies characterize the specifically visual attributes of objects.
The third important aspect of vision we shall consider concerns the fact that we humans, unlike artificial systems, have visual awareness. One thing that we mean by this is that we know that we are able to use the information we are currently acquiring from the environment, in order to influence our behavior and decisions. That is, not only are sensorimotor contingencies being obeyed that are fixed by the visual apparatus and the attributes of objects, but the fact that they are being obeyed is itself integrated into the reasoning, action-guiding activities, judgments, or potential linguistic utterances of the person. This integration of ongoing patterns of exploration with the current exercise of our capacities for planning, speech and rational thought, constitutes, or so we suggest, visual awareness.
The outline of this paper is as follows. In Sections 2 and 3 we develop and extend our account of these three important aspects of vision. We argue that the sensorimotor contingency approach we propose enables us better to understand the phenomenon of visual experience. In Section 4 we discuss a consequence of these ideas, namely that visual perception does not (contrary to what has been widely supposed) require that the brain constructs detailed internal models of the environment. The world can serve as its own representation--as a kind of "outside memory," as argued by O'Regan [1992]. In Section 5 we review a broad range of empirical phenomena. These phenomena span a variety of domains such as visual stability despite eye movements, filling-in of scotomas, adaptation to visual distortions, color perception, and prosthetic perception, but fall into a common mold when viewed from the standpoint of the sensorimotor contingency theory. Of special interest is the discussion of experiments on so-called "change blindness," which were in fact motivated by the present framework.
The last sections of the paper (Sections 6 and 7) look at the status of neurophysiological data in relation to the present theory and consider the philosophical question of whether visual consciousness has really been captured by the present framework.
There is an analogy to be drawn between this example and the situation faced by the brain. In the brain's case, there would seem to be nothing that in itself differentiates nervous influx coming from retinal, haptic, proprioceptive, olfactory and other senses, and there is nothing to discriminate motor neurons that are connected to extraocular muscles, skeletal muscles or any other structures. Even if the size, the shape, the firing patterns, or the places where the neurons are localized in the cortex differ, this does not in itself confer them with any particular visual, olfactory, motor or other perceptual quality.
We propose that what differentiates vision, from audition or touch, say, is the structure of the rules governing the sensory changes produced by various motor actions, that is, what we call the sensorimotor contingenciesgoverning visual exploration. Because the sensorimotor contingencies within different sensory domains (vision, audition, smell, etc.) are subject to different (in)variance properties, the structure of the rules that govern perception in these different modalities will be different in each modality.
A first law distinguishing visual percepts from perception in other modalities is the fact that when the eyes rotate, the sensory stimulation on the retina shifts and distorts in a very particular way, determined by the size of the eye movement, the spherical shape of the retina and the nature of the ocular optics. In particular, as the eye moves, contours shift and the curvature of lines changes. For example, as shown in Figure 1, if you are looking at the midpoint of a horizontal line, the line will trace out a great arc on the inside of your eyeball. If you now switch your fixation point upwards, the curvature of the line will change; represented on a flattened-out retina, the line would now be curved. In general, straight lines on the retina distort dramatically as the eyes move, somewhat like an image in a distorting mirror.
Similarly, because of the difference in sampling density of the
retinal photoreceptors in central and in peripheral vision, the distribution
of information sensed by the retina changes drastically, but in a lawful
way, as the eye moves. When the line is looked at directly, the cortical
representation of the straight line is fat in the middle and tapers off
to the ends. But when the eye moves off of the line, the cortical representation
peters out into a meager, banana-like shape, and the information about
color is radically undersampled, as shown in the bottom right hand panel
of Figure 1. Another law that characterizes the sensorimotor contingencies
that are particular to visual percepts is the fact that the flow pattern
on the retina is an expanding flow when the body moves forwards, and contracting
when the body moves backwards. Visual percepts also share the fact that
when the eyes close during blinks, the stimulation changes drastically,
becoming uniform (i.e. the retinal image goes blank).
Figure 1. as shown in the top part of the figure, if the eye fixates
the middle of a straight line and then moves to a point above the line,
then the retinal stimulation moves from a great arc on the equator of the
eye to a different, smaller great arc, that is, one which would appear
more curved on a flattened-out retina (bottom left). Because the retina
is strongly non-homogeneous, with sampling density falling off drastically
with eccentricity even within the central fovea, the cortical representation
of a straight line is distorted, as shown on the right, bottom. Moving
off a straight line therefore creates a very large distortion in the cortical
excitation: As shown on right, bottom, the thick packet tapering off towards
its ends which represents the centrally fixated line distorts into a thin,
banana shape when the eye moves off the line. Another aspect of cortical
representation is the fact that on the central retina color information
is richly sampled by photoreceptors sensitive to short, medium and long
wavelength light (shown as three types of triangle). But as eccentricity
increases even through only a degree or so, sampling is done more and more
by rod photoreceptors (shown as dots), sensitive only to a single wavelength.
Note finally that, as to be explained in Section 2.2, if the eye moves
along
the straight line instead of perpendicular to it, there is virtually no
change at all in the cortical representation. This would be true even
if the cortical representation were completely scrambled. This is the idea
underlying the theory that shape in the world can be sensed by the laws
obeyed by current sensorimotor contingencies.
In contrast to all these typically visual sensorimotor contingencies, auditory sensorimotor contingencies have a different structure They are not, for example, affected by eye movements or blinks. They are affected in special ways by head movements: rotations of the head generally change the temporal asynchrony between left and right ears. Lateral shifts of the head mainly affect the amplitude but not the frequency of the sensory input.
We therefore suggest that a first important aspect of vision is the fact that visual exploration obeys certain laws of sensorimotor contingency. These laws are determined by the fact that the exploration is being done by the visual apparatus. Indeed we propose that the distinctively visual character of vision is actually constituted by these particular laws. Just as people feel rich when they know they can do certain things like buy a yacht or go out for an expensive dinner, people have the experience of seeing rather than hearing, when they know[4] that if they were to do certain things like blink, move their eyes, etc., the resulting changes in sensory input would obey those laws corresponding to exploration using the visual apparatus.
Note, in passing, that because this aspect of vision is related not so much to the properties of objects, but to the visual apparatus itself and to the way three-dimensional objects present themselves to the visual apparatus, this aspect of vision does not imply any categorization or interpretation of the content or identity of the objects. It could thus be considered to be a fundamental, underlying aspect of visual sensation -- as distinct from visual perception , which, in the sense generally employed by psychologists, additionally involves some identification, or at least categorization, of what is being seen. The sensorimotor contingencies which are more related to visual perception, that is those that pertain to visual attributes, and which allow the visual categorization of objects, constitute the second important aspect of vision and are considered next.
On the other hand, tactile exploration of an object, even though it may be sampling the same objective properties, obeys different sensorimotor contingencies: you do not touch an object from a "point of view" -- your hand can often encompass it more or less completely for example, and you don't apprehend it from different distances; its tactile aspect does not change with lighting conditions.
There is thus a subset of the sensorimotor contingencies that are engendered by the constraints of visual-type exploration, and which correspond to visual attributes of sensed objects.
Note that unlike the sensorimotor contingencies that are visual-modality related, the sensorimotor contingencies that are visual-attribute related do nonetheless have strong links to the tactile sense: this is because attributes of three dimensional objects can also sometimes be apprehended via the tactile exploratory mode, where they present themselves as tactile shape, texture, size, distance. As shown eloquently by Piaget's work, the observer's cognitive notion of space in general will also have strong links to the laws that the brain has abstracted from the visual-attribute related sensorimotor contingencies discussed in the present section
A good illustration of sensorimotor contingencies as applied to the perception of one particular kind of visual attribute, namely visual shape, can be obtained from the records of patients whose vision has been restored after having been born blind with congenital cataract (c.f. reviews by Morgan [1977], Jeannerod [1975] and Gregory [1973]). One such patient, cited by Helmholtz [1925/1909], is surprised that a coin, which is round, should so drastically change its when it is rotated (becoming elliptical in projection). The fact that objects also drastically change in extent as a function of distance is poignantly illustrated by the case of a 13-14 year old boy treated by Cheseldon (1728, cited by Morgan, 1977, p. 20): "Being shewn his father's picture in a locket at his mother's watch, and told what it was, he acknowledged a likeness, but was vastly surpriz'd; asking, how it could be, that a large face could be express'd in so little room, saying, it should have seem'd as impossible to him, as to put a bushel of any thing into a pint."
These examples make us realize how second nature it is for people with normal vision to witness the perspective changes that surfaces undergo when they are shifted or tilted, or when we move with respect to them. The idea we wish to suggest here is that the visual quality of shape is precisely the set of all potential distortions that the shape undergoes when it is moved relative to us, or when we move relative to it. Although this is an infinite set, the brain can abstract from this set a series of laws, and it is this set of laws codes shape.
Another example of how sensorimotor contingencies can be used as indicators of visual attributes is illustrated in an aspect of Figure 1 we have not yet mentioned. We saw in the introduction that movement of the eye away from a line creates a very strong distortion in its cortical and retinal representation. Under the classical view of what shape perception requires, it would be necessary to postulate that in order to see lines as straight despite eye movements, a transformation mechanism would have to exist that compensates for these distortions. This mechanism would take the cortical representation illustrated in the bottom right of the figure, and transform it so the two dissimilar packets of stimulated neurons shown in the figure now look identical.[5] There would additionally have to be another cortical locus where this new, corrected representation was projected. The view presented here does away with these unnecessary steps
Consider the following fact: if the eye moves along the straight line instead of perpendicular to it, the set of photoreceptors on the retina which are stimulated does not change, since each photoreceptor that was on the image of the line before the eye moves is still on the image after the eye moves. This is due to an essential property of lines -- they are self-similar under translation along their length (we assume for simplicity that the line is infinite in length). Since exactly the same photoreceptors are being stimulated before and after eye movement along the line's length, the cortical representation of the straight line is therefore identical after such a movement: there is this time no distortion at all. Another interesting fact is that the argument we have just made is totally independent of the code used by the brain to represent the straight line. Even if the optic nerve had been scrambled arbitrarily, or if the retina were corrugated instead of spherical, thereby causing the image of the line to be wiggly instead of straight, or if the eye's optics gave rise to horrendous distortions, movement of the eye along the line would still not change the pattern of cortical stimulation. We see that this particular law of sensorimotor invariance is therefore an intrinsic property of straight lines, and is independent of the code used to represent them. Platt [1960] has extended such considerations to other geometrical invariants, and Koenderink [1984] has considered the more general, but related problem of how spatiotemporal contingencies in the neural input can be used to deduce intrinsic geometrical properties independently of the code by which they are represented.
In general, it will be the case that the structure of the laws abstracted from the sensorimotor contingencies associated with flat, concave, and convex surfaces, corners, etc., will be a neural-code-independent indication of their different natures. Though currently psychophysical work is being done, for example, to determine the respective importance, in determining shape, of cues derived from changes caused by movement of the object versus movement of the observer (e.g. Cornilleau-Peres & Droulez [1994], Dijkstra, Cornilleau-Peres, Gielen, & Droulez [1995], Rogers & Graham [1979; 1992]), the idea that the laws of sensorimotor contingency might actually constitute the way the brain codes visual attributes has not been developed in the literature (although see Section 3.8). However this idea is essential in the present theory.
We now turn to a third important aspect of vision about which little has been said, namely visual awareness.
Suppose you are driving your car and at the same time talking to a friend. As you talk, the vista in front of you is impinging upon your eyes. The sky is blue, the car ahead of you is red, there is oncoming traffic, etc. Your brain is tuned to the sensorimotor contingencies related to these aspects of the visual scene. In addition, some of these sensorimotor contingencies are also being used to control your driving behavior, since you are continuously adjusting your steering and adapting your speed to the moment-to-moment changes in the road and the traffic. But, since you are talking to your friend, you are not aware of most of these things. You do not notice that the car ahead is red, you do not think about the sky being blue: you just drive.
You lack, as we shall say, visual awareness of many of the aspects of the visual scene. For those scene aspects, you are no different from an automatic pilot controlling the flight of an airplane. Your behavior is regulated by the appropriate sensorimotor contingencies, but you remain visually unaware of the associated aspects of the scene.
But if you should turn your attention to the color of the car ahead of you, and think about it, or discuss it with your friend, or use the knowledge of the car's color to influence decisions you are making, then, we would say, you are aware of it.
Note an important point about this view of what awareness is, namely that it is a matter of degree. In particular, in our view, all seeing involves some degree of awareness, and some degree of unawareness. For example, if you were to probe the unaware driver waiting at the light, there would probably be some aspects of the red light that were at least indirectly being integrated into the driver's current action-guidance, rational reflection and speech. Perhaps, though not noticing the light's redness, the fact that the light was red may make him realize that he was going to be late. Or though not noting that the light was red, the driver could be noting that it was difficult to see because the sky was too bright. On the other hand, even the driver who was aware of seeing the red light may not have been aware of all its aspects, for example that the shape of the light was different from usual. A visual stimulus has a very large (perhaps infinite) number of attributes, and only a small number can at any moment be influencing one's action-guidance, rational reflection and speech behavior.
Related to the fact that awareness comes in degrees is another important fact: there is nothing magical about awareness -- nothing about it requires postulating anything except normal information-processing capacities such as many computers possess. All that is needed is that the sensorimotor contingencies that are being obeyed at this moment be currently being used in the person's planning and rational behavior. To the extent that computers and robots can plan and have rational behavior, they too would be able to become aware of the color of the car, in our definition of awareness.
We will return to some of these issues in Section 7 of this paper. For now, let us note that the present framework of sensorimotor contingencies would seem to allow for the explanation and clarification, and certainly for the scientific study of, a good deal of what makes for the subjective character of experience. Thus, one important dimension of what it is like to see is fixed by the fact that there is a lawful relation of dependence between visual stimulation and what we do, and this lawful relation is determined by the character of the visual apparatus. A second crucial feature that contributes to what it is like to see is the fact that objects, when explored visually, present themselves to us as provoking sensorimotor contingencies of certain typically visual kinds, corresponding to visual attributes such as color, shape, texture, size, hidden and visible parts. Finally, and most importantly, given that these two aspects are in place, visual awareness would seem to account for just about all the rest of what goes into making up the character of seeing. For visual awareness is precisely the availability of the kinds of features and processes making up the first two aspects for the purposes of control, thought and action.
Within this framework then, what is visual experience or visual consciousness? It is, we propose, awareness of seeing. So, for example, you may become visually aware (in the sense we have already explained above) that an object is a plate, or of the fact that the plate is round, or of the fact that the plate's profile actually looks elliptical (even though the plate is circular), or of the fact that the plate is white, or of the fact that though it is white, there are parts of it which are dark because they are in shadow. Or of some more "raw" aspect of the plate such as simply the fact that there is something of a visual nature present in front of you. All these forms of awareness are aspects of visual experience.
Our proposal has an important advantage. Most current approaches to the question how does visual experience arise proceed by looking for a special new mechanism which would cause experience or consciousness to emerge, e.g. mechanisms like reverbatory feedback loops, synchronous oscillations or quantum gravity processes. But as noted in the introduction, whatever mechanism is ultimately postulated, some explanation will have to be given of how that mechanism actually imbues the system with consciousness. This is the so-called problem of the "explanatory gap" (Levine [1983]).
This problem of the explanatory gap vanishes on our sensorimotor approach. Experience is not a feeling or a sensation or an ingredient that is added to perceptual activity. Thus there is no need for the discovery of a mechanism whose function is to produce the experience. Rather, on our view, experience consists in our awareness of our perceptual activity, that is, our ability to make use of and govern it. And this ability is fully open to scientific scrutiny, since it manifests itself as a range of behaviors that can be described and accounted for in scientific (e.g. mechanistic) terms.
As said, the question of visual experience and consciousness is extremely controversial, and we will make some further attempts to render our view plausible in Section 7.
One point concerns the notion of "mastery" or "attunement", which we need in order to understand what it means to say that a person has registered the laws of sensorimotor contingency. Two other points concern the relation of this "mastery" to propositional knowledge, and to other ways of knowing.
A particularly important point in this section concerns the role that the brain plays in our theory -- we shall see that the link that we postulate between brain activity and visual experience is not a direct link. This curious fact is actually what gives the theory a means of escaping from the "explanatory gap" problem of understanding how a physical system might generate consciousness. We will make use of this point much later in section 6, where we come back to the role of evidence from neuroscience.
In the last paragraphs of this section we recall the advantages of the sensorimotor approach: escaping from the explanatory gap; allowing the phenomenology of sensory modalities to be accounted for, and finally we acknowledge our debt to other workers, in particular D. MacKay.
Now consider what happens when the missile guidance system is out of order. The visual information is being sampled by its camera, it is getting into the system, being registered, but it is not being properly made use of. The missile guidance system no longer has mastery over airplane tracking.
We suggest that vision requires the satisfaction of two basic conditions. First, the animal must be exploring the environment in a manner that is governed by the two main kinds of sensorimotor contingencies (those fixed by the visual apparatus, and those fixed by the character of objects). Second, the animal, or the brain, must be "tuned to" these laws of sensorimotor contingencies. That is, the animal must be actively exercising its mastery of these laws.
Note that the notion of being "tuned" or having "mastery" only makes sense within the context of the behavior and purpose of the system or individual in its habitual setting. Consider again the missile guidance system. If exactly the same system was being used for a different purpose, say, for example, as an attraction in a fun fair, it might well be necessary for the system to have a different behavior, with scary lunges and strong acceleration and deceleration which would be avoided in a real system. Thus, "mastery" of the sensorimotor contingencies might now require a different set of laws[6]. In fact even the out-of-order missile guidance system has a kind of ineffectual mastery of its sensorimotor contingencies.
Over the course of life, a person will have encountered myriad visual attributes and visual stimuli, and each of these will have associated with it particular sets of sensorimotor contingencies. Each such set will have been recorded and will be latent, potentially available for recall: the brain thus has mastery of all these sensorimotor sets. But when a particular attribute is currently being seen, then the particular sensorimotor contingencies associated with it are no longer latent, but are actualized, or being currently made use of. In the language of the missile guidance system: the system may have stored programs that are applicable to the task of following different kinds of planes with different speed and turning characteristics. All these programs are latent, and the system has mastery of them all. But only when the system is following a particular type of plane does it invoke and follow the particular recipe for that plane.
Again: among all previously memorized action recipes that allow you to make lawful changes in sensory stimulation, only some are applicable at the present moment. The set that are applicable now are characteristic of the visual attributes of the object you are looking at, and their being currently exercised constitutes the fact of your visually perceiving that object.
Consider again the missile tracking system. At the present moment the target airplane happens to have gone out of the field of view of the missile. No information is coming into the missile's sights right now. Nevertheless the missile guidance system has a certain potential: it "knows" that by making the appropriate change in its trajectory, it should be able to bring the missile back into view. Thus, even though at this particular moment the airplane is not visible and no visual information is coming in, it is still correct to say that the missile is currently tracking its target.
A similar point can be made about sensorimotor contingencies. Vision does not require receiving information in the brain. On the contrary, it requires being master of the changes that will occur in the sensory input when certain actions are undertaken. At any given moment, there may be no actual sensory input, and no activity in the brain. Even during such moments of neural inactivity, however, the brain remains "tuned" to sensorimotor contingencies. If certain actions are taken, or certain changes were to occur in the environment, sensory data will change correspondingly.
For example, if you blink, the sensory stimulation provided by the object will change drastically, and essentially become absent. Seeing an object consists precisely in your knowledge of this sort of sensorimotor contingency, in being able to exercise your mastery of the fact that if, among other things, you blink, the sensation will change in the particular way typical of what happens when you close your eyes. If the stimulation due to the object did not change in that way, then you would not be seeing the object - you might be, for example, hallucinating about it. This is what accounts for why we do not normally notice blinks or eye movements: they are constitutive of what seeing is. On the other hand we are of course able to notice the sensory effects of blinks if we focus our attention on the accompanying brightness change.
Another analogy is the experience of riding a horse: although at any one particular instant the actual tactile neural input might momentarily be the same when you are riding a horse as when you are on a camel or a motorcycle (all require sitting astraddle and holding on to something in the front), one of the reasons you have the experience of riding a horse now, rather than a camel or a motorcycle, is because of the particular way your contact with the horse would change if you were to move your legs, shift your weight, squeeze your legs (but you needn't actually do it).
In fact it is precisely this counterintuitive notion that enables us to escape from the problem, mentioned in the introduction, of finding a scientific account of how experience might be neurally generated. This problem arose because it was not clear how any physical mechanism could take activation in some brain area and generate "feeling".
The view taken here, according to which sensorimotor contingencies have only an indirect relation to brain activation, is the key to getting around the problem. Indeed, on our view, visual experience does not "occur" in the brain (see also the discussion in 2.3 on awareness). Rather, visual experience is a way of probing the environment. It is a method. It is a thing people (or other organisms) do. Alternatively it could also be said that seeing is a form of knowledge, or an ability, since, according to the theory, visual experience occurs when the brain in some sense "knows" what will happen to its sensory inputs if it were to undertake various actions -- but it need not actually do so. Thus, seen from this point of view, there is no ongoing neural mechanism that "generates" experience. We shall explain in Section 7.3 why it nevertheless seems to many people that vision constitutes an ongoing, occurrent experience.
We shall see that this approach, in which vision is considered to be a law-governed mode of encounter with the environment opens up new ways of thinking about phenomena such as synesthesia, the facial vision of the blind, and in particular for tactile visual sensory substitution, where apparently visual experience can be obtained through arrays of vibrators on the skin.
MacKay suggests that seeing a bottle is an analogous state of affairs[10]. You have the impression of seeing a bottle, if there is knowledge in your nervous system concerning a certain web of contingencies. For example, you have knowledge of the fact that if you move your eyes upwards towards the neck of the bottle, the sensory stimulation will change in a way typical of what happens when a narrower region of the bottle comes into foveal vision; you have knowledge expressing the fact that if you move your eyes downwards, the sensory stimulation will change in a way typical of what happens when the white label is fixated by central vision. Similarly, motions of an object created by manual manipulation can be part of what visually differentiates objects from one another. Unlike a bottle, an object like a pitcher with a handle can be rotated and the handle made to appear and disappear behind the body of the pitcher. It is the possibility of doing this, which is indicative of the fact that this is a pitcher and not a bottle. The visual nature of pitchers involves the knowledge that there are things that can be done to them which make a protrusion (the handle) appear and disappear.
Ryle [1949/1990] has made similar points. He says (p. 218), of a person contemplating a thimble: "Knowing how thimbles look, he is ready to anticipate, though he need not actually anticipate, how it will look, if he approaches it, or moves away from it; and when, without having executed any such anticipations, he does approach it, or move away from it, it looks as he was prepared for it to look. When the actual glimpses of it that he gets are got according to the thimble recipe, they satisfy his acquired expectation-propensities; and this is his espying the thimble."
Other authors have, over the last decades, emitted similar views. Hochberg [1970] (p. 323), for example, in the context of his notion of schematic maps, refers to: "The program of possible samplings of an extended scene, and of contingent expectancies of what will be seen as a result of those samplings..."., and Sperry [1952] has the notion of "implicit preparation to respond". These ideas are also related to Neisser's [1976] perceptual cycle, to Noton & Stark's [1971] "scanpath" theory, and was also put forward in O'Regan [1992] in relation to the notion of the "world as an outside memory". Although, as noted by Wagemans & de Weert [1992], Gibson's notion of "affordance" (Gibson [1982], Turvey, Shaw, Reed, & Mace [1981], Kelso & Kay [1987]), is sometimes considered "mystical", it is undoubtedly strongly related to the present approach (on this see Noë [submitted-b]). Similar notions have also been found useful in "active vision" robotics (Ballard, Hayhoe, Pook, & Rao [1997]; Brooks [1987; 1991]).
All these views of what it is to see, and particularly MacKay's and Ryle's, are based on the same notion of sensorimotor contingency that is so central to the view we are proposing in the present article. In particular, MacKay's work was the main source of inspiration of our theory. However it should be emphasized that our view contains several novel elements not to be found in the works of these authors.
The first point we have stressed is that there is an important distinction to be made between the two classes of sensorimotor contingencies, those which are particular to the visual apparatus, and those which are particular to the way objects occupy three-dimensional space and present themselves to the eye. This distinction enables us to explain the qualitative differences between the different sensory modalities, as we have emphasized above. In addition, as will be explained in Section 6.5, the distinction between the different kinds of sensorimotor contingencies also provides insight into the function of the so-called "when" and "where" pathways that neuroscientists distinguish in the visual system.
A second innovative point in our approach is the realization that it is precisely by adopting the view that the experience of vision is a mode of exploring the environment, rather than a brain activity, that gives us the key to the "explanatory gap" puzzle: With brain activity no longer a prerequisite for vision, we escape having to postulate magical mechanisms to instill "experience" into the brain.
A final, related, point concerning our approach is the fact that it enables us to make inroads on the problem of understanding visual consciousness by separating, as we have done, aspects of vision which involve attunement to sensorimotor laws, from aspects of a different nature, namely aspects which involve the use of these laws in controlling ongoing action-guiding, rational and linguistic behavior. The first two, sensorimotor attunement aspects, are aspects which could, in principle, be shared by inanimate machine-vision devices. One might speculate, however, that only living beings with social interactions, and who therefore can enjoy a much richer, more flexible, variety of behaviors, can manifest the third aspect of vision.
Indeed there is no "re"-presentation of the world inside the brain: the only pictorial or 3D version required is the real outside version. What is required however are methods for probing the outside world -- and visual perception constitutes one mode via which it can be probed. The experience of seeing occurs when the outside world is being probed according to the visual mode, that is, when the knowledge being accumulated is of the three kinds described above, that are typical of the visual modality.
Thus, as argued in O'Regan [1992], it could be said that the outside world acts as an external memory that can be probed at will by the sensory apparatus.
To further clarify this, it is useful to make the relation with normal memory. You know many things about where you live. But as you sit in your office, you may not be thinking about them. If you should start doing so, you can conjure up in your mind all manner of things. Each thing can be thought about in detail, but meanwhile, the other things, though latent, are not being thought about. As you think about your kitchen, your bedroom is not in your mind, though you can cause it to come to mind by merely thinking about it. Remembering is casting one's awareness onto parts of latent memories.
Similarly, seeing is casting one's awareness onto aspects of the outside world made available by the visual apparatus. As you look at a visual scene, you can interrogate yourself about different aspects of the scene. As soon as you do so, each thing you ask yourself about springs into awareness, and is perceived -- not because it enters into a cortical representation, but because knowledge is now available about how sensations will change when you move your eyes, or move the object. However, before you actually wonder about some aspect of the scene, although the information is "out there", and although you know you can obtain it by making the appropriate eye movement or attention shift, it is not currently available. It is not currently available for being visually "chewed upon" or "manipulated", and cannot at this moment be used to control judgments and utterances: the third, "awareness" aspect of seeing is missing. Thus, even though the image of the object is impinging on your retina, and even though its aspects may be being analyzed by the feature-extracting modules of your visual system, under the current theory of seeing we must say that the object is not actually being seen.
As will be described in Section 5, this way of thinking about vision brings with it a number of consequences about some classic problems related to the apparent stability of the visual world despite eye movements, and to the problem of "filling-in" or compensating for "imperfections" of the visual apparatus such as the blind spot. It also provided the impetus for the change-blindness experiments described in Section 5.10.
As pointed out by Noë, Pessoa, & Thompson [2000] and Noë [submitted-a] this paradox is actually only apparent, and rests on a misunderstanding of what seeing really is. It is true that normal perceivers take themselves to be aware of a detailed environment. But what this means is that they perceive the environment surrounding them as detailed. It does not mean that they think that inside their brains there is a detailed copy of the environment. It is only those perceivers -- and there are many scientists among them -- who make the mistake of thinking that "seeing" consists of making such a copy, who are led to think there is a problem.
Another way of understanding why our visual phenomenology is of seeing everything in front of us, derives from the fact that since the slightest flick of the eye or attention allows any part of a visual scene to be processed at will, we have the feeling of immediate availability about the whole scene. In other words, despite the fact that we are only currently processing a small portion of the scene, under the present definition of seeing, we really are seeing the whole scene.
Suppose you should ask yourself, "Am I currently consciously seeing everything there is to see in the scene?" How could you check that you were seeing everything? You would check by casting your attention on each element of the scene, and check that you have the impression of consciously seeing it. But obviously as soon as you do cast your attention on something, you see it. Conclusion, you will always have the impression of consciously seeing everything, since everything you check on, you see. There is an interesting and unfortunate consequence of this: If for some reason you should not be able to mentally attend to some aspect of the scene, you will not be able to consciously see it. Some examples of this are given in Section 5.10-5.12 on empirical evidence.
It is easy to be misled by this argument, which for some reason is peculiarly compelling. Perhaps the fact that we can imagine things in great pictorial detail even in the absence of external stimulation makes it even more urgent to postulate some almost tangible internal picture. But clearly, be it for dreams or for normal vision of the present environment, the argument is another instance of the error of thinking that when we see things as picture-like (be it when we look at reality or when we have a dream), this must be because there is some kind of internal picture. But this is as misguided as the supposition that to see red, there must be red neurons in the brain. The supposed fact that things appear pictorial to us in no way requires there to be pictures in the head. Therefore, the fact that we dream and imagine does not provide evidence in favor of the view that brain contains pictures of the detailed environment[13].
A corollary of the confusion about dreams and mental imagery is the idea, expressed by a number of authors (e.g. Zeki [1993], Kosslyn [1994], Farah [1989]) that feedback from higher brain areas into the retinotopic cortical map of area V1 would be a good way of creating mental imagery. This argument is somewhat misleading. It could be taken to be based on the implicit assumption that mental imagery occurs because of activation in V1: the topographic, metric layout of V1 would make it a good candidate for the cortical areas that possess what Zeki [1993] has called an "experiential" quality -- i.e. the capacity to generate experience. But again, the metric quality of V1 cannot in any way be the cause for the metric quality of our experience. It is as though in order to generate letters on one's screen, the computer had to have little letters floating around in its electronics somewhere.
But because highly familiar stimuli (like words or animals) are used in these experiments, observers may be making use of a few distinctive features available in the images in order to accomplish the task. As argued by Neisser [1976] it probably cannot be said that observers are "seeing" the pictures in the normal sense of the word. As an illustration, consider an experiment we performed in which observers were asked to learn to distinguish three previously unknown symbols resembling Chinese characters (Nazir & O'Regan [1990]). These were presented under the control of a computer linked to an eye movement measuring device. In one experiment, conditions were arranged so that observers could contemplate each Chinese symbol with their eyes fixated at the middle of the symbol, but as soon as the eyes moved, the symbol would disappear. Observers found this procedure extremely disrupting and irritating, and, contrary to what happens when the eye is free to move, hundreds of trials were necessary before they were able to distinguish the symbols. Furthermore, once the task was learnt, observers often failed when asked to recognize the learnt patterns at a new retinal location, only as little as half a degree away from the learnt position. Schlingensiepen, Campbell, Legge, & Walker [1986] also found that without eye movements, observers had difficulty distinguishing patterns composed of arrays of random black and white squares, and Atkinson, Campbell, & Francis [1976] showed by using an after-image technique that it is impossible to count more than four dots that are fixed with respect to the retina: a rather surprising fact. In a task of counting assemblies of debris-like pixel clumps, Kowler & Steinman [1977] found that observers had difficulties when eye movements were not permitted[14]. Because the stimuli used in these experiments were well above the acuity limit, the results are not explicable by acuity drop-off in peripheral vision. Even though a portion of the results may be due to lateral interaction effects (e.g. Toet & Levi [1992]), it seems clear that observers are not at ease when doing a recognition task when eye movements are prohibited. It is like tactually trying to recognize an object lain on your hand without manipulating it.
A further suggestion of the need for visual exploration concerns the phenomenon of fading that occurs when the retinal image is immobilized artificially by use of an optical stabilization device. Under these circumstances a variety of perceptual phenomena occur, ranging from loss of contrast, to fragmentation, to the visual field becoming gray or "blacker than black" (Ditchburn [1973]; Gerrits [1967]). A portion of these phenomena can undoubtedly be accounted for in terms of the temporal response of the first stages of the visual system. Kelly [1982] for example has suggested that detectors sensitive to oriented lines such as those discovered by Hubel and Wiesel actually are silent unless the oriented line stimulation is temporally modulated. Certainly the response of the retina to static stimulation is weak, and temporal modulation is necessary for optimal response (Arend [1973]; Krauskopf [1963]; Laming [1986; 1988]; Kelly [1981]; Gerrits [1978]).
From the point of view of the present theory, these phenomena are compatible with the idea that sensing of the visual world is a dynamic probing process. It could be that even the presence of a static external stimulus is not registered by a static sensory input, but by the dynamic pattern of the inputs that would potentially be produced by changes in the sensor position.
On the other hand, closer to normal seeing, consider an object which is partially occluded by another object. As you move your head, previously occluded parts appear, and previously unoccluded parts may disappear behind the occluder. This ability to make parts of the occluded object appear and disappear is similar to the ability to make objects appear and disappear by blinking, or to make their retinal projections change by moving the eye towards and away from them. This kind of ability is typical of what it is to see, so, even though the object is partially occluded, you nevertheless have the impression of seeing it, or at least "almost" seeing it. Furthermore, if you suddenly close your eyes and ask yourself exactly how much of the object was actually visible just before you closed your eyes, you will not generally know, and indeed, as suggested by results of Intraub & Richardson [1989], you will generally think you saw more than you did (see Figure 2). This demonstrates that seeing is not directly related to having a retinal image, but to being able to manipulate the retinal image.
Figure 2. Subjects tend to remember having seen a greater expanse of a scene than was shown in a photograph. For example, when drawing the close-up view in Panel A from memory, the subject's drawing (Panel C) contained extended boundaries. Another subject, shown a more wide-angle view of the same scene (Panel B), also drew the scene with extended boundaries (Panel D). (Note: to evaluate the drawings in the figure, it is important to study the boundaries of each drawing and its associated stimulus). (Figure and caption from Intraub, http://www.udel.edu/psych/fingerle/intraub.htm)
In understanding the epistemological role of the present theory, an analogy can be made with the situation facing physicists in the 19th Century, who were trying to invent mechanisms by which gravitational or electrical forces could act instantaneously at a distance. To solve this problem, Faraday's idea of a field of force was, according to Einstein, the single most important advance in physics since Newton (c.f. Balibar [1992]). But in fact the idea of a field of force is not a theory at all, it is just a new way of defining what is meant by force. It is a way of abandoning the problem being posed, rather than solving it. Einstein's abandoning the ether hypothesis is another example of how advances can be made by simply reformulating the questions one allows oneself to pose.
In the experiments to be described below, a first group relate to the notion that there is no picture-like internal representation of the outside world and that the world serves as an outside memory. These studies concern the problem of the apparent stability of the visual world despite eye movements, the filling-in of the blind spot and other (supposed) visual defects, and "change blindness": the fact that large changes in a visual scene sometimes go unnoticed. The second group of studies is more related to the idea that visual experience only occurs when there is the potential for action. These studies concern sensorimotor adaptation, sensory substitution, and synesthesia-related effects.
These results are not surprising when considered from the point of view of the theory of vision presented here. From this viewpoint there is no need to postulate a mechanism that re-positions the retinal image after eye saccades so that the world appears stationary, because what is meant by "stationary", is precisely one particular kind of sensory change that occurs when the eye moves across an object. Having the feeling of seeing a stationary object consists in the knowledge that if you were to move your eye slightly leftwards, the object would shift one way on your retina, but if you were to move your eye rightwards, the object would shift the other way. The knowledge of all such potential movements and their results constitute the perception of stationarity. If on actually moving the eyes there were no corresponding retinal motion, the percept would not be of stationarity. From this point of view, there is no need to construct a stationary internal "image" of an object in order to see it stationary. If there is such a thing as an internal signal in the brain that signals the eye's instantaneous position, then its purpose could not be to construct such an internal image (for there would be no one to look at it).
The question nevertheless arises of how the brain is able to accurately judge whether an object is stationary, or to control visuomanual coordination. If there is no way for retinal and extraretinal information to be combined to yield the true spatial coordinates of an object, how can the motion of an object ever be accurately ascertained, or how can an object be located with respect to the body and grasped? A possible answer may be that, whereas there is no extraretinal signal, there is nevertheless extraretinal information about the eye's location or velocity in the orbit. This information could be present in distributed form, and confounded with information about retinal stimulation. Such a distributed, representation that mixes sensory and motor information (both of a static kind -- position -- and of a dynamic kind -- velocity, acceleration) could provide the knowledge about sensorimotor contingencies required in the present theory. It could be used to perform accurate localization, but would not require the existence of a metric-preserving representation of the eye's position, or a picture-like internal image of objects on the retina or in space. Perhaps the multisensory neurons observed in parietal cortex, whose responses may be modulated by imminent eye movements are compatible with this idea (Colby, Duhamel, & Goldberg [1996]; Duhamel, Colby, & Goldberg [1992]; see also Zipser & Anderson [1988]). Also of interest with respect to these ideas is a model of visual localization despite eye movements that has been constructed by Pouget & Sejnowski [1997]. The model uses basis functions to code nonlinear mixtures of retinal and eye position. Linear combinations of these basis functions can provide pure retinal position, pure eye position, or head-centered coordinates of a target, despite the fact that no coherent internal map of the visual field has been constructed.
This conclusion is consistent with the theory presented here, where the problem of visual stability is a non-problem. Seeing does not require compensating for the effects produced by eye shifts in order to ensure accurate accumulation of partial views into a composite patchwork projected on some internal screen. There is no need to re-create another world inside the head in order for it to be seen. Instead, as suggested in Section 4, the outside world acts as an "external memory" store, where information is available for probing by means of eye movements and shifts of attention O'Regan [1992][16].
The empirical evidence showing diminished sensitivity to flashes during saccades cannot be denied, and the origin of this effect has been estimated by Li & Matin [1997] to be 20% due to the retinal smearing and masking caused by the image displacement (there may also be mechanical effects, as suggested by Richards [1969]), and 80% due to central inhibitory mechanisms (some portion of this may be due to spatial uncertainty caused by the new eye position, c.f. Greenhouse & Cohn [1991]).
The important point however is that whatever inhibitory effects are occurring during saccades, these certainly do not constitute a suppression mechanism designed to prevent perception of the saccadic smear. If they did, then why would we not perceive a dimming of the world during saccades? Would we have to postulate a further un-dimming mechanism to compensate for the dimming? The notion of saccadic suppression probably constitutes another instance of the homunculus error, and is no less naive than postulating the need for a mechanism to right the upside-down retinal image so that the world appears right-side up. As explained in the theory presented above, there is no need to postulate mechanisms that compensate for the smear that is created by eye saccades, because this smear is part of what it is to see. If the retinal receptors did not signal a global smear during saccades, then the brain would have to assume that the observer was not seeing, and that he or she was perhaps hallucinating or dreaming.
Related problems involve understanding the apparent filling in of brightness or color that occurs in phenomena such as the Craik-O'Brian-Cornsweet effect and neon color spreading; the apparent generation of illusory contours as in the Kanisza triangle; and other phenomena of modal or amodal completion (c.f. reviews of Kingdom & Moulden [1992]; Pessoa et al. [1998]).
Taking the case of the blind spot, from the point of view of the present theory, and in agreement with analyses of a number of theoreticians (Todorovic [1987]; Kingdom & Moulden [1992]; Pessoa et al. [1998]) there is no need for there to be any filling in mechanism (O'Regan [1992]). On the contrary, the blind spot can be used in order to see: if retinal sensation were not to change dramatically when an object falls into the blind spot, then the brain would have to conclude that the object was not being seen, but was being hallucinated. Suppose you explore your face with your hand: you can put your hand in such a way that your nose falls between two fingers. This does not give you the haptic impression of having no more nose. On the contrary, being able to put the nose between two fingers gives information about the size and nature of a nose. It is part of haptically perceiving the nose.
Monitoring the way the sensory stimulation from the retina changes when the eye moves to displace an object in the vicinity of the blind spot, is, for the brain, another way of gaining information about the object.
One can argue however that even though there may be no need for filling in processes, such filling in processes may nevertheless actually exist. In support of this, Pessoa et al. [1998] though critical of some neurophysiological and behavioral studies purporting to be evidence for filling in, concluded that several studies do point to the existence of precisely the kind of mechanisms which would be required for a filling in process. For example, Paradiso & Nakayama [1991], by using a masking paradigm, were able to measure the temporal dynamics of the phenomenal filling in of the inside of a bright disk. De Weerd, Gattass, Desimone, & Ungerleider [1995] found cells in extrastriate cortex whose responses correlate well with the time it takes holes in textures presented in peripheral vision to perceptually fill in.
Just as was the case for the problem of the extraretinal signal or of saccadic suppression, the theory being advocated here does not deny the existence of neural mechanisms that underlie the perceptual phenomena that each of us observe. There can be no doubt that something is going on in the brain which is in relation to the fact that observers have no experience of a blind spot, and which makes Kanisza triangles have illusory contours. The question is: Is whatever is going on, actually serving to create an internal copy of the outside world, which has the metric properties of a picture, and which has to be completed in order for observers to have the phenomenology of a perfect scene? In the example of Paradiso & Nakayama's data for example, there can be no denying that there must be retinal or cortical processes that involve some kind of dynamic spreading activation and inhibition, and that these processes underlie the percept that observers have in their paradigm and possibly also when a disk is presented under normal conditions. But even though these processes act like filling in processes, this does not mean that they are actually used by the brain to fill in an internal metric picture of the world. They may just be providing information to the brain about the nature of the stimulation, but without this information being used to create a picture-like representation of the world.
In other words, our objection is not to the mechanisms themselves, whose existence we would not deny, but to the characterization of these mechanisms as involving "filling in". Consider this caricature: Spatio-temporal integration in the low level visual system is a mechanism which explains much phenomenology (e.g. why fast flickering lights appear continuous and very closely spaced dots look like lines). But surely no one would want to claim that the purpose of spatiotemporal integration is to "fill in" the temporal gaps in what would otherwise look like a stroboscopic world, or to make dotted lines look continuous. Spatiotemporal integration is a mechanism used in our visual systems to sample the environment, but its purpose is not to compensate for gaps in what would otherwise be a granular, pixel-like internal picture.
In additional to this non-homogeneity in spatial sampling, the retina also suffers from a non-homogeneity in the way it processes color: whereas in the macular region, the presence of three photoreceptor cone classes permits color discrimination, in peripheral retina the cones become very sparse (Anderson, Mullen, & Hess [1991]; Marcos, Navarro, & Artal [1996]; Coletta & Williams [1987]). The lack of the ability to accurately locate colors can easily be demonstrated by attempting to report the order of the colors of four or five previously unseen colored pencils when these are brought in from peripheral vision to a position just a few degrees to the side of one's fixation point.
A further, surprising non-homogeneity derives from the macular pigment, a yellowish jelly that covers the macula, that absorbs up to 50% of the light in the short wavelength range (Bone, Landrum, & Cains [1992]), thereby profoundly altering color sensitivity in central vision.
Despite these non-homogeneities, the perception of spatial detail and color does not subjectively appear non-uniform to us: most people are completely unaware of how poor their acuity and their color perception are in peripheral vision. Analogously to the filling-in mechanism that is sometimes assumed to fill in the blind spot, one might be tempted to postulate some kind of compensation mechanism that would account for the perceived uniformity of the visual field, However, from the point of view of the present theory of visual experience, such compensation is unnecessary. This will be illustrated in relation to color perception below.
Another example arises from the perception of color in dichromats. When carefully tested in controlled conditions of illumination, dichromats exhibit deficiencies in their ability to distinguish colors, generally along the red-green dimension, which can be accounted for by assuming that they lack a particular type of cone, generally either the long or medium wavelength type. Curiously however, in real-life situations, dichromats are often rather good at making red-green distinctions. As suggested by Jameson & Hurvich [1978] (c.f. also Lillo, Davies, Collado, Ponte, & Vitini [1998]; Montag [1994]) this is undoubtedly because they can make use of additional cues deriving from what they know about objects and what they can sense concerning ambient lighting. Thus for example, when a surface is moved so that it reflects more yellowish sunlight and less bluish light from the sky, the particular way the spectrum of the reflected light changes, disambiguates the surface's color, and allows that color to be ascertained correctly even when the observer is a dichromat.
Though it is not surprising to find observers using all sorts of available cues to help them in their color discriminations, this kind of finding can be taken to support a much more far-reaching, fundamental hypothesis, put forward by Broackes [1992]. This is that the perceived color of a surface is not so much related to the spectrum of the reflected light, but rather, to the way the surface changes the light.
It must be stressed that more is being said here than was said by Jameson & Hurvich, who merely noted that information is available that allows dichromats to make judgments similar to trichromats. Broackes' idea is that the color experience provided by surfaces are exactly the laws governing the way the surface changes the reflected light[18]. The same laws apply to dichromats and trichromats, so they have the same kinds of color perception: the difference is that dichromats have fewer clues to go by in many situations. Thus Broackes, who has color vision deficiencies[19] himself, claims that he has different experiences for red and green as do normals. His only problem is that sometimes, when lighting conditions are special, he can see certain dark red things as dark green, just as sometimes, in shadow, people with normal vision are convinced a garment is dark blue when in fact it is black, or vice versa.
Broackes' theory of color is a special case of the theory of visual perception that we have presented here. In our theory, the qualities of a visual sensation are the laws that govern the way the sensation change when we act in different ways upon the environment.
Whereas exactly this kind of experiment has not yet been done, a variety of related manipulations were performed by McCollough [1965b] and by Kohler [1951][20]. For example, Kohler had observers wear spectacles in which one half of the visual field was tinted with blue, and the other half tinted with yellow. This is similar to the proposed armchair experiment in the sense that perceived color will be different depending on which way the observer moves the eyes. Results of the experiment seem to show that after adaptation, observers apparently came to see colors "normally". Similar phenomena were observed with half-prisms, in which the top and bottom portion of the visual field were shifted by several degrees with respect to each other. Observers ultimately adapted, so that manual localization of objects in the upper and lower visual fields was accurate.
Of particular interest in these studies would have been to know whether observers perceived the world as continuous despite the discontinuity imposed by the colored glasses or prisms. However, it is difficult to rigorously evaluate the reports, as they were only described informally by Kohler. Since then, though a large literature has developed over the last decades concerning many forms of perceptual adaptation, not very much work seems to have been done to investigate the effects of modifications like those imposed by the two-color glasses or the half-prisms, which produce strong discontinuities in the visual field.
Nevertheless partial insight into such situations may be obtained by considering people who wear spectacles with bifocal lenses[21]: here a discontinuity exists in the visual field between the upper and lower part of the glasses. Depending on where an observer directs the eyes, the size and focus of objects will be different, because of the different power of the two parts of the lens. The question is then, does the world appear discontinuous to viewers of bifocals? The answer is that the world does not appear discontinuous, any more than the world appears "dirty" to someone who has not wiped his spectacle lenses clean. This is not to say that the observer cannot become aware of the discontinuity or the dirt on the lenses by attending to the appropriate aspect of the stimulation, just as it is possible to become aware of the blind spot in each eye by positioning a stimulus appropriately. But under normal circumstances the wearer of bifocals takes no notice of the discontinuity. Furthermore, even though image magnification as seen through the different parts of the lens are different, thereby modifying perception of distance, manual reaching for objects seen through the different parts of the lenses adapts and becomes accurate, as does the vestibulo-ocular reflex. Gauthier & Robinson [1975] and Gauthier [1976] have for example shown that wearers of normal spectacles with strong corrections, as well as scuba divers come to possess a bistable state of adaptation, whereby their distance perception and reaching can instantaneously switch from one to the other state, as they take their spectacles on and off, or look through their underwater goggles (see also Welch, Bridgeman, Anand, & Browman [1993] for a similar effect with prisms). In fact an observer can be tricked into inappropriately switching adaptation state by surreptitiously removing the lenses from his or her eyeglasses, so that he or she incorrectly expects magnification to change when the eyeglasses are put on (Gauthier, personal communication).
What is interesting about these experiments is that during the course of adaptation, perception of the world is subject to a sort of fragmentation, and to a dependence on context and task. For example, Kohler [1951] reports that visual context allows something that is seen upside-down to be righted (e.g. a candle flips when it is lit because flames must go up, a cup flips when coffee is poured into it, because coffee must pour downwards). Ambiguities and inconsistencies abound: Dolezal reports sometimes being unable to prevent both his hands from moving when he tries to move only one. Kohler reports cases where two adjacent heads, one upright, the other inverted, were both perceived as upright. Observer Grill, after 18 days of wearing reversing spectacles, stands on the sidewalk and correctly sees vehicles driving on the "right", and hears the noise of the car motor coming from the correct direction. On the other hand, Grill nevertheless reports that the license plate numbers appear to be in mirror writing. Other observations are that a "3" is seen as in mirror writing, even though its open and closed sides are correctly localized as being on the left and right respectively. The bicycle bell seems on the unusual side, even though the observer can turn the handle bars in the correct direction. Taylor [1962] has performed a study similar to Kohler's, except that instead of wearing the inverting spectacles continuously, his subject wore them only for a limited period each day. Under these conditions the subject rapidly obtains a bistable form of adaptation, adapted to both wearing and not wearing the spectacles. A point stressed by Taylor, in support of his behaviorist theory[23], is that adaptation is specific to the particular body parts (arms, legs, torso) or activities (standing on both feet, on one foot, on the toes, riding a bicycle) that the subject has had training with, and that there is little "interpenetration" from one such sensorimotor system to another.
A theory of vision in which there is a picture-like internal representation of the outside world would not easily account for the fragmentation of visual perception described in these experiments: for example it would be hard to explain the case of the license plate, where one aspect of a scene appears oriented accurately, and yet another aspect, sharing the same retinal location, appears inverted. On the other hand, the present theory, in which vision is knowledge of sensorimotor transformations, and the ability to act, readily provides an explanation: reading alphabetic characters involves a subspecies of behavior connected with reading, judging laterality involves another, independent, subspecies of behavior, namely reaching. An observer adapting to an inverted world will in the course of adaptation only be able to progressively probe subsets of the sensorimotor contingencies that characterize his or her new visual world, and so inconsistencies and contradictions may easily arise between "islands" of visuo-motor behavior[24].
Particularly interesting are cases of double vision when only one eye is open, i.e. not explicable by diplopia. For example, Kohler's observer Grill saw two points of light when only one was presented slightly to the right of the median line (the second point was seen weaker, on the left, symmetrical to the original point). Similar observations of symmetrical "phantoms" were noticed by Stratton [1897], and can be compared to cases of monocular diplopia reported in strabismus (Ramachandran, Cobb, & Levi [1994b]; Ramachandran, Cobb, & Levi [1994a]; Rozenblium Iu & Korniushina [1991]). Taylor [1962] says of his subject wearing left-right inverting spectacles:
"Anotherof the training procedures he adopted was to walk round and round a chair or table, constantly touching it with his body, and frequently changing direction so as to bring both sides into action. It was during an exercise of this kind, on the eighth day of the experiment, that he had his first experience of perceiving an object in its true position. But it was a very strange experience, in that he perceived the chair as being both on the side where it was in contact with his body and on the opposite side. And by this he meant not just that he knew that the chair he saw on his left was actually on his right. He had that knowledge from the beginning of the experiment. The experience was more like the simultaneous perception of an object and its mirror image, although in this case the chair on the right was rather ghost-like." (p. 201-202)
Presumably what happens in these experiments is that, because the spatial location or orientation of an object with respect to the body can be attributed either with respect to the pre- or the post-adapted frame of reference, during the course of adaptation it can sometimes be seen as being in both. Furthermore, orientation and localization of objects in the field of view can be defined with respect to multiple referents, and within different tasks, and each task may have adapted independently, thereby giving rise to incoherent visual impressions.
The impression we have of seeing a coherent world thus arises through the knitting together of a number of separate sensory and sensory-motor components, making use of visual, vestibular, tactile and proprioceptive information, and in which different behaviors (e.g. reading, grasping, bicycle riding) constitute components that adapt independently, but that each contribute to the experience of seeing. Conclusions of this kind have also been reached in a wealth of research on sensorimotor control, where it is shown that a gesture such as reaching for an object is composed of a number of sub-components (e.g. ballistic extension of the arm, fine control of the final approach and finger grasping, etc.), each of which may obey independent spatial and temporal constraints, and each of which may be controlled by different cerebral subsystems, which adapt separately to perturbations like changes in muscle proprioception, or in vestibular and visual information (for reviews of these results, see Jeannerod [1997]; Rossetti, Koga, & Mano [1993][25]).
However in the experiments we performed, conditions were arranged such that the transient that would normally occur was prevented from playing its attention-grabbing role. This could be done in several ways. One method consisted in superimposing a very brief global flicker over the whole visual field at the moment of the change. This global flicker served to swamp the local transient caused by the change, preventing attention from being attracted to it. A similar purpose could be achieved by making the change coincide with an eye saccade, an eye blink, or a film cut in a film sequence (for reviews, see Simons & Levin [1997][27]). In all these cases a brief global disturbance swamped the local transient and prevented it attracting attention to the location of the change. Another method we used to prevent the local transient from operating in the normal fashion was to create a small number of additional, extraneous transients distributed over the picture, somewhat like mudsplashes on a car windscreen (cf. O'Regan et al. [1999]). These local transients acted as decoys and made it likely that attention would be attracted to an incorrect location instead of going to the true change location.
The results of the experiments showed that in many cases observers have great difficulty seeing changes, even though the changes are very large, and occur in full view -- they are perfectly visible to someone who knows what they are. Such results are surprising if one espouses the view that we should "see" everything that we are looking at: It is very troubling to be shown a picture where a change is occurring repetitively and in full view, without being able to see the change. The experience is quite contradictory with one's subjective impression of richness, of "seeing everything" in the visual field. However, the results are completely coherent with the view of seeing which is being defended here.
Another aspect of these experiments which relates to the present theory is a result which we observed in an experiment in which we measured observers' eye movements as they performed the task O'Regan et al. [2000]. We found that in many cases, observers could be looking directly at the change at the moment the change occurred, and still not see it. Again, under the usual view that one should see what one is looking at, this is surprising. But under the view that what one sees is the aspect of the scene which one is currently "visually manipulating", then it is quite reasonable to observe that only a subset of scene elements that share a particular scene location should at a given moment be perceived.
A striking result of a similar nature had been observed by Haines [1991] and Fisher, Haines, & Price [1980], who had professional pilots land an aircraft in a flight simulator under conditions of poor visibility, and using a head-up display (or "HUD") -- that is, a display which superimposed flight guidance and control information on the windshield. On various occasions during the pilot's landing approach, they were presented with unexpected "critical" information in the form of a large jet airplane located directly ahead of them on the runway. Although the jet airplane was perfectly visible despite the head-up display (see Figure 3), presumably because of the extreme improbability of such an occurrence, and because the pilots were concentrating on the head-up display or the landing maneuver, two of the eight experienced commercial pilots simply did not see the obstacle on the two occasions they were confronted with it, and simply landed their own aircraft through the obstacle. On later being confronted with a video of what had happened, they were incredulous[28].
Fig. 3. Simulator pilot's forward visual scene at an altitude of
72 feet and 131 knots with runway obstruction clearly visible. From Haines
[1991]
Other results showing that people can be looking directly at something and not see it had previously been obtained by Neisser & Becklen [1975], who used a situation which was a visual analogue of the "cocktail party" situation, where party-goers are able to attend to one of many superimposed voices. In their visual analogue, Neisser & Becklen visually superimposed two independent film sequences, and demonstrated that observers were able to single out and follow one of the sequences, while being oblivious of the other. Simons & Chabris [in press] have recently replicated and extended these effects.
Finally Mack & Rock [1998] and Mack, Tang, Tuma, Kahn, & Rock [1992] have done a number of experiments using their paradigm of "inattentional blindness". In this, subjects will be engaged in an attention-intensive task such as determining which arm of a cross is longer. After a number of trials, an unexpected, perfectly visible, additional stimulus will appear near the cross. The authors observe that on many occasions this extraneous stimulus is simply not noticed.
Wolfe et al. [1999] use a standard visual search paradigm in which a subject must search for a target symbol among a number of distractor symbols. The authors estimate the efficiency of the search in milliseconds per item searched. However, instead of using a new display of distractors on each trial as is usually done, the authors use exactly the same visual display over a number of repetitions, but each time change the target that the subject is looking for. Since subjects are looking at the same display, which remains continuously visible on the screen for anything from 5 to 350 repetitions, depending on the experiment, one might have expected that an internal representation of the display would have time to build up, allowing search rate to improve over repetitions. However this is not what is found: Over a number of experiments using different kinds of stimuli, Wolfe et al. [1999] find no evidence of improvement in search rate. It seems that no internal representation of the display is being built up over repetitions. In fact, search rate is as bad after many repeated searches as in the normal visual search conditions when the display changes at every trial: in other words, it is as though the subjects think they are searching through a brand new display at each trial, even though it is exactly the same display as before. Furthermore, an experiment done where the display is memorized and not visually presented at all, actually shows faster search speeds than when the display is present.
The results of these experiments are surprising under the view that what we see consists of an internal, more or less picture-like, representation of the visual world. However they are exactly what would be expected under the present view, according to which "seeing" consists, not of having a "picture" in the mind, but of having seeking-out-routines that allow information to be obtained from the environment. Thus, observers generally do not bother to recreate within their minds a "re-"presentation of the outside world, because the outside world itself can serve as a memory for immediate probing. Indeed, the last result showing faster performance in the pure memory search shows that the very presence of a visual stimulus may actually obligatorily cause observers to make use of the world in the "outside memory" mode, even though it is less efficient than using "normal" memory.
This way of interpreting the results is also in broad agreement with Wolfe's point of view (Wolfe [1997; 1999]) -- Wolfe also refers to the notion of "outside memory". However Wolfe lays additional emphasis on the role of attention in his experiments: Following the approach of Kahneman, Treisman, & Gibbs [1992] adopted by many workers in the attention literature, Wolfe believes that before attention is brought to bear on a particular region of the visual field, the elementary features (such as line segments, color patches, texture elements) analyzed automatically by low-level modules in the visual system constitute a sort of "primeval soup" or undifferentiated visual "stuff". Only once attention is applied to a particular spatial location, can the features be bound together so that an object (or recognizable visual entity) is perceived at that location. Wolfe's interesting proposition is now that when visual attention subsequently moves on to another location, the previously bound-together visual entities disagregate again and fall back into the "primeval soup": the previously perceived entity is no longer seen. This idea prompts Wolfe to use the term "inattentional amnesia", to emphasize the fact that after attention has moved on, nothing is left to see.
The status of the notion of attention in this explanation, and its relation to the theory presented here, is not entirely clear. One possibility would be to assume that what Wolfe means by "attention" is nothing other than visual awareness. In that case the result of the experiment could be summarized by saying "once your awareness has moved off a part of the scene, you are no longer aware of it"... which is tautological. Presumably therefore what Wolfe means by attention is something independent of awareness: there would be forms of attention without awareness and forms of awareness without attention. It is clear that further thought is needed to clarify these questions.
Independently of the framework within which one places oneself, it remains an interesting question to ask: What does the primeval soup "look like"? In other words, what does the visual field look like when the observer is not attending to anything in particular in it? Our preference would be to take the strict sense of attention in which attention = awareness, and to say that without attending to something (i.e. without being aware of anything), by definition the visual field cannot look like anything at all. Only when the observer attends to something will he or she be aware of seeing it. Note that what the observer attends to can be something as basic as overall brightness or color, or something like the variability in these ("colorfulness"?, "texturedness"?), or some attribute like "verticality" or "blobiness". If such features constitute the "primeval soup", then, like normal targets in the search task, the primeval soup would also only be "seen" if it was being attended to.
Proofreading is notoriously difficult: when you look at words, you are processing words, not the letters that compose them. If there is an extra, incorrect letter in a word, it will have been processed by your low-level vision modules, but it will not have been "seen". Thus, for example, you will probably not have noticed that the "a"s in the last sentences were of a different shape than elsewhere[29]. Nonetheless on several occasions you were undoubtedly looking directly at them. It may take you a while to realize that the sign below (Fig. 4) does not say: The illusion of "seeing".[30] You may be furious to find confirmation of years of the scientific study of reading showing that in this sentence there are in fact more "f"s than you think (count them!).[31]
Fig. 4.Ceci n'est pas: The illusion of "seeing".
The phenomena of figure-ground competition (see Figure 5) and of ambiguous
figures are also striking examples of how you do not see everything that
you could see: when looking at such stimuli, you only see one of the possible
configurations, even though more than one may be simultaneously available
at the same location in your visual field.
Fig. 5. Figure-ground competition.
It sometimes occurs that as you walk in the street you look directly at someone without seeing them. Only when the person gesticulates or manifests their irritation at not being recognized, do you become aware of who they are. While driving; it sometimes happens that you realize that you have been looking for a while at the brake lights of the car ahead of you without pressing on the brake.
A number of devices have been devised to allow people with deficits in one sensory modality to use another modality to gain information. In the domain of vision, two main classes of such sensory substitution devices have been constructed: echolocation devices and tactile visual substitution devices.
Echolocation devices provide auditory signals which depend on the direction, distance, size, and surface texture of nearby objects, but they provide no detailed shape information. Nevertheless such devices have been extensively studied as prostheses for the blind, both in neonates (Sampaio & Dufier [1988]; Sampaio [1989]; Bower [1977]) and in adults (Ifukube, Sasaki, & Peng [1991]). It is clear that while such devices obviously cannot provide visual experience, they nevertheless provide users with the clear impression of things being "out in front of them".
Particularly interesting is the work being done by Lenay [1997], using an extreme simplification of the echolocation device, in which a blind or blindfolded person has a single photoelectric sensor attached to his or her forefinger, and can scan a simple environment (e.g. consisting of several isolated light sources) by pointing. Every time the photosensor points directly at a light source, the subject hears a beep or feels a vibration. Depending on whether the finger is moved laterally, or in an arc, the subject establishes different types of sensorimotor contingencies: lateral movement allows information about direction to be obtained, movement in an arc centered on the object gives information about depth. Note several interesting facts. First, users of such a device rapidly say that they do not notice vibrations on their skin or hear sounds, rather they "sense" the presence of objects outside of them. Note also that at a given moment during exploration of the environment, subjects may be receiving no beep or vibration whatever, and yet "feel" the presence of an object before them. In other words the experience of perception derives from the potential to obtain changes in sensation, not from the sensations themselves. Note also that the exact nature or body location of the stimulation (beep or vibration) has no bearing on perception of the stimulus -- the vibration can be applied on the finger or anywhere else on the body. This again shows that what is important is the sensorimotor invariance structure of the changes in sensation, not the sensation itself.
Lenay's very simple setup provides a concrete example of what is meant by the laws of sensorimotor contingency. Suppose that the photosensor were mounted on the forearm of an articulated arm, with the arm making an angle [[alpha]] with the torso, and the forearm making an angle [[beta]] with the arm, as shown in the Figure 6. Then we can define the sensorimotor manifold as the two-dimensional space [[alpha]] : [0, [[pi]]/2] and [[beta]] : ]3[[pi]]/2-a, 2[[pi]][. Consider the situation where we are obtaining information about depth by making movement in an arc. If a luminous source at distance L is being "fixated", the angles [[alpha]] and [[beta]] will lie on orbits in the sensorimotor sensorimotor manifold defined by the relation shown in the lower part of the figure. In reality of course the angles [[alpha]] and [[beta]] will be nonlinear functions of high-dimensional neural population vectors corresponding to arm and forearm muscle parameters. But the laws of contingency will be the same.
The arm (with the forearm) has a length of 1. The distance from the target, L (0S), can then be obtained by a trigonometrical relation, according to the following formula: (1) L = sin [[alpha]] - cos [[alpha]] tan([[alpha]]+[[beta]]), where [[alpha]] : [0, [[pi]]/2] and [[beta]] : ]3[[pi]]/2-a, 2[[pi]][
Curve representing angle [[beta]] in relation to angle [[alpha]] (both expressed in radians) for the following values of L = 0,1,...,7. [[alpha]] varies from 0 to [[pi]]/2. According to (1) one can determine [[beta]] for any given L and [[alpha]]: [[beta]] = 2[[pi]] - [[alpha]] + Atan( (sin[[alpha]]-L)/cos[[alpha]] )
Figure 6. Figure from Lenay, Canu, & Villon [1997] showing
the sensorimotor contingency orbits for a simple photocell mounted on an
arm and a forearm, in the case where the photocell is continuously fixating
a luminous source at distance L.
On further reflection it is apparent that the simple device studied by Lenay is an electronic variant of the blind person's cane. Blind persons using a cane do not sense the cane, but the environment outside of them that they are exploring by means of the cane. It has been said that the tactile sensations provided by the cane are somehow "relocated" or "projected" onto the environment. The cane itself is forgotten or ignored. But this way of describing experience with the cane, though in a way correct, is misleading, since it suggests that sensations themselves originally possessed a location which had to be relocated. The present theory shows that in themselves, sensations are situated nowhere. The location of a sensation (and for that matter any perceived aspect including its moment of occurrence) is an abstraction constructed in order to account for the invariance structure of the available sensorimotor contingencies.
Note that similar experiences to those of the blind person with the cane are experienced every day even by sighted persons: Car drivers "feel" the wheels on the road, and extend the sense of their bodies to include the whole car, allowing them to negotiate into parking spaces with only centimeters to spare. A particularly poignant example of having one's perceived body extend outside of the boundary formed by the skin was given to the first author by a friend who is a talented viola player. Spending most of the day with the viola under his chin, on one occasion he went into the kitchen to drink some hot tea, and some drops fell on the viola. He said he was surprised not to have felt the hot drops on the instrument: it felt anesthetized. Another everyday example of remote tactile sensing occurs when you write on a piece of paper with a pen: you feel the paper at the end of the pen: it is rough, it is smooth, it is soft. You locate the contact at the end of the pen, not on your fingers where the force is actually felt (this example is given by James [1890/1950]).
One might consider these examples as surprising at first sight. But then we ask: should it not also be considered surprising that fingertip sensations are felt on the fingertips, since after all, it is presumably in the brain where the sensations are registered? Why would one not tend to think that one should be able to walk through a door no wider than one's brain, since body sensations presumable arrive in the brain? Indeed, given that visual sensation impregnates the retina, why does one not feel the outside world as situated on one's retina, instead of outside one? These obviously ridiculous extensions of the "relocation" idea discussed above make one realize that actually the perceived location of a sensation logically cannot be determined by where the nerves come from or where they go to. Perceived location is, like other aspects of sensation, an abstraction that the brain has deduced from the structure of the sensorimotor contingencies that govern the sensation[32].
Some very interesting experiments of Tastevin [1937] are related to these points. Tastevin had shown that the sensed identity or position of a limb can be transferred to another limb or to a plaster model of the limb. Thus, for example, when an experimenter feigns to touch a subject's forefinger with one prong of a compass, but actually touches the middle finger with the other prong, the subject feels the touch on the forefinger. Sensation has thus been relocated from the middle finger to the forefinger. Whole body parts can be relocalized by this means. A recent experiment along very similar lines was described by Botvinick & Cohen [1998] (and also extended by Ramachandran & Blakeslee [1998]). These authors used a life-size rubber model of a left arm placed before a subject whose real left arm was hidden by a screen. Using two small brushes, the experimenters synchronously stroked corresponding positions of the rubber and real arm. After ten minutes, subjects came to feel that the rubber arm was their own.
All these phenomena show how labile the perceived location of a stimulation can be, and how it depends on correlation with information from other modalities (in this case vision). Even neural representations of body parts are known to be labile, as has been shown by Iriki, Tanaka, & Iwamura [1996] whose macaque monkeys' bimodal visual somatosensory receptive fields moved from their hands to the ends of a rake they used as a tool. However a facile interpretation of such phenomena in terms of "neural plasticity" of cortical maps would be misleading, since such an interpretation would implicitly assume that perceived location of a stimulus is directly related to activity in cortical maps -- an idea we reject.
Figure 7. A blind subject with a "Tactile Visual Substitution system".
A TV camera (mounted on spectacle frames) sends signals through electronic
circuitry (displayed in right hand) to an array of small vibrators (left
hand) which is strapped against the subject's skin. The pattern of tactile
sitmulation corresponds roughly to a greatly enlarged visual image. (Photograph
courtesy of P. Bach-y-Rita). From Morgan [1977].
A first point concerns the importance of the observer's being able to manipulate the TV camera himself or herself (Bach-y-Rita [1972; 1984]; Sampaio [1995]).
In the earliest trials with the TVSS device, blind subjects generally unsuccessfully attempted to identify objects that were placed in front of the camera, which was fixed. It was only when the observer was allowed to actively manipulate the camera that identification became possible. In fact Bach y Rita recounts (personal communication) how he accidentally discovered this when a blind person, after frustration with attempting to recognize objects statically, grabbed the camera off the tripod and more or less immediately came to 'see'. This important point constitutes an empirical verification of the mainstay of the present theory of visual experience, namely that seeing constitutes the ability to actively modify sensory impressions in certain law-obeying ways.
Once observers have had practice with the TVSS, several further aspects of the experience provided by the system suggest that it is similar to the experience of vision. First, though initially observers locate stimulation on the body part which is stimulated, with practice, although they are still able to feel the local tactile sensation (e.g. if it is painful or if it itches), the observers locate objects in space, and not on the skin. Indeed, after using one skin location (e.g. the back), an observer has no problem transferring to a different skin location (e.g. the forehead).
An interesting example shows that the localization of objects outside the body is not just a cognitive strategy but truly resembles visual experience. In an anecdote reported by Bach y Rita, the zoom control of the camera was accidentally knocked, causing a sudden magnification or "looming" of the tactile image. The surprised observer jumped back, as though he expected the object to collide with him. This occurred even though stimulation was on the observer's back -- a point stressed by early investigators, who seem to have expected the observer to jump forward.
Another interesting observation caused puzzlement in the early investigations with the TVSS. For practical reasons the battery of 400 vibrators mounted on the observer's back consisted in two ramps of 200 vibrators, one on each side of the observer's backbone. A large gap was therefore present in the tactile representation of the visual field. "Curiously" however, no gap was apparent in observers' perceived visual field. This tactile analog of what might incorrectly be called "filling-in" of the retinal blind spot is of course unsurprising in the light of the present theory, where no filling-in mechanism need be postulated (c.f. Section 5.5).
Do blind people actually see with the TVSS? The question has been raised by Bach y Rita who prefers to put the word "see" in quotes. One justification for this, he claims, is the fact that people who have learnt to see with the device are disappointed when shown pictures of their loved ones, or erotic pictures: they have no emotional reaction. Bach y Rita interprets this as a failure of the device to provide true visual experience. But an alternative is to admit that the device does provide true visual experience, but that emotional and sexual reactions are strongly linked to the sensations that are experienced during the period when emotional attachment occurs and sexual interest develops. If, during the course of development, these experiences are initially non-visual, then they will remain non-visual.
Morgan [1977] also discusses the issue of whether people really do see with the TVSS and concludes, that either people really do see with the TVSS, or there can be no scientific psychology. Clearly from the point of view of the present theory, seeing is not something all or none. There are many aspects to seeing, and the TVSS provides some but not all of them. The invariants related to position and size changes of the tactile image are similar to those in normal vision. Color and stereo vision however are absent, and resolution is extremely poor. But just as color blind, stereo blind, one-eyed or low-sighted people can be said to see, people using the TVSS should also be said to see. The fact that stimulation is provided through the skin should be irrelevant, providing the stimulation obeys the required sensorimotor laws. Of course seeing with the skin probably involves laws that are not exactly the same as seeing with the eyes, just as seeing colors in the dark is not quite the same as in the light. The experience associated with the TVSS will thus also be somewhat different from normal visual experience.
As an example, consider the following quote given by James [1890/1950] from the blind author of a treatise on blindness of the time: "Whether within a house or in the open air, whether walking or standing still, I can tell, although quite blind, when I am opposite an object, and can perceive whether it be tall or short, slender or bulky. I can also detect whether it be a solitary object or a continuous fence; whether it be a close fence or composed of open rails, and often whether it be a wooden fence, a brick or stone wall, or a quick-set hedge. ... The currents of air can having nothing to do with this power, as the state of the wind does not directly affect it; the sense of hearing has nothing to do with it, as when snow lies thickly on the ground objects are more distinct, although the footfall cannot be heard. I seem to perceive objects through the skin of my face, and to have the impressions immediately transmitted to the brain. ..." (Vol. 2, p. 204).
At least since Diderot's "Letter on the blind", facial vision had often been considered to truly be a kind of tactile or even possibly an extrasensory form of perception (c.f. historical review by Hayes [1935], cited by Rice [1966]). James [1890/1950] (Vol 2, p. 204; 140) compares the sense to what he believes is a tactile, pressure-related "tympanic sense", that is, the ability we all have of sensing with closed eyes whether an object brought before our face is large or small, or more or less solid. Despite such claims however, James, by stopping up the ears of blind people with putty, demonstrated to his satisfaction that audition was involved in the facial sense. This was then definitively established by Dallenbach, Supa, & Cotzin [1944] and facial vision is now known to be essentially caused by intensity, direction and frequency shifts of reflected sounds (see review by Arias [1996]). Kohler [1967] actually went so far as to anesthetize the faces of blind people, who nevertheless continued to have these sensations.
As noted by Worchel, Mauney, & Andrew [1950] (cited by Strelow & Brabyn [1982]), the question arises of why this form of object perception is experienced as having a tactile feeling rather than an auditory quality. A possibility along behaviorist lines has been suggested by Taylor [1962], who supposes that collisions with obstacles will often involve the face -- the hands may often rise and protect the face. This may create, by association, feelings on the face in the case of impeding collisions. Further correlations (apparently not mentioned by Taylor) might be the fact that objects that are close to the face tend to provoke slight disturbances of the air as well as changes in heat radiation that could be detected by receptors on the face. Although Taylor's associationist hypothesis may have some truth in it, from the point of view of the present theory another possibility arises: the prediction would be that the sensorimotor contingencies created by the particular, very subtle information received through the auditory modality would, in this particular case, have an invariance structure that resembles the contingencies caused by tactile stimuli like those created by a veil upon the face. Indeed it appears conceivable that the object sense, requiring more subtle auditory distinctions, would be much more critically dependent on distance than normal hearing. In particular, moving a few centimeters forward or backwards, might create a radical change analogous to moving a few centimeters forward or backwards and bringing the head into and out of contact with a veil. Similarly it may be that when the head is facing the object that is being sensed, slight sideways shifts of the head might create systematic changes similar to the systematic rubbing that occurs when one is touching a piece of cloth with the head. Note however that it would be exaggerated to take too literally the comparison that blind people make with veils and cobwebs: Kohler has verified that when touched with actual veils the same blind people say that the sensations are actually quite different. Perhaps the inability to specify precisely the nature of the experience produced prompted the author cited by James to say that the impressions were "immediately transmitted to the brain".
The facial sense of the blind may be related to the phenomenon of synesthesia (Cytowic & Wood [1982]; Baron-Cohen & Harrison [1996])[33], where a stimulus in one sensory modality evokes sensations in another, the most frequently occurring case being colored hearing (Marks [1978]). Ventriloquism is another type of example where information from one sensory modality modifies that in another: in "visual capture" or the "ventriloquism effect", the perceived location of a sound source is influenced by its seen position, and, to a lesser extent, vice versa (Radeau & Bertelson [1974]; Warren, Welch, & McCarthy [1981]; Hatwell [1986]). A related phenomenon is the McGurk effect (McGurk & MacDonald [1976]) in which the identity of a heard phoneme is altered by simultaneously observing a visual display of a different phoneme being pronounced. Radeau [1997] (c.f. also Marks [1978]) has reviewed a number of inter-sensory interactions such as these, both in humans and animals, and concludes that such effects are compatible with the notion that the different qualities of the senses are not present ab initio, as Piaget might have claimed, but rather (following Gibson [1966]; Bower [1979]) are the result of a progressive differentiation process that occurs in the developing organism through the influence of environmental experience.
The view taken within the context of the present theory as concerns all such intermodal interactions would be related to this. More precisely however it would say that the experience associated with a modality exist only within the context of the acting organism, and within the context of the other senses available to the organism. Although vision, audition, touch, etc. will have their own specificities due to the particularities of the sensors and sensorimotor contingencies involved -- with these specificities defining the particular experience associated with each sense -- interactions between the senses are to be expected when there are systematic correlations and common sensorimotor contingencies[34]. Perceptual adaptation effects like the McCollough effect (McCollough [1965b]; McCollough [1965a]; Humphrey, Gurnsey, & Bryden [1994]; Harris [1980]) and the related disappearance of color fringes on adaptation to displacing prisms (Kohler [1951]; Held [1980]) may be manifestations of similar nature, except that they are intramodal rather than intermodal.
We propose that vision is an extended exploratory activity on the part of the animal. The brain does not see, and activity in the brain is not sufficient for vision. The brain stands to our ability to see in something like the relation that the engine stands to a car's capacity to drive, or in the relation that the heart stands to being alive. It is necessary, but not sufficient. Thus for example, when the heart stops, life ceases. But this does not mean that life resides in the heart. Similarly it is true that vision ceases when the brain dies. But vision does not reside in the brain. Failure to recognize that the brain is one element in the mechanisms of vision, and not where vision is generated, is not only an impediment to our understanding of vision, but holds back our attempts to understand the role that the brain actually does play.
In the following sections we review some recent work on the neural bases of vision and consider their status within the context of our theory.
Other kinds of neural correlates of conscious perceptual experience arise in the context of perceptual completion phenomena. A classical example is the work of von der Heydt and his colleagues, who found neurons in V2 that fire for illusory contours in a very similar way that they fire for real contours (von der Heydt & Peterhans [1989; von der Heydt, Peterhans, & Baumgartner [1984]; Peterhans & von der Heydt [1989]). A number of other examples involving perceptual completion have been reviewed by Pessoa et al. [1998].
Work like that described above has been received with enthusiasm: researchers presumably think that the discovery of neural correlates of perceptual experience brings us closer to understanding what gives rise to the perceptual experience. The underlying assumption is that if a set of neurons is found in the brain which correlates strongly with aware perceptual states, then, because these neurons are probably somehow linked to the mechanisms that are generating awareness, we are likely to be able to explain perceptual awareness by appeal to this neural activity.
But this reasoning is unsound. Indeed, consider what would happen if we were actually to find a set of neurons that correlated perfectly with visual awareness. For the sake of illustration, suppose we were to discover that in the pineal gland of macaque monkeys there were a tiny projection room in which what is seen by the monkey was projected on an internal screen, and whose activity correlated perfectly with the monkey's visual awareness. On reflection it is clear that such a discovery (this would surely be the Holy Grail of an NCC-seeker!) would not bring us any closer to understanding how monkeys see. For we would still lack an explanation of how the image in the pineal gland generates seeing, that is, how it enables or controls or modulates the forms of activity in which seeing consists. We would certainly be entitled, on the basis of the strong correlation between features of what is seen and features of what is projected onto the pineal projection screen, to assume that this neural activity played some role in vision. But nothing more could be said about such a discovery.
Why do some researchers believe that to understand the nature of consciousness or vision it is necessary to track down the neural correlates of conscious experience? One possible explanation is that these researchers are (perhaps unwittingly) committed to the idea the discovery of perfect correlation would give us reason to believe that we had discovered the neural activity sufficient to produce the experience (as suggested by Chalmers [1996a]). Teller & Pugh Jr. [1983] call such a neural substrate of experience the bridge locus. In addition, thinkers may unwittingly subscribe to what Pessoa, Thompson and Noë have called analytic isomorphism. This is the view that for every experience there will be a neural substrate whose activity is sufficient to produce that experience (a bridge locus) and that there will be an isomorphism (though not necessarily spatial or topographic) between features of the experience and features of the bridge locus. It is the existence of such an isomorphism that works to justify the claim that the discovery of such a neural substrate would explain the occurrence of the percept.
We believe that one must reject the metaphysical dogma of analytic isomorphism. As argued by Pessoa et al., no neural state will be sufficient to produce experience. Just as mechanical activity in the engine of a car is not sufficient to guarantee driving activity (suppose the car is in a swamp, or suspended by a magnet) so neural activity alone is not sufficient to produce vision.
Note also that if this view is correct, then it is a mistake to expect to find neurons which are perfectly correlated with visual consciousness. Ultimately visual consciousness is not a single thing, but rather a collection of capacities each of which can be appropriately deployed when necessary. Thus, we expect that if neurophysiologists do find neurons that correlate strongly with awareness, then most likely this will only be for one or other set of conditions or tasks.
In defense of this claim, there is considerable evidence that when neural correlates of consciousness have been found, these are sensitive to mood, attentional set, and task. So, for example, as stressed by Varela [1984] and Varela, Thompson, & Rosch [1991] (p. 93) (see also Thompson [1995], p. 217; Pessoa et al. [1998], p. 736, and Thompson & Varela [forthcoming]), responses in visual cells depend on behavioral factors, such as body tilt (Horn & Hill [1969]), posture (Abeles & Prut [1996]), and auditory stimulation (Morell [1972]; Fishman & Michael [1973]). Other studies show that attention and the relevance of a stimulus for the performance of a behavioral task can considerably modulate the responses of visual neurons (Moran & Desimone [1985]; Haenny, Maunsell, & Schiller [1988]; Chelazze, Miller, Duncan, & Desimone [1983]; Treue & Maunsell [1996]). Leopold and Logothetis themselves write of binocular rivalry: "we propose that the perceptual changes are the accidental manifestation of a general mechanisms that mediates a number of apparently different behaviors, including exploratory eye movements and shifts of attention. We also propose that while the different perceptions of ambiguous stimuli ultimately depend on activity in the `sensory' visual areas, this activity is continually steered and modified by central brain structures involved in planning and generating behavioral actions" (Leopold & Logothetis [1999], p. 254) Leopold and Logothetis suggest that to understand perceptual reversals of the kind encountered when we view an ambiguous figure, or when we undergo binocular rivalry, it is necessary to consider not only neural activity in visual cortex, but the animal's capacities for thought and action.
There are two motivations in the reasoning which underlies these types of investigations: one concerns temporal unity, and the other concerns what could be called 'conceptual' unity.
Certainly it is true that we have the impression that, when we recognize an object, all its attributes are seen simultaneously at one "perceptual moment". This leads scientists to think that the objects' attributes must be bound together synchronously in the internal representation in order to provide the singleness of the perceptual moment. But this is a fallacy. Thinking that physical synchrony is necessary for having a synchronous experience is the same kind of fallacy as thinking that because things look like 3D models or picture postcards to us, there must be a topologically equivalent map in the brain. Underlying this fallacy is the implicit assumption that the synchrony or coherence of perception requires presenting information in a synchronous or coherent way to an internal homunculus. In fact, just as the perception of the 3D world does not require 3D maps in the brain, subjective simultaneity does not require simultaneity of brain events[35]. This point has been made by Dennett & Kinsbourne ([1992]; see also O'Regan [1992]; Pessoa et al. [1998]).
Coming now to the issue of 'conceptual' coherence, a similar argument can be made: the fact that object attributes seem perceptually to be part of a single object does not require them to be "represented" in any unified kind of way, for example at a single location in the brain, or by a single process. They may be so represented, but there is no logical necessity for this. Furthermore, if they are represented in a spatially or temporally localized way, the fact that they are so represented cannotin itself be what explains the spatial, temporal or 'conceptual' phenomenology of perceptual coherence[36].
In the previous section we noted that were we to discover pictures in the brain that correlated with the experience of seeing, we would still have moved very little closer towards an explanation of seeing. But once we recognize this , then we further realize that there is no reason to suppose that to explain seeing we should seek for detailed internal pictures. There is no longer any rationale for supposing that there is a place in the brain where different streams of information are brought together and "unified" (whether conceptually or temporally). With the appreciation of this point we can dismiss the problem of binding as, in essence, a pseudo-problem.
Sadato et al. [1998] has suggested that in proficient Braille readers, tactile processing is "rerouted" to occipital visual cortex (see also Cohen et al. [1999]). Iriki et al. [1996] found that receptive fields of bimodal (somatosensory and visual) neurons in the caudal postcentral gyrus of macaque monkeys were altered during tool use "to include the entire length of the rake or to cover the expanded accessible space."
Despite the inherent interest of such work, it may actually be the case that the phenomenon of neural plasticity provides another nail in the coffin of the NCC program. This is because work like this shows that to understand the role the brain plays in supporting perceptual and motor capacities, it is necessary to keep clearly in view the broader context of the animal's skillful task-oriented activity. Specific neural states cannot be perfectly correlated with specific perceptual states. You cannot understand the contribution of neural activity if you restrict yourself to a brain's-eye view. This fits with our model of vision and visual consciousness. Seeing is not a simple occurrence; it is a rich, exploratory activity within a certain environment and with certain sensory apparatus, drawing on a number of heterogeneous capacities. Neural activity does not in itself produce experience. Neural activity contributes to experience only as one component within a bodily and environmental context.
Note that important evidence for this latter interpretation of the two streams hypothesis comes from studies of the effects of lesions in humans (Milner & Goodale [1995]). As they point out, damage to the dorsal stream is associated with impairments of visuo-motor control such as optic ataxia (Harvey [1995]) in the absence of impairments of the subject's ability to make verbal reports about the shape, features and location of what is seen. Conversely, damage to the ventral stream produces visual agnosias (Benson & Greenberg [1969], Milner et al. [1991]) without impairing visuo-motor functioning.
From the standpoint of the sensorimotor contingency view we propose here, the possibility of this kind of double dissociation is not surprising. In our view, seeing is an activity depending, to some extent, on a broad range of capacities, e.g. capacities for bodily movement and guidance, on the one hand, and capacities for speech and rational thought on the other. To the extent that these capacities are independent, it is not surprising that they can come apart in the manner described. It is not surprising, therefore, that the dorsal system can operate in relative isolation from the ventral system.
These points lead us to doubt, on certain interpretations at least, Milner and Goodale's claim that what the visual agnosia patient DF (who retains normal visuo-motor skill) lacks is visual awareness of what she sees. Indeed, Milner and Goodale suggest that, like DF, normals carry out visually guided actions using information that is not present in awareness. Only information in the ventral stream enters awareness. According to the view developed here (the sensorimotor contingency view), people are aware of what they see to the extent that they have control over that information for the purposes of guiding action and thought. Awareness is always, we argued, a matter of degree. Even the distracted driver is somewhat aware of what he sees, to the extent that, if we were to ask him, he would tell us what he is looking at. The case of DF is thus a case of what would seem to be partial awareness. She is unable to describe what she sees, but she is otherwise able to use it for the purpose of guiding action.
This may seem like a purely verbal dispute, but there is an important point at stake here. What makes the information conscious or aware, on our view, cannot consist just in the activity or lack of activity in a certain brain region (the ventral stream). Consciousness or awareness is not a property that informational states of the brain can just come to have in that way. Rather, visual awareness is a fact at the level of the integrated behavior of the whole organism. The work of Milner and Goodale suggests that damage to the ventral stream disrupts non-visuo-motor aspects of seeing. This is an important finding. But it would be a mistake to infer from this that the ventral stream is therefore the place where visual awareness happens.
Apart from the above provisos, the "two visual systems" view fits well with the position we develop in this paper. First, as expected from the sensorimotor contingency based approach, at the neural level there is a tight connection between seeing and moving. Second, the two-systems approach provides evidence supporting a claim we have made at different stages in this paper, namely, that seeing does not depend on the existence of unified representations of what is seen. On the two-systems approach, for example, there is not one single representation of space in the brain.
Processing in the two streams will in general have to be done in different coordinate systems. The "where" system, in order to allow reaching for an object and manipulating it, will have to make use of the position of the object relative to the observer's body. On the other hand, in the "what" system, recognition and classification of an object will require knowledge about the intrinsic shape of the object within an object-centered coordinate system, irrespective of the observer's position with respect to it.
This distinction between an observer-centered and an object-centered coordinate system is relevant to our classification of sensorimotor contingencies. In that section we emphasized that there are a subset of contingencies which are due to the particular spherical structure of the eyes, to the way they move, and due to the fact that sampling is being done by means of a two-dimensional perspective projection taken at a certain distance from the object, through optics and via a retinal mosaic that have very particular properties. These visual-apparatus-dependent rules are also constrained by the fact that the objects and the eyes are embedded in three-dimensional space (rather than, say, two dimensional space): laws of variation like the inverse square law for the amount of light reaching the eye, the linear relation between distance and projected size, will be common to all objects which are sampled through the visual apparatus.
If we were to construct a neural network connected to an eye-like input device and to a muscle-like eye-mover, and have it learn the rules of association between its input and output, then the first rules which the neural network would adduce , would be apparatus-based rules like the ones we have just described: these rules apply in all reasonably rich environments, irrespective of the objects contained in them. Furthermore we see that the coordinate system useful to the neural network would be the observer-based coordinate system, since what has to be learnt by the network is those (in)variance laws of the space of sensorimotor contingencies which are occasioned by the observer's own movements. This learning would also provide the system with the notion of "self", since it will allow it to distinguish between parts of its environment which it can systematically control, and parts which it cannot. The notion of "object" would also be something that would emerge from such learning: an object is something which can be removed and put back into the visual scene. These facts about what might be called object-combinatorics are independent of the identity of the objects themselves, and are related simply to their intrinsic "objectness" and their embeddedness in three dimensional space. It is possible that the dorsal "where" system could have evolved to serve this function.
The second subset of sensorimotor contingencies we referred to in our classification was the subset which we described as "object-related". These contingencies are those that allow objects to be distinguished from one another, and to be recognized independently of their position and orientation. Clearly a neural network which was trying to adduce laws of (in)variance from sensorimotor contingencies of this kind would have an advantage in coding information in object-centered rather than observer centered coordinates. This is known to be the case for the ventral "what" system, which could have evolved for this purpose.
Note that our classification into apparatus-related and object-related sensorimotor contingencies is a somewhat artificial division. Many of the laws underlying sensorimotor contingencies could be said to be both related to the visual apparatus and to the nature of objects. For example, the fact that objects are embedded in three dimensional space has as consequence that they can only show one face to the eye, and that as they are turned or as the observer turns around them, different parts appear and disappear. These facts are both a consequence of the fact that the eye is operating from a distance and so capturing only a single point of view -- an aspect of the apparatus -- and a consequence of the fact that objects have different sides -- an aspect of the objects.
Space prevents us from attempting to enter into these debates here. Instead, we follow a different strategy. On our view the qualia debate rests on what Ryle [1949/1990] called a category mistake. Qualia are meant to be properties of experiential states or events. But experiences, we have argued, are not states. They are ways of acting. They are things we do. There is no introspectibly available property determining the character of one's experiential states, for there are no such states. Hence, there are no (visual) qualia.
It is interesting to note that vision may, under certain circumstances, involve feelings or sensations of a non-visual nature. So, for example, if you are trying to track the movement of an object without moving your head, you may feel a certain distinctive eye strain. If you witness an explosion, you may feel dazzled in a way which causes definite sensation in the eyes. If vision is, as we have argued, a mode of activity, then there may be all sorts of features of that in which the activity consists that contribute to its "felt character." But crucially these are not intrinsic or defining properties of the experiencing, that is, they are not what philosophers call qualia. They are rather more or less accidental accompaniments of the activity of seeing on a particular occasion.
To explain our proposal, consider once again the phenomenon of change blindness. Many people say that they have the impression that when they see, the entire visual field is present to consciousness in all its nearly infinite detail. The change blindness results suggest that we do not have such detailed, picture-like awareness. What explains the conviction that we do? As we have discussed above, and as argued by O'Regan [1992] (c.f. also O'Regan et al. [1999]), the explanation is that we have access to all the detail by means of the mere flick of an eye or turn of the head, and so it is as if we had everything in view all the time.
We believe that something very similar is going on in the case of qualia as well. What explains the illusion that seeing consists in the occurrence of a feeling or a sensation is the fact that, at any moment, one can direct one's attention to one's seeing and there encounter such things as a red wall or a distinctive object. Because we have continuous access to any aspects or qualities of the environment, it may seem as if we are mentally in contact with them continuously. This leads us to say, mistakenly, that there is a feeling of redness (say) in our heads all along.
But note also that the qualities with which we come into contact by means of the direction of attention are not themselves inner, mental somethings. We encounter the redness of the wall, or the shape of the object. These distinctive qualities are not qualities of the experience, but qualities of what we experience. Visual experience is a mode of activity which enables us to have access to that environmental detail. But to thus have access to the detail, it is not required that we represent the detail in the head. The detail is there, in the outside world, ready for use when we need it.
The illusion of qualia, then, has two sources. First, there is the tendency to believe that because we have continuous access to the world, we are continuously experiencing the world in all its detail. Second, there is the false notion that qualities of experienced objects are qualities of the experience we have when we visually experience them.
Is the illusion of qualia really as widespread as it would seem? Perhaps not. If you ask a person what he sees, he will not bring up visual experiences and their intrinsic features. In everyday life, discussions of what we see are for the most part confined to discussions of things themselves. Even when we are viewing a piece of art, when we may deliberately try to reflect on the way the work affects us, nonphilosophers will rarely confuse the question of what it is like to look at the piece (what it reminds one of, how it makes one feel, whether one finds it pleasant, or not) with that favorite question of philosophers', namely, what is it like to have an experience as of seeing a painting (that is, what are the intrinsic, qualitative features of the visual experience)?
It is probable, therefore, that the illusion of qualia is a theoretical illusion and that it is not as commonplace as philosophers and vision scientists seem to think. If this is so, then it is an interesting question (but one best left to another occasion) why the view is so widespread among theorists.
We are skeptics about phenomenal consciousness understood the way Block and Chalmers understand it. As we stated above, what explains the illusion that seeing consists in the occurrence of an internal qualitative state is the fact that, at any moment, one can direct one's attention to one's activity of looking and so encounter such qualities as the redness of a wall, or the distinctive shape of a seen object. Moreover, we are able to track not only objects of awareness, but we can track our tracking activity and thus become aware (in the functional sense) of the percepts induced by the patterns of sensorimotor contingency governing our seeing. The experience of red, for example, arises when we know (though this is not propositional, but rather, practical knowledge) that, for example, if we move our eyes over a red region, there will occur changes typical of what happens when our non-homogeneously sampling retinas move over things whose color is red. It is, then, our continuous access to the redness that provides the key to understanding why it (mistakenly) seems to us as if we are continuously undergoing experience as of something red.
Our account of seeing and visual awareness thus cuts across the distinction between awareness and consciousness (as Chalmers puts it), or between A- and P-consciousness (in Block's terms). Visual awareness, is a matter of access, but access to the world, and to one's activity of tracking and interacting with the surrounding scene, not to one's internal information-bearing states. The felt or qualitative character of seeing is to be explained in terms of this active conception.
Block contrasts blindsight with a nonactual, but he thinks conceptually possible phenomenon of superblindsight. In superblindsight, as Block describes it, subjects have apparently normal access to information acquired in their blind fields, but they lack experience of the information. He invites us to imagine that these individuals have been trained to trust their "guesses" about what is present in the their blind fields. This information, therefore, is available to guide action and speech. Indeed, we are asked to imagine that as far as speech and behavior goes, people with superblindsight seem normal. There is one noteworthy exception, of course. If you ask them whether they visually experience what is present in their blind field, as they experience what is present to their non-blind field, they reply that they experience nothing. They are as good as blind as far as feeling goes.
Block's main contention then is two-fold: (1) that superblindsight is visual A-consciousness in the absence of P-consciousness, and (2) that superblindsight is conceptually possible. We doubt both points. As for (1), it seems that we have grounds for doubting that the patient really has a blind field. After all, the patient appears to see just fine. As Dennett [1995] notes, Block's account appears to trade, illegitimately, on the fact that in actual blindsight the kinds of information involved are remarkably sparse. The subject is correct, for example, about the orientation of a line grating. But if we imagine that informational content greatly enriched, as would seem required in the case of superblindsight, then the claim, on the part of the subject, that he lacks P-consciousness, becomes highly implausible. It is difficult to make sense of the claim that a person might offer an accurate description of a painting say, describing all the colors and the geometry of the composition in a natural manner, all the while having no experience of the painting. One loses all grip on what it could mean to say that the subject has no experience. And this indicates the nature of our misgivings about (2). If you are perceptually alert to the presence of environmental detail in a manner that allows you to describe what is present, and if you are sensitive to the appropriate, visual, laws of sensorimotor contingency (for example the detail is no longer accessible when you close your eyes or the lights go out), then surely it would be very peculiar to say that you are not experiencing/seeing the painting.
Nor do we find Block's [1995] examples of cases of P-consciousness without A-consciousness convincing. He gives the example of having a conversation while a power drill makes a racket outside the window. One is engrossed in the conversation, and one does not notice the drill. All of a sudden one notices it. Block proposes that in a case such as this, in so far as one did hear the drill before noticing it, one was P-conscious of the drill while at the same time A-unconscious of it. When one noticed the drill, one becomes A-conscious of what one had previously heard and been P-conscious of all the time.
But did one hear the drill before one noticed it? The view developed here requires a negative answer to this question. One does not hear the sound of the drill because one does not make use of one's auditory tracking. This is of course compatible with its being the case that we are sensitive to the sound before we hear it (before we become conscious of it). The auditory system will analyze and store (perhaps only in a short-term memory buffer) information pertaining to the drill. But we do not use that information nor are we, before we notice the drill, poised to use that information or able to use that information to guide our behavior, thought, movement, or perceptual exploration.
One might challenge Block's view in a different way. Consider a slightly different but familiar example. A bell is chiming. All of a sudden you notice not only that there is a bell chiming, but that there were ten chimes in all. Surely this shows that you heard the chimes even before you noticed it? Indeed, what this would show, as Chalmers has argued, is that there is a sense in which one was in fact poised to make use of the unnoticed sounds one was hearing even as one failed to notice them. That is, according to this line of reasoning, one was A-conscious of the unexperienced sounds (contrary to what Block would say).
One virtue of this account is that it perhaps fits somewhat better with ordinary usage of words like "hear" and "see." That is, it seems quite natural to say that you heard the clock chime without noticing it. But there are substantive empirical reasons to reject this account nonetheless. The fact that a stimulus is present and is actively impinging on the senses, does not entail that you perceive it. This is the central upshot of the change blindness studies (discussed in Section 5.10) and also recent work on so-called "inattentional blindness" (discussed in Section 5.11). The fact that a stimulus is present means that it is available to be probed by the active animal. Only while the active probe is occurring do you get conscious perception (seeing or hearing say).
The conflict between our view and that of commonsense is actually more apparent than real. As we noted earlier in our discussion of awareness, awareness is a matter of degree. Part of what makes it seem so reasonable to say that you heard the noise without noticing it, or that you (a driver) saw a car without noticing it, is that we may call to mind cases where you are in fact noticing a sound or an object a little bit. For example, you are trying to have a conversation and there's that irritating noise in the background which threatens to interrupt you but to which you are paying very little attention. Nevertheless, having said all this, we are quite prepared to bite the bullet and insist that in the complete absence of current access, there is no perception.
Note that to say that there is no perception is not to say that there may not be significant unconscious influence on behavior or action[38].
Consider, first, that our theory is supported by reflection on what it is like to have perceptual experience. It is commonly asserted by both philosophers and scientists, that it seems to normal perceivers as if perception involves detailed internal representations of the environment in the head. As noted in Section 7.1, we believe this misdescribes the character of seeing. First of all, in seeing we commit ourselves to no beliefs about what is going on in our heads. Seeing is directed to the world, not the brain. Second, when we see, we take the perceived detail to be out there in the world, not in our head. Indeed, we take ourselves to be embedded in the environment and to have access to detail through active exploration. On our view, it is just bad phenomenology to assert that we take ourselves to have a 3D-model or picture in the head when we see. In short, we believe that phenomenological reflection on the character of experience, once it has broken free of clichés about pictures in the head, supports the approach developed here.
But second, as argued by Noë [submitted-b], it now seems clear that the conception of perception developed here helps place phenomenological considerations on firm footing. The subject matter of perceptual phenomenological reflection is not an ephemeral, ineffable, sensation-like momentary occurrence in the mind, but rather the real-world, temporally extended, activity of exploring the environment and the structure of sensorimotor contingencies. There is a qualitative or phenomenological difference between seeing and hearing and touching, as stated. These are different activities, corresponding to different modes of exploration of the structure of sensory-motor contingencies. To see a bottle, for example, is to explore visual-motor contingencies, e.g. transformations in the appearance of the bottle as one moves in relation to it. To touch it, on the other hand, is to explore the structure of tactile-motor contingencies. The bottle impedes, guides and informs tactile exploration of the bottle. To reflect, then, on what it is like to see the bottle, or to touch it, is to reflect on just these sorts of fact about the active engagement the perceiver undertakes with the environment (See Noë [submitted-a]).
We strongly endorse Dennett's criticism of what he calls Cartesian materialism (Dennett [1991; Dennett & Kinsbourne [1992]). Many philosophers and scientists assume that consciousness is an intrinsic property of neural states. The idea is that, among the multitude of content-bearing states in the brain, some subset of states have an additional property of being phenomenally conscious. (This is in contrast to states which, in the terminology of Block, are access-conscious. This access consciousness is not thought to be an intrinsic property of the state but one that depends on the relation between that state and others in the broader system.) The problem of consciousness, on this general picture, is to understand what processes or mechanisms or events in the brain make certain contents phenomenally conscious. Where, and how, does consciousness happen in the brain?
We reject not only specific attempts to answer this question (microtubules, etc), but the assumptions implicit in the question itself. That is, like Dennett, we reject Cartesian materialism. Phenomenal consciousness is not a property of states in what Dennett calls the subpersonal system (i.e. the brain -- whether thought of in neural or in more abstract cognitive or computational terms) (Dennett [1978]; [1969]; [1987]). There need be no one-to-one correlation between states of consciousness and events in the brain.
But this brings us to the main point of our disagreement with Dennett. Although we reject accounts of phenomenal consciousness as a property of subpersonal states, we do not deny (as we have made clear in the previous section), that there are experiences and that there are facts about what experiences are like. But these, however, are facts not about a person's inner life (about qualia or raw feels). They pertain, rather, to the person's (or animal's) active engagement with the world he or it inhabits. They are facts at the personal (as opposed to subpersonal) level. We return to this point below in section 7.11.
One of the cornerstone's of Dennett's approach to the problem of consciousness, is his conception of heterophenomenology (Dennett [1991]). In many respects, we are very sympathetic with this approach. The best way to understand what Dennett means by heterophenomenology is to contrast this view with what we call introspectionism. Introspectionism is the view that the conscious subject has immediate and infallible epistemic access to his or her conscious states. Perhaps not too many writers would endorse introspectionism when put forward in this blunt manner, but it is clear that something like this idea drives a good deal of discussion in contemporary consciousness studies. Theorists believe that we know, on the basis of reflection on our own case, what our own conscious states are like. Dennett rejects introspectionism. Dennett has a lot to say about why introspectionism is untenable, and we are sympathetic to his position. For our purposes it is enough to point out that, according to Dennett, as scientists we cannot assume that subjects are right in their first-personal avowals of conscious experience. Such reports are just further bits of evidence about the nature of mental life and they have no privileged status with respect to other forms of evidence (e.g. psychophysical, neural, psychological, etc).
The basic upshot of these considerations, then, is that, according to heterophenomenology, first-person reports of experience have no special status attached to them. There is no deep and unfathomable asymmetry between what can be known in the first person, and what can be known in the third person. Note that to deny any privileged status to the first person is not to deny the importance of the first-person perspective. This is a point that Dennett would surely accept.
Our main disagreement with Dennett on these matters is, in some respects, a tentative one. Like Dennett, we reject the naïve introspectionism. And like Dennett, we endorse the fruitfulness of the heterophenomenological perspective. But nothing in the argument as we have reviewed it entails that all first-person approaches to consciousness must take the form of naïve introspectionism. Specifically, if one accepts the conception of experience we have endorsed here, then first-person reflection on the character of experience would not consist in introspection at all, but rather in attentiveness to the complexity of the activity of perceptual exploration. Consider an example: Dennett criticizes what he takes to be the widespread assumption, on the part of perceivers, that the visual field is in sharp detail and uniform focus from the center out to the periphery. Simple tests (e.g. the colored pencil test mentioned earlier in Section 5.6), and well-known facts about the non-homogeneity of the retina, suffice to show that this account of the quality of the visual field is misguided. But is it really true that normal perceivers think of their visual fields this way? Do normal perceivers really make this error? We think not. As noted earlier in connection with change blindness (and see Noë et al. [2000]; Noë [submitted-a] and Pessoa et al. [1998]), normal perceivers do not have ideological commitments concerning the resolution of the visual field. Rather, they take the world to be solid, dense, detailed and present and they take themselves to be embedded in and thus to have access to the world. (And of course they do.)
The point of this example is that Dennett seems to mischaracterize how things seem to perceivers, that is, he mischaracterizes their first-person judgments as to the quality of experience. Apart from the specifics of the example, the upshot of this is that there are substantive empirical questions about the quality of experience. This opens up the possibility, considered in the previous section, that there are modes of reflection on the character of experience that are both first-personal and nonintrospectionist..[39]
We have adopted the same strategy that biologists adopted in the Nineteenth Century in their study of the phenomena of life. Life is not something in addition to the patterns of biological activity involving growth, reproduction, nourishment, excretion, etc. Life just is all that. There is no need, therefore, for appeal to vital forces in addition to one's chemical and physical explanation of these other phenomena in order to understand the phenomenon of life. In just this way, we do not deny that there is experience or consciousness, or even that there is something it is like to have a perceptual experience. What we deny is the idea that the problem of qualia is a problem over and above that of accounting for the activities in which seeing consists and which we have been at pains to describe.
In this way we believe that the sensorimotor theory of vision enables us to close up the explanatory gap.
Many philosophers object to functionalism on the same grounds that they object to behaviorism, namely, that it denies the existence of inner states of which we have direct first-person knowledge. Such criticisms have been made by, for example, Block [1978] and Searle [1980]. This objection can be reformulated as the claim that not all mental states are functional states (or perhaps as the claim that not all properties of mental states are functional properties of those states). In addition to mental states such as beliefs and desires, there are mental states that are intrinsically qualitative. Such qualitative states are not functional states. One could have two functionally identical individuals, according to this line of criticism, who disagreed with respect to their qualitative states. One familiar argument for this is the so-called hypothesis of undetectable spectrum inversion (Shoemaker [1970]; Palmer [1999]). Suppose that the sensation A has when he looks at something both A and B call "green" is the sensation B has when he looks at something both A and B call "red." Suppose, in addition, that all our color sensations are in this way "inverted" with respect to the experiences of another. If this were to occur then we would have radically different experiences of color but there would nothing in our functional description corresponding to this difference. A second influential argument concerns the possibility of what philosophers call zombies (e.g. Kirk [1974]; Chalmers [1996a]). A zombie is just like a person, say, but there is nothing that it is like to be that person. That person has no subjective perspective, no experience, no point of view. But the zombie is, by hypothesis, "normal" with respect to its functional description. Sameness of qualitative states, therefore, is not guaranteed by sameness of functional states. If these considerations are sound, then although functionalism may provide the correct framework for the study of a broad range of mental states, its stops short of providing a complete framework for the study of the mind. Functionalism leaves an explanatory gap.
We are unconvinced by these arguments. Without addressing them head-on, we have tried, in these final sections of this paper, to give some indication of how we believe it is possible to maintain a broadly functionalist theory in the face of these objections. We do this by providing an account of experience as a mode of active exploration of the environment which not only does away with talk of qualitative states, but also provides the elements to explain why people find it so natural to suppose the existence of qualia. We have also shown, in this philosophical chapter, how we believe that the talk of such would-be qualitative states is actually confused.
First, as we noted above, according to functionalism, beliefs, desires, experiences are functional states of a person or organism. On our view, however, perceptual experiences are not states of a person or organism. Perceptual experiences do not happen in animals. Animals, if you like, do them. Seeing is the activity of exploring the environment as mediated by patterns of sensorimotor contingency. Indeed, one might doubt whether any of the kinds found in traditional folk psychology--e.g. beliefs, desires, etc--are states.
Second, on our view one cannot entirely divorce the study of vision from the study of the way vision is implemented. Consider the importance, for our account, of the notion of a sensorimotor contingency. How things look or sound to an animal depends on the movements and actions of the animal. But the movements or actions of the animal depend on the nature of the sensory organs and other features of the animal's body (e.g. degrees of freedom of eye movements, head movements, etc). The body and even the brain figure directly in the specification of what seeing consists in. Because of this, there are a priori limits to the multiple realizability of vision. Only a creature who is, at some appropriate level of characterization, embodied as we are embodied can see as we see, for only a creature similarly embodied can do what we do and seeing is, after all, something we do.
In place of the sharp distinction then between explanation at the abstract functional level and explanation at the level of implementation we can substitute Dennett's distinction between the personal (or animal) and the subpersonal levels. Talk of the subpersonal level, it should be noted, is itself ambiguous between talk of the subpersonal system functionally characterized, or talk of the subpersonal system as a physical system. On our view, vision is a phenomenon of the personal level, not of the subpersonal level however construed. For vision does not consist in something that happens in a part of the animal (although it depends on what goes on inside animal parts). Vision is something that the animal does, using its parts (either deliberately in the way we use the hands and the eyes or automatically in the way we use the brain itself).
From this point of view, understanding vision amounts to understanding the various facets of the things people do when they see. We suggest that the basic thing people do when they see is that they exercise mastery of the sensorimotor contingencies governing visual exploration. Thus, visual sensation and visual perception are different aspects of a person's skillful exploratory activity (that is, exploratory activity guided by practical knowledge of the effect movement will have on nervous influx). Visual awareness depends, further, on the person's integration of these patterns of skillful exercise into ongoing planning, reasoning, decision-making and linguistic activities. As we have argued, these ingredients are sufficient to explain the otherwise elusive character of visual consciousness.
In addition, this way of thinking has the advantage of providing an account of what differentiates the sensory modalities. The problem is solved naturally, without appealing to the existence of sensory-modality-specific essences or mechanisms, by the stance taken here of saying that perceptual experience is a way of acting: Just as horse riding is different from motorcycling, so seeing is different from hearing. These differences can be explained without appeal to the essences of horseback riding and motorcycle riding, and without appeal to specific nerve energies or pathways devoted to seeing and hearing.
These are matters of great philosophical significance, and we have sought, in developing our position, to make clear that in denying the need for qualia, we are not denying the existence of perceptual experience, or the possibility of phenomenological reflection on experience. Rather, we have proposed a new way of thinking about what goes on when we experience that, as we have argued throughout, captures what we believe, as experiencers, about our experiential lives, but does so in a manner that does not give rise to the mystery of the explanatory gap.
These are matters of great empirical significance. The sensorimotor approach to vision we have laid out here has provided the impetus for a series of important experiments on what has come to be known as change blindness. The robustness of these results in turn serve to vindicate the framework itself. In addition, we have tried to demonstrate that the sensorimotor view presented here allows for a wide range of otherwise baffling perceptual phenomena--e.g. perceptual completion, inattentional blindness, perceptual stability despite observer eye-movement, prosthetic perception, color vision, inverting lens adaptation, the surgical restoration of sight in the congenitally blind, blindsight, the double dissociation of optic ataxia and visual agnosia, and so on--to find economical and straightforward explanation. We think it is striking evidence of the power of the position we develop here that it is able, so elegantly, to account for such a broad range of perceptual phenomena.
Finally, if we are correct in our analysis, there is need for some very
radical reassessment on the part of working scientists, particularly in
the branch of visual neuroscience concerned with finding the NCC. We invite
neuroscientists to join in exploring ways of thinking about the neural
bases of perception and action that do not rest on the false assumption
that the brain is the seat of consciousness. We invite philosophers to
consider the way in which empirical controversies in this area require
the formulation of what is, in effect, a new metaphysics of mind and body.
ACKNOWLEDGMENTS. The authors deeply thank the following people
for their help with different aspects of this work: Paul Bach y Rita, Benédicte
de Boysson Bardies, Hasna Ben Salah, Ned Block, Justin Broackes, Philippe
Capet, Jim Clark, Frédérique Devignemont, Juan Gonzalez,
Andrei Gorea, Keith Humphrey, Pierre Jacob, Marcel Kinsbourne, Christof
Koch, Stephen Kosslyn, Ariane Lévy-Schoen, Michael O'Regan, Elias
O'Regan, Malcolm MacIver, Erik Myin, Ernst Niebur, Luiz Pessoa, Bill Phillips,
Ronan Reilly, Juan Segui, Paulo Sousa, Jean-Pierre Rospars, Evan Thompson,
Andreas Wohlschläger, Jeremy Wolfe, Tiziana Zalla. A.N. gratefully
acknowledges the support of faculty research funds granted by the University
of California, Santa Cruz.
FOOTNOTES
1 C.f also Abeles & Prut [1996]; Milner [1974]; von der Malsburg [1983]; Kahn, Pace-Schott, & Hobson [1997]; Rodriguez et al. [1999].
2 We can frame the problem in terms of Teller's ([1983]; Teller & Pugh Jr. [1983]; Teller [1984]) notion of linking propositions. Linking propositions specify the mappings between the forms of neural responses and the qualities of percepts.
3 See Gorea [1991] for a discussion of the relation of modern psychophysics to Müller's concept.
4 The analogy with richness breaks down somewhat because one could claim that richness does not in fact feel like anything at all. Another difference with richness is that the "knowing" we refer to in the case of visual sensation is much more implicit and non-propositional.
5 This classical reasoning is an instance of what Pessoa, Thompson, & Noë [1998] call "analytic isomorphism," that is, the view that at the neural substrate of an experience there must be an isomorophism between percept and substrate. Analytic isomorphism comes up again in our discussion of the neural basis of vision in Section 6.
6 Koenderink [1984] has a very perspicacious discussion of what it is to perceive, rather than simply to record information, where he makes this point.
7 Heil [1983], agreeing with Gibson and rejecting Müller's idea of physiological "channels" associated with different senses, also attempts a taxonomy of the different senses but does not suggest the idea that it could be the laws obeyed by the sensorimotor contingencies that are the essential fact that differentiates them.
8 Note that it could be claimed that Müller's idea of specific nerve quality could be salvaged by supposing that what differentiates the senses is different calculations that are done in the different pathways. This was suggested by Wittmann, Pöppel and Schill, reviewers of the original version of this manuscript. In a way this is what is being proposed by the present approach, although the present approach emphasizes that the calculation itself is not enough. What is needed is for the structure of the input/output relationships to obey different laws.
9 Note that we have been careful not to say that vision or horseriding provide different experiences: the experience is the fact of engaging in the activities. The activities, we claim, are not providing an experience -- though people often use the word provide in this way, we claim this is a figure of speech, and not indication of a true experience-generating mechanism. It is precisely this kind of misunderstanding which gives rise to the problem of the explanatory gap. Cf Section 3.7.
10 Merleau-Ponty [1968] has also compared vision to palpation. c.f. also Berthoz [1997], who stresses the rôle of action in perception.
11 Minsky [1988] also suggested a similar idea.
12 It is often assumed that transients must necessarily direct attention to a location. But presumably location is only one feature of visual stimuli, and in the brain, location may have a similar status to other features, like color, orientation, contrast, etc. Could it be that attention can be directed to aspects of a stimulus defined by such other features? For example is it possible to direct attention to all the red items in a scene, or to scene region constituted by a 3D surface? C.f Pylyshyn [1988] on this issue.
13 It is worth noting, in addition, that although people certainly say that their dream images are pictorial in character, there is reason to doubt that this is really so (or perhaps even that it is even intelligible). Is it supposed that dreaming is like looking at pictures? Clearly it is not. Is the idea that when we dream, or when we imagine, it is as if the dream represents the dream world with the same level of detail as one would find in a picture? If this is what the claim that "dreams are pictorial in character" comes to, then it is not at all clear that it is true. Does it really seem to you, when you dream, as if all the dreamt environmental detail is represented in sharp focus and uniform detail from the center out to the periphery? Note, it seems pretty certain that that is not what actually seeing is like. It might very well be the case that only those currently processed features of the dream-world are in fact really experienced. A somewhat paradoxical conclusion, but by no means obviously mistaken.
14 They observed no eye movement advantage but less than 50% correct performance in counting a grating pattern of identical vertical bars: it may be that the observers were using a strategy of estimating the number of bars by evaluating the number on the basis of the overall width of the pattern.
15 A portion of the existing data purportedly measuring the extraretinal signal under conditions of normal viewing can, to some degree, probably be explained by assuming that they are due to purely retinal effects (smear, retinal persistence, differences in spatio-temporal effects in central and peripheral vision; c.f. O'Regan [1984]).
16 It could be argued that people actually do have a detailed, picture-like internal representation of the outside world, but that it is destroyed at each saccade or on interruption by flicker and other transients. Alternately, as suggested by reviewers Wittmann, Schill and Pöppel of our manuscript, it may be that we deceive ourselves as to the amount of detail that we think we see in the representation. Such arguments are hard to square with data showing interaction of the change blindness effects with central/marginal interest manipulations, and with the data from the "mudsplash" experiments, among others. (cf. O'Regan, Rensink, & Clark [1996; Rensink, O'Regan, & Clark [1997; Rensink, O'Regan, & Clark [2000]). Similar alternatives have also been discussed by Simons [2000].
17 D'Zmura & Lennie [1986] have a theory of color constancy that depends on eye movements.
18 Broackes additionally notes: "And if it is puzzling how a dynamic property can make itself manifest in a static perception ('how can a disposition to present a variety of appearances be visible in a single appearance?'), then we already have, in familiar discussions of aspect-shift, the theoretical apparatus for a solution. It is because there is 'the echo of a thought in sight'". Broackes quotes Strawson [1974], p. 52-53 who says: "To see [a newly presented object] as a dog, silent and stationary, is to see it as a possible mover and barker, even though you give yourself no actual images of it as moving and barking."
19 Broackes says that contrary to what he said in Broackes [1992], he is either protanomolous or protanope. (Broackes, personal communication).
20 Cole [1991] has also invoked these studies in a functionalist defense against the inverted spectrum problem.
21 Curiously, many people wearing normal glasses seem to voluntarily peer over the rims of their glasses when they look at you, as though this procured some kind of advantage in seeing.
22 Dolezal stresses that the use of the terms upside down and right side up is confusing, and guards against saying that the world comes again to appear right side up. He says that in his experiment the final state of adaptation could be distinguished from the state before the experiment. However presumably this could be because the duration of the adaptation was perforce limited, and because use of inverting goggles necessarily involves other constraints like the limited field of view and the weight of the apparatus. Howard & Templeton [1966] also stress the need to be wary of the terms upside down and right side up (see also Smith & Smith [1962]).
23 Chapter 8 in Taylor's book contains a detailed, behaviorist theory of the effects of inversion of the visual world, referring to specific results of Stratton, Ewert and Kohler. The outcome appears to be that the observed adaptation effects are to be expected, and that the nativist theory is "shattered" (p. 168).The chapter includes a mathematical appendix by Seymour Papert, who was the subject in Taylor's left-right inversion experiment.
24 The situation may be similar to what happens when you move to a new town, and attempt to orient yourself. It takes some time before local and global landmarks merge into a coherent representation of the town. Until that happens (and it may never do), you may make gross mistakes. For example you may be perfectly able to orient yourself locally, but be unable to correctly indicate the direction of a well-known global landmark.
25 Bedford [1995] has a theory of perceptual learning which is related to the theory presented here.
26 Movies of the demonstrations can be found on the first author's web page at http://nivea.psycho.univ-paris5.fr. See also http://www.cbr.com/~rensink/flicker/flickDescr.html.
27 Movie demonstrations of some film-cut and other effects can be found on Simons' page on http://coglab.wjh.harvard.edu.
28 The following quote from Haines [1991] is an example: 'Pilot F was a high-flight-time Captain who demonstrated exceptionally good performance both with and without HUD. The runway obstruction run was his seventh data run. He indicated his "Decision (140 ft) . . to land (110 ft)," and proceeded to do so. The experimenter terminated the run at an altitude of 50 ft. * The pilot was surprised. Captain: "Didn't get to flare on this one." First Officer "No you didn't . . . I was just looking up as it (the picture) disappeared, and I thought I saw something on the runway. Did you see anything?" Captain: "No, I did not." The experimenters suggested that an equipment failure was probably to blame. Both of these pilots saw the obstruction during the second exposure without HUD (13 runs and 21 runs later, respectively) and executed missed approaches. Later, when he was shown the videotape of this run, Pilot D said, "If I didn't see it (the tape), I wouldn't believe it. I honestly didn't see anything on that runway."'
29 This demonstration may not work if the file is being viewed on the web or has been printed with the option of substitution of typography enabled. The point is that there are two ways of forming an "a"; one similar to the hand-written a (a circle with a line next to it), and one similar to a typewriter a. If hand-written-like "a"'s are mixed into a text, provided they have the same height and density as normal a's, this will generally not be noticed.
30The word "of" is repeated. Repetitions of the word "the" can also be easily missed.
31 There are nine f's. Many people fail to count the f's in the three occurrences of the word "of".
32 In fact sensation itself is an abstraction, as already noted by James [1890/1950] (Vol 2, p. 3).
33 Howells [1944] cited by Taylor [1962] p. 246 is an interesting example where association of a low and high pitched tone with red and green respectively, over 5000 trials, gave rise to a perception of white being tinged with red and green when white was associated with the tones.
34 Bedford [1995] has a theory bringing together the McCollough effect and adaptation to prism displacement which is similar in concept to the present theory.
35 In a criticism of Crick & Koch's arguments, Cogan [1995] also suggests that the notion of "perceptual moment" may not be useful. Dennett [1991] notes a similar point in his "multiple drafts" theory of consciousness.
36 A similar argument was made in Section 5.5 with regard to the "filling in" of the blind spot: there may actually be what look like filling in processes in the brain, but these can not be what provide us with the impression of the blind spot being filled in.
37 There is a large family of similar debates, e.g. the hypothesis of the inverted spectrum Block [1990]; Harman [1990]; Shoemaker [1970]; Palmer [1999]; the what-Mary-knows puzzle Nemirov [1984]; Lewis [1988]; Jackson [1986].
38 This has been shown by Chun & Nakayama [2000] in the context of experiments on change blindness (cf. also Chun & Jiang [1998]).
39 The phenomenological movement in philosophy is, to some extent, concerned precisely with the development of such first-personal methods.
40 Varela et al. [1991] and Thompson, Palacios, & Varela
[1992] also make this point. In their terminology, consciousness is something
we enact.
Anderson, S. J., Mullen, K. T., & Hess, R. F. (1991). Human peripheral spatial resolution for achromatic and chromatic stimuli: limits imposed by optical and retinal factors. J Physiol (Lond), 442, 47-64.
Apkarian, P. A. (1983). Visual training after long term deprivation: a case report. Int J Neurosci, 19(1-4), 65-83.
Arend, L. E., Jr. (1973). Spatial differential and integral operations in human vision: implications of stabilized retinal image fading. Psychol Rev, 80(5), 374-395.
Arias, C. (1996). L'écholocation humaine chez les handicapés visuels. L'Année Psychologique, 96, 703-721.
Atkinson, J., Campbell, F. W., & Francis, M. R. (1976). The magic number 4 +/- 0: a new look at visual numerosity judgements. Perception, 5(3), 327-334.
Bach-y-Rita, P. (1972). Brain mechanisms in sensory substitution. New York: Academic Press.
Bach-y-Rita, P. (1983). Tactile vision substitution: past and future. Int J Neurosci, 19(1-4), 29-36.
Bach-y-Rita, P. (1984). The relationship between motor processes and cognition in tactile vision substitution. In A. F. Sanders & W. Prinz (Eds.), Cognition and motor processes . Berlin: Springer.
Balibar, F. (1992). Einstein 1905. De l'éther aux quanta. Paris: PUF.
Ballard, D. H., Hayhoe, M. M., Pook, P. K., & Rao, R. P. N. (1997). Deictic codes for the embodiment of cognition. Behavioral & Brain Sciences, 20(4), 723-767.
Baron-Cohen, S., & Harrison, J. (1996). Synesthesia: Classic and Contemporary Readings. Oxford: Blackwells.
Bedford, F. L. (1995). Constraints on perceptual learning: Objects and dimensions. Cognition, 54(3), 253-297.
Benson, D. F., & Greenberg, J. P. (1969). Visual form agnosia: a specific deficit in visual discrimination. Arch. Neurol., 20, 82-89.
Berthoz, A. (1997). Le sens du mouvement. Paris: Odile Jacob.
Block, N. (1978). Troubles with functionalism. In C. W. Savage (Ed.), Perception and Cognition: Issues in the foundations of psychology (Vol. 9, pp. 261-325). Minneapolis: University of Minnesota Press.
Block, N. (1990). Inverted earth. Philosophical Perspectives, 4, 53-79.
Block, N. (1995). On a confusion about a function of consciousness. Behavioral and Brain Sciences, 18(2), 227-247.
Bone, R. A., Landrum, J. T., & Cains, A. (1992). Optical density spectra of the macular pigment in vivo and in vitro. Vision Res, 32(1), 105-110.
Botvinick, M., & Cohen, J. (1998). Rubber hands 'feel' touch that eyes see [letter]. Nature, 391(6669), 756.
Bower, T. G. R. (1977). Blind babies see with their ears. New Scientist, 73(255-257).
Bower, T. G. R. (1979). Human development. San Francisco: Freeman.
Brecht, M., Singer, W., & Engel, A. K. (1998). Correlation analysis of corticotectal interactions in the cat visual system. J Neurophysiol, 79(5), 2394-2407.
Breitmeyer, B. G., & Ganz, L. (1976). Implications of sustained and transient channels for theories of visual pattern masking, saccadic suppression, and information processing. Psychol Rev, 83(1), 1-36.
Bridgeman, B. (1995). Extraretinal signals in visual orientation. In W. Prinz, Bridgeman, Bruce (Ed.), Handbook of perception and action, Vol. 1: Perception (pp. 191-223). London, England UK: Academic Press, Inc.
Bridgeman, B., & Stark, L. (1991). Ocular proprioception and efference copy in registering visual direction. Vision Res, 31(11), 1903-1913.
Broackes, J. (1992). The autonomy of colour. In K. Lennon & D. Charles (Eds.), Reduction, Explanation, and Realism (pp. 421-465). Oxford: Oxford University Press.
Brooks, R. (1987). Autonomous mobile robots. In W. E. L. Grimson & R. S. Patil (Eds.), AI in the 1980s and beyond . Cambridge, Mass.: MIT Press.
Brooks, R. (1991). Intelligence without representation. Artificial Intelligence, 47, 139-159.
Burr, D. C., Morrone, M. C., & Ross, J. (1994). Selective suppression of the magnocellular visual pathway during saccadic eye movements. Nature, 371, 511-513.
Castelo-Branco, M., Neuenschwander, S., & Singer, W. (1998). Synchronization of visual responses between the cortex, lateral geniculate nucleus, and retina in the anesthetized cat. J Neurosci, 18(16), 6395-6410.
Chalmers, D. J. (1996a). The conscious mind: In search of a fundamental theory. New York, NY, USA: Oxford University Press.
Chalmers, D. J. (1996b). On the search for the neural correlate of consciousness. Paper presented at the Tucson II.
Chelazze, L., Miller, E. K., Duncan, J., & Desimone, R. (1983). A neural basis for visual search in inferior temporal cortex. Nature, 363.
Chun, M., & Nakayama, K. (2000). On the functional role of implicit visual memory for the adaptive deployment of attention across scenes. Visual Cognition, in press.
Chun, M. M., & Jiang, Y. (1998). Contextual cueing: implicit learning and memory of visual context guides spatial attention. Cognit Psychol, 36(1), 28-71.
Cogan, A. I. (1995). Vision comes to mind. Perception, 24(7), 811-826.
Cohen, L. G., Weeks, R. A., Sadato, N., Celnik, P., Ishii, K., & Hallett, M. (1999). Period of susceptibility for cross-modal plasticity in the blind. Ann Neurol, 45(4), 451-460.
Colby, C. L., Duhamel, J. R., & Goldberg, M. E. (1996). Visual, presaccadic, and cognitive activation of single neurons in monkey lateral intraparietal area. J Neurophysiol, 76(5), 2841-2852.
Cole, D. (1991). Functionalism and inverted spectra. In J. H. Fetzer (Ed.), Epistemology and cognition. Studies in cognitive systems, Vol. 6 (pp. 85-100). Dordrecht, Netherlands: Kluwer Academic Publishers.
Coletta, N. J., & Williams, D. R. (1987). Psychophysical estimate of extrafoveal cone spacing. J Opt Soc Am [A], 4(8), 1503-1513.
Cornilleau-Peres, V., & Droulez, J. (1994). The visual perception of three-dimensional shape from self-motion and object-motion. Vision Res, 34(18), 2331-2336.
Crick, F., & Koch, C. (1990). Toward a neurobiological theory of consciousness. Seminars in the Neurosciences, 2, 263-275.
Crick, F., & Koch, C. (1995). Are we aware of neural activity in primary visual cortex? Nature, 375, 121-123.
Crick, F., & Koch, C. (1998). Consciousness and Neuroscience. Cerebral Cortex, 8, 97-107.
Cytowic, R. E., & Wood, F. B. (1982). Synesthesia. I. A review of major theories and their brain basis. Brain Cogn, 1(1), 23-35.
D'Zmura, M., & Lennie, P. (1986). Mechanisms of color constancy. J Opt Soc Am [A], 3(10), 1662-1672.
Dallenbach, K. M., Supa, M., & Cotzin, M. (1944). "Facial vision" the perception of obstacles by the blind. American Journal of Psychology, 57, 133-183.
De Weerd, P., Gattass, R., Desimone, R., & Ungerleider, L. G. (1995). Responses of cells in monkey visual cortex during perceptual filling-in of an artificial scotoma. Nature, 377(6551), 731-734.
De Yoe, E. A., & van Essen, D. C. (1988). Concurrent processing streams in monkey visual cortex. Trends in Neuroscience, 11, 219-226.
Dennett, D. C. (1969). Content and Consciousness. London: Routledge & Kegan Paul.
Dennett, D. C. (1978). Brainstorms. Cambridge, MA: The MIT Press.
Dennett, D. C. (1987). The intentional stance. Cambridge, Mass.: MIT Press.
Dennett, D. C. (1991). Consciousness explained. Boston, MA, USA: Little, Brown and Co.
Dennett, D. C. (1995). The path not taken. Behavioral and Brain Sciences, 18(2), 252-253.
Dennett, D. C., & Kinsbourne, M. (1992). Time and the observer: the where and when of consciousness in the brain. Brain and Behavioral Sciences, 15, 183-247.
Dijkstra, T. M., Cornilleau-Peres, V., Gielen, C. C., & Droulez, J. (1995). Perception of three-dimensional shape from ego- and object-motion: comparison between small- and large-field stimuli. Vision Res, 35(4), 453-462.
Ditchburn, R. W. (1973). Eye-movements and visual perception. Oxford: Clarendon Press.
Dolezal, H. (1982). Living in a World Transformed. New York: Academic Press.
Duhamel, J. R., Colby, C. L., & Goldberg, M. E. (1992). The updating of the representation of visual space in parietal cortex by intended eye movements. Science, 255(5040), 90-92.
Durgin, F. H., Tripathy, S. P., & Levi, D. M. (1995). On the filling in of the visual blind spot: Some rules of thumb. Perception, 24(7), 827-840.
Easton, R. D. (1992). Inherent problems of attempts to apply sonar and vibrotactile sensory aid technology to the perceptual needs of the blind. Optom Vis Sci, 69(1), 3-14.
Edelman, G. M. (1989). The Remembered Present. New York: Basic Books.
Farah, M. J. (1989). The neural basis of mental imagery. Trends Neurosci, 12(10), 395-399.
Fisher, E., Haines, R. F., & Price, T. A. (1980). Head-up transition behavior of pilots with and without head-up display in simulated low-visibility approaches (Technical Paper ). Washington, D.C.: NASA.
Fishman, M. C., & Michael, C. R. (1973). "Integration of auditory information in cat's visual cortex". Vision Research, 13, 1415.
Gauthier, G. M. (1976). Alterations of the human vestibulo-ocular reflex in a simulated dive at 62 ATA. Undersea Biomed Res, 3(2), 103-112.
Gauthier, G. M., & Robinson, D. A. (1975). Adaptation of the human vestibuloocular reflex to magnifying lenses. Brain Res, 92(2), 331-335.
Gerrits, H. J. (1978). Differences in peripheral and foveal effects observed in stabilized vision. Exp Brain Res, 32(2), 225-244.
Gerrits, H. J. M. (1967). Observations with stabilized retinal images and their neural correlates. A theory on the perception of constant brightness. , Catholic University of Neijmegen, Neijmegen.
Gibson, J. J. (1966). The senses considered as perceptual systems. Boston: Houghton Mifflin.
Gibson, J. J. (1982). Notes on affordances. In E. Reed & R. Jones (Eds.), Reasons for realism: Selected essays of James J. Gibson . Hillsdale, N.J.: Erlbaum.
Goodale, M. A., & Milner, A. D. (1992). Separate visual pathways for perception and action. Trends in Neurosciences, 15(1), 20-25.
Gorea, A. (1991). Thoughts on the Specific Nerve Energy. In A. Gorea (Ed.), Representations of Vision. Trends and Tacit Assumptions in Vision Research (pp. 219-229). Cambridge: Cambridge University Press.
Gray, C. M., & Singer, W. (1989). Stimulus-specific neuronal oscillations in orientation columns of cat visual cortex. Proc Natl Acad Sci U S A, 86(5), 1698-1702.
Greenhouse, D. S., & Cohn, T. E. (1991). Saccadic suppression and stimulus uncertainty. J Opt Soc Am [A], 8(3), 587-595.
Gregory, R. L. (1973). Eye and brain. The psychology of seeing. New York: McGraw Hill.
Grüsser, O. J. (1986). Interaction of efferent and afferent signals in visual perception. A history of ideas and experimental paradigms. Acta Psychol (Amst), 63(1-3), 3-21.
Guarniero, G. (1974). Experience of tactile vision. Perception, 3(1), 101-104.
Guarniero, G. (1977). Tactile Vision: a personal view. Visual Impairment and Blindness, 125-130.
Haenny, P. E., Maunsell, J. H. R., & Schiller, P. H. (1988). State dependent activity in monkey visual cortex. Experimental Brain Research, 69, 245-259.
Haines, R. F. (1991). A breakdown in simultaneous information processing. In G. Obrecht & L. W. Stark (Eds.), Presbyopia research. From molecular biology to visual adaptation (pp. 171-175). New York: Plenum Press.
Hameroff, S. R. (1994). Quantum coherence in microtubules: A neural basis for emergent consciousness? Journal of Consciousness Studies, 1, 91-118.
Harman, G. (1990). The intrinsic quality of experience. Philosophical Perspectives, 4, 31-52.
Harris, C. S. (1965). Perceptual adaptation to inverted, reversed, and displaced vision. Psychol Rev, 72(6), 419-444.
Harris, C. S. (1980). Insight or out of sight?: Two examples of perceptual plasticity in the human adult. In C. S. Harris (Ed.), Visual coding and adaptability . Hillsdale, N.J.: Lawrence Erlbaum Associates.
Harvey, M. (1995). Translation of 'Psychic paralysis of gaze, optic ataxia, and spatial disorder of attention' by Rudolph Balint. Cognitive Neuropsychology, 12, 261-282.
Hatwell, Y. (1986). Toucher l'espace: la main et la perception tactile de l'espace. Lille, France: Presses Universitaires de Lille.
Hayes, S. P. (1935). Facial vision or the sense of obstacles. Watertown, Md.: Perkins Publications.
Heil, J. (1983). Perception and Cognition. Berkeley, Ca.: University of California Press.
Held, R. (1980). The rediscovery of adaptability in the visual system: Effects of extrinsic and intrinsic chromatic dispersion. In C. S. Harris (Ed.), Visual Coding and Adaptability . Hillsdale, N.J.: Lawrence Erlbaum Associates.
Helmholtz, H. v. (1925/1909). Physiological optics (Vol. 3); (J.P.C. Southall, Trans.) (Southall, J.P.C., Trans.). Rochester, New York: Optical Society of America.
Hochberg, J. (1970). In the mind's eye. In R. N. Haber (Ed.), Contemporary theory and research in visual perception .
Horn, G., & Hill, R. M. (1969). Modifications of the receptive field of cells in the visual cortex occurring spontaneously and associated with bodily tilt. Nature, 221, 185-187.
Howard, I. P., & Templeton, W. B. (1966). Human spatial orientation. New York: Wiley.
Howells, T. H. (1944). The experimental development of color-tone synesthesia. Journal of Experimental Psychology, 34, 87-103.
Humphrey, G. K., Gurnsey, R., & Bryden, P. J. (1994). An examination of colour-contingent pattern aftereffects. Spatial Vision, 8(1), 77-94.
Ifukube, T., Sasaki, T., & Peng, C. (1991). A blind mobility aid modeled after echolocation of bats. IEEE Trans Biomed Eng, 38(5), 461-465.
Intraub, H., & Richardson, M. (1989). Wide-angle memories of close-up scenes. Journal of Experimental Psychology: Learning, Memory, & Cognition, 15(2), 179-187.
Iriki, A., Tanaka, M., & Iwamura, Y. (1996). Coding of modified body schema during tool use by macaque postcentral neurones. Neuroreport, 7(14), 2325-2330.
Irwin, D. E. (1991). Information integration across saccadic eye movements. Cognitive Psychology, 23, 420-456.
Irwin, D. E. (1993). Perceiving an integrated visual world. In D. E. Meyer & S. Kornblum (Eds.), Attention and performance XIV: A silver jubilee (pp. ????). Cambridge, Mass.: MIT Press ????
Irwin, D. E., & Andrews, R. V. (1996). Integration and accumulation of information across saccadic eye movements. In T. Inui & J. L. McClelland (Eds.), Attention and performance XVI: Information integration in perception and communication (pp. 125-155). Cambridge, MA, USA: Mit Press.
Irwin, D. E., & Gordon, R. D. (1998). Eye movements, attention, and transsaccadic memory. Visual Cognition, 5(1-2), 127-155.
Jackson, F. (1986). What Mary didn't know. The Journal of Philosophy, 83.
James, W. (1890/1950). The principles of psychology. New York: Dover Publications.
Jameson, D., & Hurvich, L. M. (1978). Dichromatic color language: "reds" and "greens" don't look alike but their colors do. Sensory Processes, 2, 146-155.
Jeannerod, M. (1975). Déficit visuel persistant chez les aveugles-nés opérés. Données cliniques et expérimentales. Année Psychologique, 1, 169-195.
Jeannerod, M. (1997). The cognitive neuroscience of action. Oxford, England UK: Blackwell Publishers, Inc.
Jenkins, W. M., Merzenich, M. M., Ochs, M. T., Allard, T., & Guic-Robles, E. (1990a). Functional reorganization of primary somatosensory cortex in adult owl monkeys after behaviorally controlled tactile stimulation. J Neurophysiol, 63(1), 82-104.
Jenkins, W. M., Merzenich, M. M., & Recanzone, G. (1990b). Neocortical representational dynamics in adult primates: implications for neuropsychology. Neuropsychologia, 28(6), 573-584.
Kahn, D., Pace-Schott, E. F., & Hobson, J. A. (1997). Consciousness in waking and dreaming: the roles of neuronal oscillation and neuromodulation in determining similarities and differences. Neuroscience, 78(1), 13-38.
Kahneman, D., Treisman, A., & Gibbs, B. J. (1992). The reviewing of object files: object-specific integration of information. Cognit Psychol, 24(2), 175-219.
Kelly, D. H. (1981). Disappearance of stabilized chromatic gratings. Science, 214(4526), 1257-1258.
Kelly, D. H. (1982). Motion and vision. IV. Isotropic and anisotropic spatial responses. J Opt Soc Am, 72(4), 432-439.
Kelso, J. A. S., & Kay, B. A. (1987). Information and control: A macroscopic analysis of perception-action coupling. In H. Heuer & A. F. Sanders (Eds.), Perspectives on perceptioin and action . Hillsdale, N.J.: Erlbaum.
Kingdom, F., & Moulden, B. (1992). A multi-channel approach to brightness coding. Vision Res, 32(8), 1565-1582.
Kirk, R. (1974). Zombies vs. materialists. Aristotelian Society Supplement, 48, 135-152.
Koenderink, J. J. (1984). The concept of local sign. In A. J. van Doorn, W. A. van de Grind, & J. J. Koenderink (Eds.), Limits in perception (pp. 495-547). Zeist, Netherlands: VNU Science Press.
Kohler, I. (1951). Über Aufbau und Wandlungen der Wahrnehmungswelt. Österreichische Akademie der Wissenschaften. Sitzungsberichte, philosophish-historische Klasse, 227, 1-118.
Kohler, I. (1967). Facial vision rehabilitated. In R.-G. Busnel (Ed.), Les systèmes sonars animaux (Vol. 1, pp. 187-196). Jouy-en-Josas, France: Laboratoire de Physiologie Acoustique.
Kosslyn, S. M. (1994). Image and brain. The resolution of the imagery debate. Cambridge, Mass.: MIT Press.
Kottenhoff, H. (1961). Was ist richtiges Sehen mit Unkehrbrillen und in welchem Sinne stellt sich das Sehen um? Meisenheim am Glan: A. Hain.
Kowler, E., & Steinman, R. M. (1977). The role of small saccades in counting. Vision Res, 17(1), 141-146.
Krauskopf, J. (1963). Effect of retinal image stabilization of the appearance of heterochromatic targets. Journal of Optical Society of America, 53, 741-744.
Laming, D. (1986). Sensory Analysis. London: Academic Press.
Laming, D. (1988). A re-examination of sensory analysis. Behavioral and Brain Sciences, 11, 316-339.
Lenay, C. (1997). Le mouvement des boucles sensori-motrices aux représentations cognitives et langagières. Paper presented at the Sixième Ecole d'Eté de l'Association pour la Recherche Cognitive.
Lenay, C., Canu, S., & Villon, P. (1997). Technology and perception: the contribution of sensory substitution systems. Paper presented at the Second International Conference on Cognitive Technology, Aizu, Japan.
Leopold, D. A., & Logothetis, N. K. (1996). Activity changes in early visual cortex reflect monkeys' percepts during binocular rivalry [see comments]. Nature, 379(6565), 549-553.
Leopold, D. A., & Logothetis, N. K. (1999). Multistable phenomena: changing views in perception. Trends in Cognitive Studies, 3(7), 254-265.
Levi, D. M., Klein, S. A., & Aitsebaomo, A. P. (1985). Vernier acuity, crowding and cortical magnification. Vision Res, 25(7), 963-977.
Levine, J. (1983). Materialism and qualia: The explanatory gap. Pacific Philosophical Quarterly, 64(354-361).
Lewis, D. (1988). What experience teaches. Paper presented at the Proceedings of the Russellian Society, Sydney, Australia.
Li, W., & Matin, L. (1990). Saccadic suppression of displacement: Influence of postsaccadic exposure duration and of saccadic stimulus elimination. Vision Research, 30(6), 945-955.
Li, W., & Matin, L. (1997). Saccadic suppression of displacement: Separate influences of saccade size and of target retinal eccentricity. Vision Research, 37(13), 1779-1797.
Lillo, Davies, Collado, Ponte, & Vitini. (1998). . Paper presented at the ECVP.
Livingstone, M., & Hubel, D. (1988). Segregation of form, color, movement and depth: anatomy, physiology and perception. Nature, 240, 740-749.
Llinas, R., & Ribary, U. (1993). Coherent 40-Hz oscillation characterizes dream state in humans. Proc Natl Acad Sci U S A, 90(5), 2078-2081.
Logothetis, N. K. (1998). Single units and conscious vision. Philos Trans R Soc Lond B Biol Sci, 353(1377), 1801-1818.
Logothetis, N. K., Leopold, D. A., & Sheinberg, D. L. (1996). What is rivalling during binocular rivalry? [see comments]. Nature, 380(6575), 621-624.
Mack, A., & Bachant, J. (1969). Perceived movement of the afterimage during eye movements. Perception & Psychophysics, 6, 379-384.
Mack, A., & Rock, I. (1998). Inattentional blindness. Cambridge, MA, USA: The Mit Press.
Mack, A., Tang, B., Tuma, R., Kahn, S., & Rock, I. (1992). Perceptual organization and attention. Cognitive Psychology, 24(4), 475-501.
MacKay, D. M. (1962). Theoretical models of space perception. In C. A. Muses (Ed.), Aspects of the theory of artificial intelligence (pp. 83-104). New York: Plenum Press.
MacKay, D. M. (1967). Ways of looking at perception. In W. Wathen-Dunn (Ed.), Mopdels for the perception of speech and visual form (pp. 25-43). Cambridge, Mass.: MIT Press.
MacKay, D. M. (1973). Visual stability and voluntary eye movements. In R. Jung (Ed.), Handbook of sensory physiology, Vol. VII/3A (pp. 307-331). Berlin: Springer.
Marcos, S., Navarro, R., & Artal, P. (1996). Coherent imaging of the cone mosaic in the living human eye. J Opt Soc Am A, 13(5), 897-905.
Marks, L. E. (1978). The unity of the senses. Interrelations among the modalities. New York: Academic Press.
Matin, E. (1974). Saccadic suppression: a review and an analysis. Psychol Bull, 81(12), 899-917.
Matin, L. (1972). Eye movements and perceived visual direction. In D. Jameson & L. M. Hurvich (Eds.), Handbook of sensory physiology (Vol. VII/4: Visual Psychophysics, pp. 331-380).
Matin, L. (1986). Visual localization and eye movements. In K. R. Boff, L. Kaufman, & J. P. Thomas (Eds.), Handbook of perception and human performance (Vol. 1, pp. 20.21-20.45). New York: Wiley.
McCollough, C. (1965a). Color adaptation of edge-detectors in the human visual system. Science, 149(1115-1116).
McCollough, C. (1965b). The conditioning of color perception. American Journal of Psychology, 78, 362-368.
McConkie, G. W., & Currie, C. B. (1996). Visual stability across saccades while viewing complex pictures. Journal of Experimental Psychology: Human Perception & Performance, 22(3), 563-581.
McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264(5588), 746-748.
Merleau-Ponty, M. (1968). Résumés de cours au Collège de France. Paris: Gallimard.
Merzenich, M. M., Nelson, R. J., Kaas, J. H., Stryker, M. P., Jenkins, W. M., Zook, J. M., Cynader, M. S., & Schoppmann, A. (1987). Variability in hand surface representations in areas 3b and 1 in adult owl and squirrel monkeys. J Comp Neurol, 258(2), 281-296.
Merzenich, M. M., Nelson, R. J., Stryker, M. P., Cynader, M. S., Schoppmann, A., & Zook, J. M. (1984). Somatosensory cortical map changes following digit amputation in adult monkeys. J Comp Neurol, 224(4), 591-605.
Milner, A. D., & Goodale, M. A. (1995). The visual brain in action. Oxford, England UK: Oxford University Press.
Milner, A. D., Perrett, D. I., Johnston, R. S., Benson, P. J., Jordan, T. R., Heeley, D. W., & al., e. (1991). Perception and action in "visual form agnosia". Brain, 114, 405-428.
Milner, P. M. (1974). A model for visual shape recognition. Psychological Review, 81, 521-535.
Minsky, M. (1988). The society of mind: Simon & Schuster.
Montag. (1994). .
Moran, J., & Desimone, R. (1985). Selective attention gates visual processing in extrastriate cortex. Science, 229, 782-784.
Morell, F. (1972). Visual system's view of acoustic space. Nature, 238, 44-46.
Morgan, M. J. (1977). Molyneux's question. Vision, touch and the philosophy of perception. Cambridge: Cambridge University Press.
Müller, J. (1838). Handbuch der Physiologie des Menschen. (Vol. V). Coblenz: Hölscher.
Nazir, T. A., & O'Regan, J. K. (1990). Some results on translation invariance in the human visual system. Spatial Vision, 5(2), 81-100.
Neisser, U. (1976). Cognition and reality. Principles and implications of cognitive psychology. San Francisco: W.H. Freeman.
Neisser, U., & Becklen, R. (1975). Selective looking: attending to visually-specified events. Cognitive Psychology, 7, 480-494.
Nemirov, L. (1984). Physicalism and the cognitive role of acquaintance. In W. G. Lycan (Ed.), Mind and Cognition: A Reader (pp. 490-499). Oxford: Blackwell's.
Noë, A. (submitted-a). Experience and the active mind. Synthese.
Noë, A. (submitted-b). On what we see. .
Noë, A., Pessoa, L., & Thompson, E. (2000). Beyond the grand illusion: What change blindness really teaches us about vision. Visual Cognition, 7, 93-106.
Noton, D., & Stark, L. (1971). Eye movements and visual perception. Sci Am, 224(6), 35-43.
O'Keefe, J., Burgess, N., Donnett, J. G., Jeffery, K. J., & Maguire, E. A. (1998). Place cells, navigational accuracy, and the human hippocampus. Philos Trans R Soc Lond B Biol Sci, 353(1373), 1333-1340.
O'Regan, J. K. (1984). Retinal versus extraretinal influences in flash localization during saccadic eye movements in the presence of a visible background. Perception & Psychophysics, 36(1), 1-14.
O'Regan, J. K. (1992). Solving the "real" mysteries of visual perception: The world as an outside memory. Canadian Journal of Psychology, 46(3), 461-488.
O'Regan, J. K., Deubel, H., Clark, J. J., & Rensink, R. A. (2000). Picture changes during blinks: looking without seeing and seeing without looking. Visual Cognition, in press.
O'Regan, J. K., Rensink, J. A., & Clark, J. J. (1996). "Mud splashes" render picture changes invisible. Invest. Ophthalmol. Vis. Sci., 37, S213.
O'Regan, J. K., Rensink, R. A., & Clark, J. J. (1999). Change-blindness as a result of 'mudsplashes'. Nature, 398, 34.
Palmer, S. E. (1999). Color, Consciousness and the isomorphism constraint. Behavioral and Brain Science, 6, in press.
Paradiso, M. A., & Nakayama, K. (1991). Brightness perception and filling-in. Vision Res, 31(7-8), 1221-1236.
Pascual-Leone, A., Cammarota, A., Wassermann, E. M., Brasil-Neto, J. P., Cohen, L. G., & Hallett, M. (1993). Modulation of motor cortical outputs to the reading hand of braille readers. Ann Neurol, 34(1), 33-37.
Penrose, R. (1994). Shadows of the Mind. Oxford: Oxford University Press.
Pessoa, L., Thompson, E., & Noë, A. (1998). Finding out about filling in: a guide to perceptual completion for visual science and the philosophy of perception. Behavioral and Brain Sciences, 21(6), 723-802.
Peterhans, E., & von der Heydt, R. (1989). Mechanisms of contour perception in monkey visual cortex. II. Contours bridging gaps. J Neurosci, 9(5), 1749-1763.
Platt, J. R. (1960). How we see straight lines. Scientific American, 202(6), 121-129.
Potter, M. C. (1976). Short-term conceptual memory for pictures. J Exp Psychol [Hum Learn], 2(5), 509-522.
Pouget, A., & Sejnowski, T. J. (1997). A new view of hemineglect based on the response properties of parietal neurones. Philos Trans R Soc Lond B Biol Sci, 352(1360), 1449-1459.
Pylyshyn, Z. W. (1988). The imagery debate: Analogue media versus tacit knowledge. In E. E. S. Allan M. Collins (Ed.), Readings in cognitive science: A perspective from psychology and artificial intelligence. (pp. 600-614): Morgan Kaufmann, Inc, San Mateo, CA, US.
Radeau, M. (1997). Du ventriloque à l'embryon: une réponse à Molyneux. In J. Proust (Ed.), Perception et intermodalité. Approches actuelles de la question de Molyneux (pp. 223-252). Paris: PUF.
Radeau, M., & Bertelson, P. (1974). The after-effects of ventriloquism. Q J Exp Psychol, 26(1), 63-71.
Ramachandran, V. S. (1992). Filling in gaps in perception: I. Current Directions in Psychological Science, 1(6), 199-205.
Ramachandran, V. S. (1995). Filling in gaps in logic: reply to Durgin et al. Perception, 24(7), 841-845.
Ramachandran, V. S., & Blakeslee, S. (1998). Phantoms in the brain: William Morrow & Co.
Ramachandran, V. S., Cobb, S., & Levi, L. (1994a). Monocular double vision in strabismus [letter]. Neuroreport, 5(12), 1418.
Ramachandran, V. S., Cobb, S., & Levi, L. (1994b). The neural locus of binocular rivalry and monocular diplopia in intermittent exotropes. Neuroreport: an International Journal for the Rapid Communication of Research in Neuroscience, 5(9), 1141-1144.
Ramachandran, V. S., & Gregory, R. L. (1991). Perceptual filling in of artificially induced scotomas in human vision. Nature, 350(6320), 699-702.
Rensink, R. A., O'Regan, J. K., & Clark, J. J. (1997). To see or not to see: The need for attention to perceive changes in scenes. Psychological Science, 8(5), 368-373.
Rensink, R. A., O'Regan, J. K., & Clark, J. J. (2000). On the failure to detect changes in scenes across brief interruptions. Visual Cognition.
Rice, C. E. (1966). The human sonar system, Vol II. Paper presented at the Animal sonar systems. Biology and bionics. (NATO Advanced study Institute), Frascati, Italy.
Richards, W. (1969). Saccadic suppression. Journal of the Optical Society of America, 59, 617-623.
Ridder, W. H., 3rd, & Tomlinson, A. (1997). A comparison of saccadic and blink suppression in normal observers. Vision Res, 37(22), 3171-3179.
Rodriguez, E., George, N., Lachaux, J.-P., Martinerie, J., Renault, B., & Varela, F. J. (1999). Perception's shadow: long-distance synchronization of
human brain activity. Nature, 397, 430-433.
Rogers, B., & Graham, M. (1979). Motion parallax as an independent cue for depth perception. Perception, 8(2), 125-134.
Rogers, S., & Rogers, B. J. (1992). Visual and nonvisual information disambiguate surfaces specified by motion parallax. Percept Psychophys, 52(4), 446-452.
Rolls, E. T. (1992). Neurophysiological mechanisms underlying face processing within and beyond the temporal cortical visual areas. Philos Trans R Soc Lond B Biol Sci, 335(1273), 11-20; discussion 20-11.
Rolls, E. T., & O'Mara, S. M. (1995). View-responsive neurons in the primate hippocampal complex. Hippocampus, 5(5), 409-424.
Rossetti, Y., Koga, K., & Mano, T. (1993). Prismatic displacement of vision induces transient changes in the timing of eye hand coordination. Perception & Psychophysics, 54(3), 355-364.
Rozenblium Iu, Z., & Korniushina, T. A. (1991). [Monocular diplopia]. Vestn Oftalmol, 107(3), 39-44.
Ryle, G. (1949/1990). The concept of mind. London: Penguin Books.
Sadato, N., Pascual-Leone, A., Grafman, J., Deiber, M. P., Ibanez, V., & Hallett, M. (1998). Neural networks for Braille reading by the blind [see comments]. Brain, 121(Pt 7), 1213-1229.
Sampaio, E. (1989). Is there a critical age for using the sonicguide with blind infants? Journal of visual impairment and blindness, 83, 105-108.
Sampaio, E. (1995). Les substitutions sensorielles adaptées aux déficits visuels importants. In A. B. Assimacopoulos & A. Saffran (Eds.), Le déficit visuel. Des fondements neurophysiologoques à la pratique de la réadaptation . Paris: Masson.
Sampaio, E., & Dufier, J. L. (1988). Suppléance sensorielle électronique pour les jeunes enfants aveugles. Journal Français Ophtalmologique, 11(2), 161-167.
Schlingensiepen, K.-H., Campbell, F. W., Legge, G. E., & Walker, T. D. (1986). The importance of eye movements in the analysis of simple patterns. Vision Research, 26(7), 1111-1117.
Searle, J. (1980). Minds, brains, and programs. Behavioral and Brain Sciences, 3, 417-424.
Searle, J. R. (1997). The mystery of consciousness. New York Review of Books(xvi), 224.
Shebilske, W. L. (1977). Visuomotor coordination, visual direction and position constancies. In W. Epstein (Ed.), Stability and constancy in visual perception: Mechanisms and processes : Wiley.
Shoemaker, S. (1970). The inverted spectrum. Journal of Philosophy, 7(4), 357-381.
Simons, D. (2000). Current approaches to change blindness. Visual Cognition, in press.
Simons, D. J., & Chabris, C. F. (in press). Gorillas in our midst: sustained inattentional blindness for dynamic events. Perception.
Simons, D. J., & Levin, D. T. (1997). Change Blindness. Trends in Cognitive Sciences, 1(7), 261-267.
Singer, W. (1993). Synchronization of cortical activity and its putative role in information processing and learning. Annu Rev Physiol, 55, 349-374.
Singer, W., & Gray, C. M. (1995). Visual feature integration and the temporal correlation hypothesis. Annu Rev Neurosci, 18, 555-586.
Smith, K. U., & Smith, W. M. (1962). Perception and Motion. Philadelphia: W.B. Saunders.
Sperry, R. W. (1952). Neurology and the mind-brain problem. American Scientist, 40, 291-311.
Stelmach, L. B., Bourassa, C. M., & di Lollo, V. (1984). Detection of stimulus change: The hypothetical roles of visual transient responses. Perception & Psychophysics, 35(3), 245-255.
Sterr, A., Muller, M. M., Elbert, T., Rockstroh, B., Pantev, C., & Taub, E. (1998). Perceptual correlates of changes in cortical representation of fingers in blind multifinger Braille readers. J Neurosci, 18(11), 4417-4423.
Stratton, G. M. (1897). Vision without inversion of the retinal image. Psychological Review, 4, 341-360, 463-481.
Strawson, P. F. (1974). Imagination and perception, Freedom and resentment and other essays . London: Methuen.
Strelow, E. R., & Brabyn, J. A. (1982). Locomotion of the blind controlled by natural sound cues. Perception, 11(6), 635-640.
Tastevin, J. (1937). En partant de l'expérience d'aristote les déplacements artificiels des parties du corps ne sont pas suivis par le sentiment de ces parties ni par les sensations qu'on peut y produire. L'Encéphale, XXXII(2), 57-84.
Taylor, J. G. (1962). The behavioral basis of perception. New Haven, Conn.: Yale University Press.
Teller, D. Y. (1984). Linking propositions. Vision Research, 24, 1233-1246.
Teller, D. Y., & Pugh Jr., E. N. (1983). Linking propositions in color vision. In J. D. Mollon & L. T. Sharpe (Eds.), Colour vision: physiology and psychophysics . London: Academic Press.
Theeuwes, J. (1991). Exogenous and endogenous control of attention: The effect of visual onsets and offsets. Perception & Psychophysics, 49(1), 83-90.
Thompson, E. (1995). Colour Vision: A study in cognitive science and the philosophy of perception. London: Routledge Press.
Thompson, E., Palacios, A., & Varela, F. J. (1992). Ways of coloring: Comparative color vision as a case study for cognitive science. Behavioral & Brain Sciences, 15(1), 1-74.
Thompson, E., & Varela, F. J. (forthcoming). Neural correlates of consciousness: A neurophenomenological perspective. .
Thorpe, S., Fize, D., & Marlot, C. (1996). Speed of processing in the human visual system. Nature, 381(6582), 520-522.
Todorovic, D. (1987). The Craik-O'Brien-Cornsweet effect: new varieties and their theoretical implications. Percept Psychophys, 42(6), 545-560.
Toet, A., & Levi, D. M. (1992). The two-dimensional shape of spatial interaction zones in the parafovea. Vision Res, 32(7), 1349-1357.
Tolhurst, D. J. (1975). Sustained and transient channels in human vision. Vision Research, 15, 1151-1155.
Treue, S., & Maunsell, J. H. R. (1996). Attentional modulation of visual motion processing in cortical areas MT and MST. Nature, 382, 539-541.
Turvey, M. T., Shaw, R. E., Reed, E. S., & Mace, W. M. (1981). Ecological laws of perceiving and acting: in reply to Fodor and Pylyshyn (1981). Cognition, 9(3), 237-304.
Uchikawa, K., & Sato, M. (1995). Saccadic suppression of achromatic and chromatic responses measured by increment-threshold spectral sensitivity. J Opt Soc Am A, 12(4), 661-666.
Ungerleider, L. G., & Mishkin, M. (1992). Two cortical visual systems. In D. J. Ingle, M. A. Goodale, & Mansfield (Eds.), Analysis of visual behavior . Cambridge, Mass.: MIT Press.
Varela, F. J. (1984). Living ways of sense-making. A middle path for neuroscience. Paper presented at the Disorder and order: proceedings of the Stanford International Symposium.
Varela, F. J., Thompson, E., & Rosch, E. (1991). The embodied mind: Cognitive science and human experience. Cambridge, MA, USA: Mit Press.
von der Heydt, R., & Peterhans, E. (1989). Mechanisms of contour perception in monkey visual cortex. I. Lines of pattern discontinuity. J Neurosci, 9(5), 1731-1748.
von der Heydt, R., Peterhans, E., & Baumgartner, G. (1984). Illusory contours and cortical neuron responses. Science, 224(4654), 1260-1262.
von der Malsburg, C. (1983). How are nervous structures organized? In E. Basar, H. Flohr, H. Haken, & A. J. Mandell (Eds.), Synergetics of the brain (pp. 238-249). Berlin: Springer.
Wagemans, J., & de Weert, C. M. M. (1992). Ways of coloring the ecological approach. Behavioral and Brain Sciences, 15(1), 54-56.
Wall, J. T., Kaas, J. H., Sur, M., Nelson, R. J., Felleman, D. J., & Merzenich, M. M. (1986). Functional reorganization in somatosensory cortical areas 3b and 1 of adult monkeys after median nerve repair: possible relationships to sensory recovery in humans. J Neurosci, 6(1), 218-233.
Warren, D. H., Welch, R. B., & McCarthy, T. J. (1981). The role of visual-auditory "compellingness" in the ventriloquism effect: implications for transitivity among the spatial senses. Percept Psychophys, 30(6), 557-564.
Weiskrantz, L. (1986). Blindsight: A case study and implications. Oxford: Oxford University Press.
Weisskrantz, L. (1997). Consciousness lost and found: a neuropsychological exploration. Oxford: Oxford University Press.
Welch, R. B., Bridgeman, B., Anand, S., & Browman, K. E. (1993). Alternating prism exposure causes dual adaptation and generalization to a novel displacement. Percept Psychophys, 54(2), 195-204.
Wolfe, J. M. (1997). Visual Experience: Less than you think, more than you remember. In C. Taddei-Ferretti (Ed.), Neuronal basis and psychological aspects of consciousness. (pp. ???). Singapore:: World Scientific.
Wolfe, J. M. (1999). Inattentional amnesia. In V. Coltheart (Ed.), Fleeting memories . Cambridge, Mass.: MIT Press.
Wolfe, J. M., Klempen, N., & Dahlen, K. (1999). Post-attentive vision. Journal of Experimental Psychology: Human Perception & Performance, in press.
Worchel, P., Mauney, J., & Andrew, J. (1950). The perception of obstacles by the blind. Journal of Experimental Psychology, 40, 746-751.
Yantis, S. (1998). Control of visual attention. In H. Pashler (Ed.), Attention (pp. 223-256). Hove, England UK: Psychology Press/Erlbaum (Uk) Taylor & Francis.
Yap, Y. L., Levi, D. M., & Klein, S. A. (1989). Peripheral positional acuity: retinal and cortical constraints on 2-dot separation discrimination under photopic and scotopic conditions. Vision Res, 29(7), 789-802.
Zeki, S. (1993). A vision of the brian. Oxford: Blackwell.
Zipser, D., & Anderson, R. (1988). A back-propagation programmed
network that simulates response properties of a subset of posterior parietal
neurons.
Nature, 331(679-684).