Localisation, form recognition and emergence of sensation in sensory substitution

Ongoing research in sensory substitution

Sensorimotor contingency group

Laboratoire de Psychologie Expérimentale, CNRS, Université Paris 5 (dir. J. K. O'Regan)

Summer 2003

Within the context of a sensorimotor approach to understanding the nature of sensory experience, our interest in sensory substitution is in studying the process by which Subjects attain mastery of sensory substitution devices. We expect there to be approximately five stages. The first stage, "contact", involves the Subject learning the sensorimotor skill necessary to maintain and control perceptual contact with a stimulus. The second stage, "exteriorisation" involves the Subject coming to experience the stimulus as no longer located at or in the sensor that conveys it (eye, ear, skin, etc), but as corresponding to an outside physical entity. The third stage, "spatialization" involves attribution of a spatial location to the experienced entity, with coherent understanding of its relation to the body. "Comprehension" involves being able not simply to spatially locate, but also to recognize the entity as a perceptual object among possible alternate objects. "Immersion" is the state where the Subject possesses all these abilities and feels he or she is physically immersed in an environment populated by objects that can be perceived through the substituted sensory modality.

Experiments in progress:

1. Visual to auditory sensory substitution (M. Auvray, J.K. O'Regan, S. Hanneton [Lab. Neurophysique & Phys. Syst. Moteur, Univ René Descartes]):

a) In this project six sighted Subjects, were blindfolded and tested with P. Meijer's "The vOICe", which scans visual scenes and transforms them into auditory "landscapes". We attempted to understand how, in advancing through the different stages, subjects pass from a kind of deductive reasoning to a form of immediate apprehension of what is perceived. We found that Subjects rapidly acquired contact, exteriorisation and spatialization, and, after 15 hours of training were able to accurately categorize 10 different household objects (shoe, flower, book, ...). However, in most Subjects but not all, when presented with previously unexperienced variants of the learnt objects, Subjects confused them with the originally learnt objects (e.g. differently shaped shoe). (A film describing this project will soon be available online on the web site http://malika-auvray.com)

b) A new project has started in which we have modified P. Meijer's device in a way to be more consistent with our hypotheses about sensorimotor contingencies. We hope this device will be much easier for Subjects to adapt to.

2. Auditory or visual to tactile sensory substitution (A. Bompas, J.K. O'Regan, V. Hayward [CIM, McGill], J. Clark [CIM, McGill]):

In this project we are constructing a simple device which can have either auditory or visual input, and which provides tactile stimulation to the forehead. Our purpose is not to make a practically useful substitution device, but to test the hypothesis, put forward under our sensorimotor theory, that the perceived quality of a sensation does not so much depend on the input channel through which sensory information is provided, but on the way sensory input is affected by the Subject's body motions, that is, what we call the sensorimotor contingencies. We shall thus ask subjects to judge the perceived quality (more auditory-like/more visual-like?) of tactile stimulation under conditions when this stimulation obeys auditory-like laws of contingency, and when the stimulation obeys visual-like laws of contingency.

3. Visual to tactile sensory supplementation (M. Auvray, C. Lenay [UTC, Compiègne])

The purpose here, again, is not to make a practical sensory substitution device, but to study the role of proprioception in the mastery of such devices. We use "Tactos", the very simple device being studied by C. Lenay at UTC Compiègne, in which a subject obtains tactile feedback via a 4 x 4 Braille pad about line drawings he or she explores by means of a graphics pen on a graphics tablet. Proprioception, that is, information concerning the movements of the pen on the graphics tablet, is provided to the Subject by means of a second braille pad. We are using different ways of coding the proprioceptive information, and investigating to what extent this information can help Subjects master the device.

4. Extraction of intrinsic spatial invariants in sensorimotor co-variation (D. Philipona, J.K. O'Regan, J.P. Nadal [ENS], O. Coenen [Sony CSL]):

We have developed a mathematical approach based on differential geometry which allows intrinsic invariants to be extracted from sensorimotor laws. In concrete terms we have implemented an algorithm which an organism (biological or robot) can use to extract the geometrical properties of outside physical space, by observing the relation between its motor outputs and its sensory inputs. The algorithm works independently of the code in which information is encoded, that is, it uses no a priori knowledge of the nature ot the organism's sensors (these can be visual, auditory, tactile, X-ray, etc.) or effectors (these can be muscles, pulleys, motors, etc.), or of the way the sensors and effectors are connected to the brain.(cf. D. Philipona, J.K.O'Regan & J.-P. Nadal. Neural Computation, in press; pdf. available on http://nivea.psycho.univ-paris5.fr).

---------

RECENT PUBLICATIONS

J. KEVIN O'REGAN

(to download these, see the web site http://nivea.psycho.univ-paris5.fr)

Skill, corporality and alerting capacity in an account of sensory consciousness.J. K. O'Regan, E. Myin & A. Noë. to be submitted.

Phenomenal consciousness explained (better) in terms of corporality and alerting capacity.J. K. O'Regan, E. Myin & A. Noë. to be submitted.

Is there something out there? Inferring space from sensorimotor dependencies.D. Philipona, J.K.O'Regan & J.-P. Nadal. Neural Computation, in press.

Voir avec les oreilles: Enjeux de la substitution sensorielle. by Auvray, M. & O'Regan, J.K. Pour la Science, 2003

Towards an analytic phenomenology: The concepts of bodiliness and grabbiness.J.K. O'Regan, E. Myin & A. Noë. In A. Carsetti (Ed.) Proceedings of the International Colloquium : "Seeing and Thinking. Reflections on Kanizsa's Studies in Visual Cognition". Univ. Tor Vergata, Rome, June 8-9, 2001. Kluwer, in press.

Perceptual consciousness, access to modality and skill theories: A way to naturalise phenomenology?E. Myin & J. K. O'Regan, in Journal of Consciousness Studies, 9, No. 1, 2002, pp. 27-45.

A sensorimotor account of vision and visual consciousness.J.K.O'Regan & A. Noë, in Behavioral and Brain Sciences, 2001, 24(5), 939-1011.

Authors' Response: Acting out our sensory experience.J.K.O'Regan & A. Noë, in Behavioral and Brain Sciences, 2001, 24(5), 1011-1031.

Experience is not something we feel but something we do: a principled way of explaining sensory phenomenology, with Change Blindness and other empirical consequences, expanded version of talk given by J.K.O'Regan & A. Noë at ASSC 4, Brussels, June 29-July 2, 2000, and by J.K. O'Regan at Bressanone, Jan 21-26, 2001.

What it is like to see: A sensorimotor theory of visual experience.J.K.O'Regan & A. Noë, Synthese, 2001, 129, 1, pp. 79-103.

Blindness to scene changes caused by "mudsplashes".J.K. O'Regan, R.A. Rensink, R.A. & J.J. Clark, in Nature, 398, 34, 1999.

Change blindness.J. K. O'Regan, in Encyclopedia of Cognitive Science, Nature Publishing group, in press.

Perception, Attention and the Grand Illusion.Alva Noë & J. K. O'Regan

PSYCHE, 6(15), October 2000

MALIKA AUVRAY

(to download more information see the web site http://malika-auvray.com)

L’influence des facteurs sémantiques sur la cécité aux changements progressifs dans les scènes visuelles, Malika Auvray, J. Kevin O’Regan (Année Psychologique, février 2003, 103, 9-32)

Bergson. une théorie sensorimotrice de la perception (Psychologie et histoire, 2003, vol. 4, 61- 100.)

Apprendre à utiliser un dispositif de substitution sensorielle : les différentes étapes, Malika Auvray, J. Kevin O’Regan, Sylvain Hanneton. (actes du séminaire "espaces d'action, espaces de perception", UTC, janvier 2003)

Voir avec les Oreilles. Enjeux de la substitution sensorielle, Malika Auvray, J. Kevin O’Regan, (Pour la science, avril 2003)

ALINE BOMPAS

Wolfe, J.M., Oliva, A., Horowitz, T.S., Butcher, S.J. & Bompas, A. Segmentation of Objects from Backgrounds in Visual Search Tasks. Vision Research 42(28): 2985-3004 DEC 2002

Bompas, A & O’Regan, J.K. Influence of sensori-motor contingencies on colour perception. (Abstract) Journées Annuelles du RESCIF, 2001.

O’Regan, J.K., Bompas, A. Implications of a sensori-motor theory of visual phenomenality for change blindness, colour perception, sensory substitution and the notion of space and object. (Abstract)Workshop on the Genesis of the Notion of Space in Humans and Machines, 2001

O'Regan, J.K., Clark, J., Bompas, A. Implications of a sensorimotor theory of vision for scene perception and colour sensation. (Abstract). Perception, 2001, 30, supplement, 94

Bompas, A., Clark, J., O'Regan, J.K. Color Perception in the Sensorimotor Contingency Theory. (Abstract). Perception, 2002, supplement.

Bompas, A & O’Regan, J.K. A Sensorimotor Approach of Color Perception. (Abstract). Perception, 2003, supplement.

DAVID PHILIPONA

(to download more information see the web site http://perso.enst.fr/~philipon)

Is there something out there? Inferring space from sensorimotor dependencies.D. Philipona, J.K.O'Regan & J.-P. Nadal. Neural Computation, in press.

Géométrie de l’espace physique et contingences sensorimotrices, D. Philipona, K. O'Regan, J.-P. Nadal, O.J.M.D. Coenen. (actes du séminaire "espaces d'action, espaces de perception", UTC, janvier 2003)

Comprendre les propriétés géométriques de l'espace à partir des contingences sensorimotrices - D. Philipona, K. O'Regan, J.-P. Nadal, O.J.M.D. Coenen ? (Abstract) Journées Annuelles du RESCIF, 2002.