Please see this link.
“FEEL” was a six-year ERC Advanced project led by J. Kevin O’Regan at the Université Paris Descartes, devoted to testing and developing the “sensorimotor” approach to consciousness. This approach purports to explain the “hard” problem of understanding the nature of “qualia”, that is to say the “feel” associated with perceptual experience: for example, why ‘red’ feels red rather than feeling like, say, the sound of a bell. One line of research in the project involved expanding the approach as a competitor to existing frameworks for understanding consciousness by situating it with respect to other “enactive” and more mainstream (e.g. representationalist) approaches, and by exploring and defending its unique explanatory advantages. As a first practical application of the approach, a mathematical line of research showed how the notion of space can emerge in a system that uses coincidence detection to capture sensorimotor invariants. This work had applications in robotic sensor calibration and gave rise to advances in deep learning. The sensorimotor theory also has much to say about the perceived nature of color, and numerous papers were published about how the approach relates to classical color theory. One interesting application concerned the well-known color-switching internet meme #theDress. Other practical applications of the FEEL project concerned “sensory substitution” or “sensory augmentation”, i.e. the possibility of using one sense modality to replace or augment another. In particular the project developed and tested a device that provides a “sense of North” using auditory input, and another device that augments perception of speech through tactile input. Finally the project investigated how 3-9 month old babies use sensorimotor invariants to understand the structure of their bodies, a question relevant not just to psychology but also to robotics. The “FEEL” project employed 9 postdocs, published more than 60 papers, and organised 7 international workshops. It generated an ERC “proof of concept” project “FeelSpeech” on sensory augmentation, and was closely involved with the EU FETopen project “GoalRobots”.
WP Philosophy:
Our main goal in the DoW was to situate the sensorimotor theory in relation to other theories and respond to criticism. With respect to enactivism, we showed that the sensorimotor approach offers a distinct approach from the ‘autopoietic’ and ‘radical’ enactive theories (Silverman, 2016; Ward et al., 2017; Villalobos & Silverman, 2017) and better ways of accounting for mental imagery (Foglia et al., 2015) and for the existence of consciousness (Degenaar et al., 2015; Silverman, 2017). Other work contrasted the theory with representationalist accounts such as the Higher Order Thought theory, arguing that it is better to appeal to bodily skill than internal representation (Degenaar et al., 2014; Silverman, 2015). In making this argument, we offered a newly expanded account of how non-representational exercise of sensorimotor skills determines perceptual phenomenology (Silverman, 2018). Lastly, we extended the notion of skill per se with a notion of improvisatory skills (Torrance & Schumann, 2018).
WP Maths:
In this WP, and following the DoW, we first addressed the question of how the notion of space can be discovered by a naive agent using undifferentiated sensorimotor information. We published papers showing how topological, group and metric properties of space manifest themselves as sensorimotor invariants (Laflaquière et al 2013, 2015, 2018) captured through a coincidence detecting algorithm (Terekhov and O’Regan 2013, 2015). As a practical application we applied this algorithm to the problem of visual sensor calibration (Montone et al 2015). Further progress on the project required developing methods for cumulative learning and we suggested a block-modular architecture for artificial neural networks (Terekhov et al 2015, Montone et al 2015, 2017a, 2017b). We also experimented with encoding more abstract spatial reasoning in neural networks using hyperdimensional computing (Montone et al 2017).
WP Color:
Our stated goal in this WP was to develop the seminal sensorimotor approach to color of Philipona and O’Regan. We started by characterising factors that determine sensory color singularities (Witzel et al., 2015), and showed that natural surfaces have particular properties that link their singularities to color categorisation and appearance (Witzel & O’Regan, 2014), further observing a strong link between metameric mismatching and colour constancy (Witzel et al., 2016). We produced a series of publications on the “color-switching dress” (internet meme #theDress) (Witzel et al., 2017a, b), on color constancy and color categorisation (Witzel, 2016; Witzel & Gegenfurtner, 2016), color preferences (Witzel, 2015), memory color effects (Witzel, 2016), on a clinical study about the dissociation between color perception and color knowledge (Siuda-Krzywicka et al., in press), on real-world search in color deficiencies (Kugler et al. 2015), and on the hue-heat-hypothesis (Huebner et al., 2016).
WP Sensory Substitution:
We designed two novel sensory augmentation devices (HearSpace and NaviEar) that provide information about geo-magnetic orientation to a user. Importantly, unlike prior work, our approach used a sensory coding scheme that built on natural sensory contingencies (“contingency-mimetics”), leading to faster learning and more genuine perceptual experiences (Schumann & O’Regan, 2017). Further, outdoor behavioral navigation experiments demonstrated that the NaviEar device is intuitive and easy to use (Witzel et al., to be resubmitted). Also within this WP we obtained an ERC Proof of Concept grant where we tried to use sensory substitution in a third device (FeelSpeech) to augment hearing through tactile stimulation. However we encountered unexpected difficulties (Rizza et al 2018). We now think that problems that previously prevented the development of effective sensory substitution devices might be overcome using our principle of contingency mimetics.
WP Babies:
Following the DOW, in the first two periods of the project we studied how 9-18 month old babies learn to extend their bodies by using a rake-like tool to get an out-of-reach toy (Fagard et al., 2015). We investigated several factors that influence learning: observation of others (Fagard et al., 2014; Somogyi et al. 2015; Rat-Fischer et al. 2014), understanding the intention of demonstrators (Esseily et al., 2013), and situations where the baby laughs (Esseily et al., 2015). The second half of the project studied how babies learn their own body structure. A first, “buzzer” technique using small coin vibrators placed at different body locations, showed that babies’ body knowledge progresses from upper to lower body (Somogyi et al. 2018). We confirmed and modelled this finding in collaboration with robotics groups (Hoffmann et al., 2017), with whom we then obtained a FETopen project (“GoalRobots”) (Mannella et al. 2018). A second, “bracelet” technique was also to be used, allowing visual, tactile or auditory stimuli to be triggered when the baby moved its limbs. However, despite requesting a one year extension of the FEEL project, we were not able to satisfactorily exploit the technique, even though we did publish a few results and literature reviews (Jacquey et al. under revision; Jacquey et al., submitted; Jacquey et al. submitted; Jacquey & O’Regan, forthcoming).
WP Philosophy:
The philosophically-founded insights of the sensorimotor approach allowed us to formulate novel ideas for skill-based interventions for dysfunctional attention control such as in ADHD (Clark, Schumann, Mostofsky, 2015). These ideas are currently being tested in a clinical NIH feasibility study by collaborators at the Kennedy Krieger Institute (Baltimore). Our publication was in the top 100 most visible publications of all Frontiers publications in 2015.
WP Maths:
Our block-modular architectures are a novel approach with potential for solving the problem of catastrophic forgetting (interference) which is currently an unsolved problem faced by existing artificial neural networks (see Terekhov et al 2015 and Montone et al 2015, 2017a, 2017b). Another innovation is that we have successfully combined the hyperdimensional computing approach with neural networks and applied it to a spatial reasoning task (Montone et al 2017).
WP Color:
One line of innovation consisted in improving previous methods of calculating sensory singularities (Witzel, Cinotti & O’Regan, 2015) and metamer mismatch volumes (Witzel, Van Alphen, Godau & O’Regan, 2015). We also developed a new, simple approach to create virtual reflectances through their basis functions (Witzel & O’Regan, ECVP2014). By showing that web-based experimentation can reproduce fine-grained memory color effects (Witzel 2016), we opened the path for investigating cultural and individual differences through web-based experimentation (Witzel O’Regan & Hansmann-Roth, 2017). We extended the nonparametric statistical McNemar test (Witzel, 2016) and developed a score for naming #theDress (and similar phenomena) based on binary principal component analyses with dummy variables (Witzel, O’Regan, & Hansmann-Roth, 2017). Finally, we produced two disambiguating versions of #theDress that have been used for experimental purposes and to explain the phenomenon of the dress to students and the public (see outreach).
WP Sensory Substitution:
We transferred the principle of bio-mimicry from research on invasive brain-machine interfaces to non-invasive use via sensory substitution and augmentation (“contingency-mimetics”, Schumann & O’Regan, 2017) using auditory signal processing with movement tracking in a wearable device (HearSpace). We also invented an App for smartphones, the NaviEar, that translates cardinal directions (North, East,...) into sound. Another novelty, related to our ERC PoC device FeelSpeech, was the discovery that true perceptual multimodal integration is much harder to obtain than usually claimed in the literature (Rizza et al 2018). In the spirit of contingency-mimetics, we were however able to show at least one instance of “true” audio-tactile integration, namely an onset in both modalities: tactile tap and auditory beep.
WP Babies:
To study how babies learn to know their own bodies, and in collaboration with developmental and robotic labs in New Orleans, Hamburg, Rome, and Genoa, we developed a new “buzzer” methodology in which we briefly put a small coin-sized vibrator on one limb of a baby, and we registered the baby’s reaction. This gave rise to several papers, collaborations, conference workshops and presentations. We also developed a second innovative wireless “bracelet” technique with our Rome collaborators and also partly within our FETopen GoalRobots project. The technique allows a baby’s limb motions to be registered online and used to produce contingent visual, auditory or tactile stimulation. Again, we have many publications and conference presentations resulting from this. In a side project, we collaborated with a lab in Berlin to test if adults can also learn to come to know their bodies (better), using a novel ‘artificial floor’ technique in fMRI to see if such improvement in body awareness involves neural areas related to action (Verrel et al., 2015). This paper was in the top 100 most visible publications in the Frontiers journals in 2015.
WP Philosophy:
Our cross-disciplinary activities included holding an international conference with leading researchers in embodied cognitive science, which helped promote fruitful interdisciplinary dialogue among psychologists, roboticists, cognitive scientist and philosophers both within and outside the embodied and enactive cognitive science traditions. Our philosophical work was heavily informed by cutting-edge scientific research from across the cognitive sciences: as an example, Silverman (2018) brought together evidence from cognitive neuroscience on predictive processing, work in robotics, and our work on sensorimotor singularities, to form novel philosophical conclusions about the embodied nature of perceptual consciousness. We presented our philosophical work at leading interdisciplinary conferences and guest edited two journal special issues in respected peer-reviewed journals.
WP Maths:
The results of the Maths WP were successfully applied in robotics. We applied our space-learning algorithm to the problem of camera calibration and obtained significant improvement as compared to existing unsupervised calibration algorithms (see Montone et al 2015). We explored the usefulness of our block-modular architectures to the control of a self-driving car. In particular we contributed to solving the problem of knowledge transfer for driving in different environments (e.g. asphalt vs dirt road) (unpublished work).
WP Color:
The development of the sensory singularity approach inspired an ongoing collaboration with a colleague from the Applied Mathematics department of our university. Our work on color perception also stimulated the interest of philosophers (cf. our participation in a seminar on experimental Philosophy at the Institut Jean Nicod, and co-organisation of a workshop with philosophers at Paris Sorbonne). Our article on sensory singularities (Witzel et al. 2015) has been cited not only in behavioural research and philosophy, but also in neuroscience research. The FEEL workshops spurred a cross-disciplinary review article in the Review of Philosophy and Psychology (Witzel, 2018; -- though completed with other funding).
WP Sensory Substitution:
Our novel contingency-mimetic approach to sensory substitution is directly derived from sensorimotor contingency theory on the one hand and from the field of brain machine interfaces (BMI) on the other. Combining concepts in both fields, we derived a novel (non-invasive) approach that applies bio-mimetics to the level of sensory rather than neural contingencies. It is this idea that was the basis for our novel wearable device providing geo-magnetic information via auditory stimulation.
WP Babies:
Thanks to the ongoing FEEL project, our collaboration with roboticists intensified, with participation and/or organisation of psychology/robotics workshops and conferences, and we obtained a FETopen project with three European robotics teams that ran in parallel with the last half of FEEL. Interaction between roboticists and developmental psychologists is a new and promising development, only just beginning, especially in Japan and in Europe, as demonstrated by increasing interest in the ICDL conference as well as increasing cross-disciplinary contributions in traditionally uni-disciplinary infant and robotics conferences.
Over all WPs we published (or have under review) 49 peer reviewed papers, 9 papers in proceedings, 8 book chapters, 3 masters/PhD theses, and made 147 dissemination activities (speeches at conferences, public events or radio/TV shows). We (co-)organised 7 international workshops, created and maintained web and facebook pages, and had numerous meetings with expert colleagues in France and abroad.
WP Philosophy:
We used web-based media (an article in Scholarpedia and Facebook) to disseminate information about our work to a general audience as well as other specialists across disciplines. We edited two journal special issues (see the passage under Inter and Cross Disciplinary Developments). We co-organized with Paris Sorbonne University a local philosophical workshop in 2016, and organized a two-day ASSC 2015 Satellite workshop on the sensorimotor theory with more than 30 international speakers and poster presentations, and a special issue of the Journal of Consciousness studies resulting from it.
WP Math:
We started a collaboration with a robotics team at Softbank Robotics (cf publications with Laflaquière), and another with Berkeley AI Research at UC Berkeley where we explored the application of our architecture to self-driving cars (unpublished). Our block-modular approach to overcome catastrophic forgetting is being continued at Berkeley AI outside of FEEL (Cheung, Terekhov, et al 2019).
WP Color:
Our theoretical and empirical work on the colour-switching dress was well received by a broader non-scientific audience. Our theoretical explanation has been liked many times on facebook (https://www.facebook.com/ercfeel/) and was advertised in the UK colour group Newsletter in 2015. Our empirical work on the dress has been presented in TV broadcasts (French M6, German ARD), in radio interviews (Antenne Bayern), newspapers (Neue Züricher Zeitung), and social media (Buzzfeed). We co-organised with Paris Sorbonne a local workshop on philosophical and psychological perspectives on color categorization. A well-known laboratory specialized in color in Giessen recruited two of our collaborators after they left the project.
WP Sensory Substitution:
The intuitive integration of a stable external reference into spatial perception provided via sensory substitution or enhancement has practical industrial potential for non-invasive compensation of perceptual deficiencies, such as blindness, spatial and vestibular disorders, or aging (Schumann & O’Regan, 2017). Our ERC Proof of Concept grant obtained in 2016 was related to this work, attempting to improve speech recognition in hearing-impaired persons using tactile stimulation. Unfortunately the work failed to reach an industrially exploitable state.
WP Babies:
We co-organised 3 international workshops linking robotics to developmental psychology (Paris Descartes 2013, 2018; ICDL Pontoise, 2016). We started active collaboration with robotics laboratories in Germany and Italy in order to model babies’ learning. This work led in 2016 to obtention of a 4-year FETopen project “GoalRobots”, currently ongoing between 3 European robotics laboratories and our team. After leaving the project, a PhD student from FEEL will be creating a “babylab” growing out of our work. The project is financed by the town of Paris and will link current scientific research to everyday experiences of parents and teachers.
The project achieved its goal of building awareness of the sensorimotor approach, showing new phenomena, demonstrating practical applications and raising new scientific questions. The Philosophy WP made contact with leading alternative approaches and clarified the distinctive features of our approach, fueling integration of the sensorimotor approach into the mainstream. For example, the currently influential “predictive coding” theory of neural functioning implements many of the tenets of the sensorimotor approach. Our mathematical work showed concrete illustrations of the usefulness of the sensorimotor approach in understanding the notion of space, with practical applications in sensor calibration and in deep learning. The Color WP raised new questions concerning the predictions of the sensorimotor approach with respect to color singularities, and attracted attention with its explanation of the #theDress internet phenomenon. In the sensory substitution work, we found that creating functional substitution devices is less trivial than assumed in the literature. These results gave us new insights about the difficulty of learning artificial coding schemes, which in turn led to a new principle (contingency mimetics) and allowed us to create a device that showed the desired perceptual integration of artificial signals. Our baby work provided the desired data about tool use and body knowledge acquisition. In our ‘bracelet’ experiments, our numerous efforts to perfect the measurement technique showed that babies are much less sensitive to contingencies than what one may conclude from classical studies. As hoped, the baby workpackage also forged active contacts with roboticians that will be constructive in future work.
The project has also raised new questions and offered interesting and important paths for further exploration: in philosophy, pinning down the fine distinctions that differentiate the theory from current alternative theories. In mathematics, showing how the theory elucidates our understanding of space and highlighting the importance of group-theoretic and invariant-based approaches to artificial intelligence and perception. In color: defining the factors that determine color singularities and relating them to the existing literature. For sensory substitution the project showed that sensory substitution is much more difficult to obtain than expected from the literature on multisensory integration. The baby work showed that babies learn better to use a tool when they laugh, and are much less sensitive to contingent stimulation than what one may conclude from classical studies.
At the beginning of the project, the state of the art in sensorimotor theory was a theoretical approach: the idea that the perceived nature of sensory qualia is determined by sensorimotor contingencies. There were also a few empirical tests, in particular as regards color, where we had predicted the existence of “singular” colors; as regards sensory substitution, where data existed from the literature suggesting its possibility; and in the classic ‘constructivist’ theories of baby development, but where proposed mechanisms were not well defined. The project then contributed significantly to expanding this state of the art and the purview of the theory: it was reinforced and solidified in relation to existing philosophical approaches; prior results on color singularities were rechecked and linked with existing literature; three sensory substitution/augmentation devices were developed and studied, thereby providing important insights for the future development of this exciting possibility. Finally the project applied the sensorimotor approach to understanding babies’ learning of tool use and body knowledge acquisition, and used the sensorimotor approach to model these in collaboration with roboticists.
We have positioned sensorimotor theory more clearly in the philosophical community, discussing it in relation to leading alternative theories. In the psychological community we have refined pre-existing examples of how the approach can be applied in color perception and in understanding learning of tool use and body knowledge acquisition in babies. For society we have theoretically analysed current limitations in the field of sensory substitution and developed and illustrated novel research topics related to making functional sensory prosthetic devices. Another type of impact of the project is through its activity at conferences and workshops as well as informal national and international meetings, where we created or reinforced cross-talk between numerous researchers, labs and other projects in philosophy, AI, neural nets, robotics, prosthetics, perceptual science, and developmental science.
Please, type in a comma-seperated list of keywords related to the project.
Consciousness, Sensorimotor Theory, Space, Color, Sensory Substitution, Body Knowledge, Infant, Robotics