También disponible en Español


The digital magazine of InfoVis.net

Beyond Autism
by Juan C. Dürsteler [message nº 150]

The application of multi-sensorial and multi-modal interaction techniques in the world of autistic people has the potential to bridge the gap between two usually distant worlds. Visualisation acquires a new dimension this way.
A kid interacting with MEDIATE
courtesy of the "grup d'Experimentació en Comunicació Interactiva" de la Universitat Pompeu Fabra

MEDIATE  is a space addressed to autistic children whose objective is to validate a new type of immersive, intelligent and multi-sensorial environment that reacts to the presence and movements of the user. 

It is, at the same time, a project funded by the European Commission whose team is made up of the “Responsive Environments Centre” of the University of Portsmouth, England, leader of the consortium, the Hogeschool voor de Kunsten Utrecht  in the Netherlands, the Institut Universitari del Audiovisual  of the Universitat Pompeu Fabra in Spain, the Kings College Psychiatry Institute in London, England and the British company Show Connections Limited 

The problem it tries to approach is that of the communication between the autistic children and their environment. Deeply autistic children do not communicate with the rest of us or they do it with great difficulty. They usually do not answer when called by their own name and they have serious limitations in interpreting tone of voice, facial expressions and, in general, picking up the emotions of other people. 

People with autism often react to sensorial stimuli like sound or touch in ways that are unusual for us. In many cases they present a reduced sensitivity to pain while they can be hypersensitive to other types of sensations, which some times is argued as an explanation of why many of them resist being embraced (see for example the page of NINDS )

In the end, it’s extremely difficult for us to penetrate the world of autistic people, probably as difficult as it is for them to interact with our world.

MEDIATE tries to tend a multi-sensorial bridge to the interaction of autistic children and the world surrounding them. It is, at the same time, an important challenge for the research of the human-machine interaction where the interaction interface is simply the bodily expression of the user.

The core objective of the project is that the children and their parents have fun. This, which could be considered a superficial objective, is an important challenge when speaking about deep autism. Secondarily, the objective is that the expressions that the environment frees in the autistic children serves as a way for parents, psychologists, teachers care-givers and researchers to better understand the world these people live in. 

It doesn’t have, therefore, a therapeutic goal, although a better comprehension of this disorder could lead to best therapeutic practices. The ethical aspects of this project have been carried out rigorously and experts on ethics have been consulted regarding these aspects of the project.

espai_panorama2.jpg (48944 bytes)
espai_en.gif (22815 bytes)The space of MEDIATE. It consists of an irregular hexagonal room with two walls occupied by big screens, two walls equipped with vibro-tactile elements and two other inert walls in one of which stands the entrance. 
Source: courtesy of the "grup d'Experimentació en Comunicació Interactiva" de la Universitat Pompeu Fabra.
Click on the image to enlarge it.

The immersive environment of MEDIATE is an irregular hexagonal space (see the figure in the graphical version) that can be accessed by a covered ramp that provides a gradual entrance to the place. Two of the walls are made up of big screens (of 3 x 2.35 meters) where interactive visual patterns are projected. The user interacts with them merely by moving or approaching them. Other two walls have vibro-tactile elements that react to the pressure when touching or leaning on them. In one of them the elements are covered by surfaces of different textures like leather, cork or wood.

Technically the system is fairly complex. It has nine video cameras that take the movements of the kids and informs the computer system of his or her location and movements. Five microphones gather the sounds the user emits and certain areas are sensitive to tactile pressure. All these devices continuously inform the central system about the activity of the user, which then integrates all the information and reacts properly in a visual, sonic or tactile way. 

This "cleverness" of the system has been adapted to the needs of the autistic children. In the beginning the system reacts with low level stimuli that increase in variety and intensity according to the user’s behaviour. For example, when the system detects that the user is falling into a repetitive movement pattern (usual in autism) it reduces the intensity of the stimuli to favour the appearance of other more original behaviours, something difficult to achieve for these people.

Visual interaction appears to me to be of special interest. I had the opportunity to see and to interact with some of the patterns tried in the installation at the premises of the grupo grup d'Experimentacio en Comunicacio Interactiva (group for the experimentation on Interactive Communication) at the Universitat Pompeu Fabra 

control_tecnic_1[1].jpg (67376 bytes)
MEDIATE: Particle based patterns. When the user approaches the particles increase in size. The screen shows waves of colour as a function of the bodily movement. 
courtesy of the "grup d'Experimentació en Comunicació Interactiva" de la Universitat Pompeu Fabra.
Click on the image to enlarge it.
MEDIATE: Part of the system's control centre. The integration system of all the information that reaches the control centre is highly sophisticated. Here resides the "intelligence" of MEDIATE. 
courtesy of the "grup d'Experimentació en Comunicació Interactiva" de la Universitat Pompeu Fabra.
Click on the image to enlarge it.

The patterns shown during interaction are abstract ones and based mainly on particles. This is due to the fact that autistic people show great interest in elements and details that make up a whole, instead of the whole itself that appears to go unnoticed for them. Different patterns have been tried that have been accepted or rejected as a result of the satisfaction or not in the interaction with users.

One of them is a set of squared particles of different colours that increase in size when the user approaches them. By moving the limbs or touching the surface of the screen you generate “waves” of colour that can be governed univocally with the body.

I had the opportunity to interact with one of the patterns named “fullaraca” (fallen leaves). In it many squares of slightly different sizes and tones of green lie near the ground resembling leaves fallen to the ground. Moving your arms you can “gather” them or you can create virtual flows of air that elevate the “leaves” that fall down under a virtual gravity. 

fullaraca2.gif (90572 bytes) fullaraca.gif (120003 bytes)
Fullaraca (fallen leaves) one of the preliminary models of interaction. The user interacts with his/her whole body with the "leaves" that can be gathered in the arms or you can create "air flow" that take them from the ground and spread them on the "air". To the left a user(Narcís Pares, main researcher of the UPF team)  interacting. To the right the computer model of a similar interaction. 
Source: courtesy of the "grup d'Experimentació en Comunicació Interactiva" de la Universitat Pompeu Fabra.
Click on the image to enlarge it.

I was struck by the sensation of freedom in the interaction and of being able to use movements and gestures that are completely natural and usual in our daily life (unlike mouse and keyboard). “Fullaraca” was abandoned for autistic children because in it the particles show a certain random behaviour (as leaves do) which resulted inappropriate for autistic people that require predictability and have serious difficulties with imaginative behaviour

MEDIATE has ended up as a very interesting experience in the field of autism, but also in the experimentation of multi-sensorial and multi-modal interaction.

Although we are still a long way from interacting with every day devices like the computer or the DVD player with our simple presence or with our natural gestures, the ease of use and freedom of interaction that these new devices have has made a big impression on me. There’s still a lot to see... 

Links of this issue:

http://web.port.ac.uk/mediate/mediate.htm   MEDIATE Official website
http://www.ninds.nih.gov/health_and_medical/pubs/autism.htm   NINDS Web site about autism
http://www.iua.upf.es/eic/mediate/index.htm   grup d'Experimentació en Comunicació Interactiva. Website about MEDIATE
© Copyright InfoVis.net 2000-2014