Interaction for visually impaired and blind people

Methods and tools have been developed to facilitate computer usage by blind people, to ease their access to visual documents, and to aid mobility. Auditory/haptic interaction was used to present drawings and images, by means of a force-feedback mouse allowing a user to “follow” contour lines, and by representing visual elements with sounds associated to particular iconic items. Real scenes, after simplification, could be expressed as sound sources distributed in a virtual three-dimensional sound space. A Web sonification tool for visually impaired and blind users has also been developed (Websound), including text-to-speech and audio representation of visual data. Educational and recreational software packages have been designed, such as sonic games.

The project “See ColOr - Seeing Colors with an Orchestra” (Swiss Hasler foundation, started Fall 2005, headed by Dr. G. Bologna and Prof. M. Vinckenbosch, University of Applied Studies, Geneva (HES-SO)), aims at achieving a non-invasive mobility aid for blind users. See ColOr uses the auditory, haptic and possibly, the tactile pathway to represent in real-time frontal image scenes. Ideally, our targeted system will allow visually impaired or blind subjects having already seen to build coherent mental images of their environment. A stereo head-mounted camera allows extraction of color and 3D location information of particular scene elements. The use of an eye gaze tracker to provide supplementary directional information is being investigated. Colored scene regions are represented by sound sources in a three-dimensional sound space; to each color corresponds a particular music instrument. Since in a given scene region several colors are usually present, the resulting sound is given by a mixture of several instrument sounds. Targeted applications are the search for objects that are of particular use for blind users (such as signposts, mailboxes, sky, trees, etc.), the manipulation of objects and the navigation in an unknown environment.

WebSound is a Microsoft Internet Explorer plug-in that allows sonification of web pages. It is part of the AB-Web project that has been closed in 2002.


Example of SeeColor prototype

Video of the SeeColor prototype, showing two experiments: matching coloured socks, and following a serpentine in an outdoor environment. ( click here to download the video)

Other videos are available on YouTube.