También disponible en Español

Inf@Vis!

The digital magazine of InfoVis.net

Sketch Understanding
by Juan C. Dürsteler [message nº 61]

Sketch Understanding is a branch of Artificial Intelligence that tries to automatically "understand" the sketches we use to communicate. It has great potential for knowledge management. 

Sketches, drawings and graphical schemes are visual tools that we use on a daily basis to transmit ideas and concepts. Most of the time these are incipient concepts, designs in their early stage of development or ways to communicate ideas or tentative solutions to existing questions or problems

Other times the sketches help us to communicate important concepts in easy to understand ways such as, for example, what our marketing strategy will be or what organisation scheme we propose.

In more specialised situations sketches are the basis of the discussions about new products or designs in the engineering or industrial design departments. Even in the most domestic domain sketches play their role (who hasn't ever sketched a map of the way to their home to help a friend get there?)

Sketch Understanding is a branch of Artificial Intelligence whose main goal is to extract semantic knowledge from the automatic (possibly with the help of the user) recognition of sketches, simple drawings and graphical schemes.

The applications of Sketch Understanding are beginning to blossom but they promise to positively influence the knowledge management of organisations, providing the possibility to create semantic information in a more or less automatic way from simple hand made sketches. Let's see some examples.

Hod Lipson, from the Technion Institute of Israel has made his PhD thesis on a system that is able to understand sketches of 3d objects and convert them into their CAD counterpart in order to bridge the gap between conceptual design and CAD.

The Artificial Intelligence lab of the Massachusets Institute of Technology (MIT) has several projects devoted to converting hand made sketches into the concepts they refer to. One of them, ASSIST, is able to read a mechanical engineering sketch, interpret it and allow the designer to interact with a simulation of its behaviour, for example a car going down a tilted plane. 

Another of the projects can read a sketch of an UML (Unified Modelling Language) diagram of a piece of software and create the diagram to include it in the documentation or to use it in the building of the software.

On the other hand one of the most interesting projects is sponsored by DARPA and intends to use the sketch understanding to improve its Command Post of the Future or CPoF. With Kenneth Forbus and Lynn Kendall of the NorthWestern University as main researchers this project tries to revolutionise the way the commanders and staff of a command post work by using multimodal interfaces allowing them to communicate with information systems and with each other. This includes the interpretation of the sketches that are frequently used to delimit the course of action (COAs), the use of graphical visualisation systems and speech synthesis. (See also the summary of the project)

The most interesting aspect of the last example is the analogy with the strategic management of the knowledge of organisations. One of the fundamental problems of knowledge management is the implicit knowledge that resides in the brains of the experts, that typically remains undocumented. 

Nevertheless many key ideas are transmitted via sketches, that properly converted to semantic information can be expressed as explicit knowledge.

While we wait for the technology to arrive maybe it's a good idea to exercise the creation of excellent sketches and schemes allowing us to communicate our ideas in an proper way.


NOTE: The first symposium on Sketch Understanding will take place in Stanford, USA, next March, 25-27, 2002.

Links of this issue:

http://www.mae.cornell.edu/lipson/thesis/overview.html  
http://www.ai.mit.edu/projects/rationale/projects.shtml  
http://www.darpa.mil/ito/psum2001/K316-0.html  
http://www.cs.nwu.edu/~bmd/cpofsi/summary.html  
http://www.me.cmu.edu/faculty1/stahovich/sketchsymposium.htm  
© Copyright InfoVis.net 2000-2014