Virtual Shoe box Task Software Report

by T. David Burns

This document contains a description of the Virtual Shoe box Task environment. I first describe the user populations and the need for the software, then I describe pre-existing work both in software and otherwise. Then I briefly describe an example application of the software in a simple game. After that, I describe the current design of the software and the enhancements I plan to make.

Children with Autism

Around one in five hundred children has Autism or a related development disorder. Children with autism have deficits in language, social ability, and play. They tend to prefer isolation, engage in repetitive activity, and may be hypersensitive to certain stimuli. The cause of Autism is unknown but currently thought to be genetic.

The Diagnostic and Statistical Manual of Mental Disorders of the American Psychiatric Association, Fourth Edition (usually referred to as the DSM-IV) lists twelve symptoms associated with Autism, grouped in three categories - social impairments, communication impairments, and stereotyped or restricted interests and behavior.  To receive a diagnosis of Autism, an individual must exhibit at least six of these symptoms and at least one from each category. Individuals that exhibit less than six symptoms or less than one symptom from one or more categories will usually be classified as pervasive development disorder, not otherwise specified (PDD-NOS).

The specific social symptoms include inappropriate use of non-verbal behaviors in social interactions, failure to engage in developmentally appropriate peer relationships, and inability to share attention with other persons. While the widely held image of the person with Autism being utterly aloof and unable to relate to others at all is not really accurate, social interaction is difficult and restricted for them. Group activities are especially difficult.

Communicative impairments mean that children with Autism may either lack spoken language entirely or if speech is present, they  may have difficulty processing language and instead engage in repetitive or idiosyncratic uses of language.  An extreme example of this is echolalia, which means that a person merely repeats back what was said to them instead of responding in some appropriate way. Ordinary children often go through a brief stage of echolalic speech early in their development, but are very unlikely to take it as far as the child with Autism, who may repeat long sentences verbatim.

The restricted interests aspect of the disorder means that the child with Autism may need to engage in rigid routines and rituals, will often not take much interest in (or be incapable of) ordinary play or group activities, and may become fascinated by tiny details of an object while entirely missing the significance of the whole. An example of this is provided by the school child with Autism that could no longer recognize his teacher if she was not wearing earrings.

For a person with Autism, work is play and play is work-playing a game of catch or having a simple conversation may seem confusing or stressful to a child with autism, but they may become fixated on some subject and be willing to spend hours looking at books concerning that subject. When young autistic children learn to read, they tend to prefer encyclopedias to fairy tales, and may read the publisher's  data page with as much interest as they would a story. They also may be hypersensitive to certain stimuli and may be distracted by small details that seems irrelevant to their teachers or peers.

These kids provide a special opportunity because 1) there are various training procedures associated with their therapy that are formal enough to survive being automated, and 2) while these kids have trouble relating to human beings, they often love fiddling with computers. While there are limits on what aspects of therapy can be supported by software (e.g. receptive language can be exercised, but for a computer to train expressive language would be a very serious undertaking requiring heavy AI), there is a rather severe shortage of trained personnel who can administer these training procedures, which makes the need for some software support urgent.

Learning Style and implications for learning

The various deficits of autism leave room for some strengths. With the bad news comes some good. Particularly, persons with autism tend to be highly visual and to respond well to routine. Their restricted interests also mean that they can become highly motivated if a learning task is cleverly combined with some aspect of that interest. Children with Autism tend to be highly visual learners. Although their ability to hear usually is not impaired, auditory information may be difficult for them to process. Often there will be a great disparity between measures of their verbal and nonverbal intelligence.

These characteristics of students with Autism provide a big challenge to educators and parents. While this is bad news for verbal tasks, it also means that there may be a way to get through to them if unconventional visual approaches are tried. Division TEACCH  at the University of North Carolina, Chapel Hill takes advantage of these strengths in their structured teaching approach. One of their most successful innovations is the shoe box task.

Shoe box Tasks

The basic idea of a shoe box task is fairly general. A game is constructed from simple materials in a way that makes the player exercise some specific skill. The shoe box task is often made from a shoe box and consists of some simple and visually striking manipulable components that help to teach following instructions, sorting, matching, etc. The strategy of the shoe box task is to capitalize on the strengths and evade the weaknesses of the visual learner. Objectives of a shoe box task include: learning matching, sorting, identification, fine motor skills, and receptive language. More broadly, the child learns to work independently, following a work system (what work, how much work, when finished, and what next?).

The achieve these goals, we must simplify and organize the presentation of the task to reduce distraction and to help the student identify and focus on the relevant portions of the task at hand. In the classroom, this is accomplished by  using containers, arrangements that highlight relevant details, and by providing visual cues and examples that in effect comprise "visual instructions". This can be achieved by use of jigs, examples, and written instructions. Colors, shapes, and pictures can be used to attract attention to relevant details. By limiting the number and capabilities of the visual elements, visual clarity is achieved. Elements should be laid out separately (not bunched together).

Tasks should have a clear beginning and end, with a minimum of irrelevant opportunities. They should exhibit visual clarity, by emphasizing the relevant elements and eliminating or de-emphasizing the irrelevant elements of the task. They should be arranged to provide visual instructions, so that by looking at the pieces or by fiddling, the goal of the task becomes obvious. The opposite side of this coin is that there should not be any obvious alternative uses for the materials. Whenever possible, they should make use of the student's individual interests, in order to increase motivation. This can be achieved by including appropriate pictures or sounds in the task. For instance, if the basic task is sorting, and the child is interested in Barney the dinosaur, the task could be configured so that the student must sort pictures of Barney and his gang.

A very useful concept used to strive toward the goal of visual instructions is the jig. A jig is a material with a hole cut in it that is the proper shape for another piece to fit in or through it. The idea of a jig comes from carpentry, where jigs are used to guide tools when the carpenter must make repetitive shapes. Here it is used as a visual clue to where an object belongs at a particular stage in the task.

figure 1. a Jig

Materials used in shoe box tasks include shoe boxes, pictures, balls, blocks, beads, words written on cards, bags, baskets, boxes, and envelopes.

advantages of a software approach

The need to emphasize the visual mode, to restrict activities to prevent perseveration and engage attention, and the need for individualized approaches make a software solution seem very important and powerful. That is, children with Autism enjoy doing computer tasks, and the computer is well suited to implementing some of the constraints and visual orientation that is called for in the most popular approaches to educating children with Autism. The fact that they tend to be more comfortable with objects than with people means that using computers for learning can be a big plus for them, at least from a motivational viewpoint.

The  learning objectives for the parent/therapist group are limited to learning how to construct  virtual shoe box tasks. Once they are comfortable making these tasks, this activity will help them to learn about the student's needs, but that is not directly supported by the shoe box task software. A software approach promises to make this process less expensive in terms of investment and effort. Another big advantage of the software approach versus the ordinary shoe box task is that virtual shoe box tasks can be built once and then cheaply reproduced, distributed, and shared throughout the community. Instead of beginning at square one, a parent, therapist or teacher may have an entire library of tasks to choose from.  Virtual shoe box tasks can be reproduced and shared among as many persons in the community as want them, at essentially no cost.  Then all they need to do is choose the right skills to exercise and add individualized elements. Virtual shoe box tasks take up almost no space, while real shoe box tasks are rather bulky. Also, some reinforcement mechanisms and animations to spark interest can be incorporated in the virtual shoe box task in a way that could not be duplicated with a physical shoe box.

dangers of a software approach

If some care is not taken in the design and use of computer programs for instructing children with Autism, at least two dangers arise. First, such children are very capable of achieving rote memorization without understanding or an ability to generalize what has been learned or apply it. And solitary use of a computer may reinforce the child's social isolation. Steps can be taken in the software design to try to avert the first danger, but only the parent or teacher supervising the learning child can prevent the second. No software product can completely replace a human teacher. There is also a danger that the primary audience for the shoe box task will be too young or otherwise unable to use a mouse, etc.


The software should be usable in the home or classroom on a simple pc setup with a sound card.

For the virtual shoe box tasks, the design of the activity should be well adapted to the characteristics of persons with autism. The software should provide good affordances for relevant activity, with very little (ideally no) stimulation available from "wrong" or irrelevant activity. Allowing typical kids to make up their own games can provide significant learning opportunities, but a child with autism would likely take such an open-ended opportunity as a chance to perseverate (repeat some simple activity endlessly). The software should emphasize the visual modality, and should be so simple that no instructions or adult intervention are necessary. The game should not require talking or listening to speech (aspects of speech can be the object of a lesson, but not an essential part of playing the game), and the users should learn by doing. The game should provide multi-sensory stimuli, but allow some modification (turn down the sound, change colors, etc.?) in case the child is hypersensitive. The game should be an interesting activity with no "losing", and ideally should adaptable to player skills. Wrong answers should either be impossible or ignored. Hints or prompts should be given when necessary. Once the software is active, play should be simple enough that no adult intervention is needed.

The software used to build virtual shoe box tasks should be simple and non threatening for unsophisticated users. It should be easily installed, i.e. one clickable icon, not a bunch of files or folders. It should have a small footprint, fitting on a single floppy. Virtual shoe box tasks should have everything that a real shoe box task has, plus variable constraints, animated rewards or demonstrations.

existing software

Children with Autism are currently served by both general market products and specialized shrink-wrapped software offerings.
The general market can be represented by the Disney edutainment CD-ROMs such as Ready for Math with Pooh. These provide a high degree of interest by including familiar animated characters. These have something to offer, but do not exercise the specific strengths of the children with Autism, and tend to have "pot-holes" - elements of the games that require skills that children with Autism tend to lack.  For instance, Ready for Math with Pooh gives copious game instructions verbally. Employment of the auditory mode rather than the visual mode is a mistake with this audience. Further, there are lots of distractions and details that may attract their attention inappropriately. There are many nonfunctional ways to fiddle with the software if one wishes to ignore the rules and objective of the game as designed and instead use it for repetitive doodling. Indeed, some of the responses the game provides for wrong answers are more interesting than the reinforcement for correct responses. Since this product was designed for ordinary children (or at least, it was designed in a way that ordinary children can overcome its weaknesses enough to get some enjoyment from it) it is not surprising that no steps were taken to fill in the pitfalls that an Autistic child faces. In spite of this, children with Autism usually love ordinary computer games, often showing more interest in them than in ordinary games or toys.

The strengths of the firms offering more specialized products (special ed., language disabilities, and Autism) usually rest on the fact that the products have some connection with a body of research specific to their target audience. Such products include Labeling_Tutor, FastForWord, Earobics, etc. (More detailed information can be found at These are aimed at slightly different applications than the software described in this report.

The most similar software product is Labeling_Tutor, which automates some aspects of discrete trial training. Discrete trial training is a variety of the applied behavior analysis developed at UCLA by Ivar Lovaas. Discrete trial training is supposed to be an interactive therapy administered by a therapist or an aid. While some of the DTT exercises can be done as independent activities, this represents a change from the objectives and techniques described in the scholarly literature about DTT. DTT emphasizes interaction and reinforcement of behavior. Both action and inaction should have immediate consequences. Hence Labeling_Tutor does not follow the TEACCH approach, which emphasizes acquisition of independent work skills.
Some other loosely related products exist for teaching aspects of language for a standpoint of ordinary speech therapy. Children with Autism may gain some benefit from using that software, but a different benefit from that envisioned for the Virtual Shoe box. None of these products capitalizes on the strengths of the child with Autism while avoiding the pitfalls of perseveration, audible mode, or overstimulation.

Virtual Shoe box Task Software

My project is to provide a software component that implements a "virtual shoe box task," a multimedia simulation of the shoe box task. This software will be used by children with autism or related developmental disorders under the minimal supervision of a parent, teacher or therapist. This is not a sophisticated group of users.

The basic approach of this project was to provide software that lets educators make virtual shoe box tasks. Instead of trying to produce a complete or even adequate set of tasks myself, I sought to build an environment that supports shoe box task  activities, so that parents, teachers, and therapists can create their own virtual shoe box tasks. They should be able to use this software to make little games that involve the player moving objects around until the objects are in a final position, in an environment that is structured to minimize the possibility of failure. The game designers should be able to define what objects are present, to what or where they can "get stuck", and what happens when they do. The players of the game can then move the objects around until they stick in the correct position and respond with some sort of feedback or reinforcement.

example game

Figure 2, a simple demonstration game.

The demo game resides at Seven colored rectangles appear on the screen, two pink, two red, two blue, and one black. Of the pink, red and blue pairs, one of each pair is mobile, one of each pair is immobile. The black rectangle is mobile. If you move the black rectangle to the left side of the mobile blue image, it becomes stuck and whenever you move one of them, the other will also move. For the remaining movable rectangles, if you put them on top of the immobile rectangle of the same color, they will stick. This demonstrates the basic required capabilities - elements can be moved, the can stick to each other at arbirary distances, and when things get stuck, they invoke threads to provide a response.

Design Structure

The system was designed with three application specific classes (VSTApplet, Element, and ElementReader) and one reusable component (Appletcation), along with some other insignificant helper classes which will not be discussed. These four classes provide the basic game capabilities of reading image descriptions from a file, displaying the images, allowing the user to move the images with the mouse, and allowing the game designer to specify where and to what image may get stuck and what happens when they do.

VSTApplet is the smallest of the four. It provides both an applet and application interfaces, so the game can be played on a web page or as a stand-alone program. It contains three methods. The init method invokes ElementReader to read a group of images. The paint method displays the image elements on the screen. The main method provides a command line
application interface, so that the program can be run as a standalone application. VSTApplet extends java.applet.Applet, so it inherits the capability to be embedded in a web page as an applet.

Appletcation is the reusable component. It is designed to allow applets to be used as stand-alone applications outside the context of a browser and without embedding the applet in a web page. It provides a simple pseudo-browser, displaying an applet in a window along with a status display and providing all the context necessary to transplant applets from the browser context to an application context. Appletcation enables VSTApplet to act as an application with a main method that is only one line long. It is reusable because it could be used to invoke any applet.

ElementReader is a class that contains knowledge about the format of the file used to define the initial conditions of the game. VSTApplet invokes one of its methods to read a file and initialize the group of image elements to be displayed at the beginning of the game.

Element is the class that stands at the center of the virtual shoe box application. It contains the knowledge of how to display an image, how to allow the user to move the image with the mouse, how to make the image stick in desired places and respond appropriately when it sticks.

Future Enhancements

The initial design has been a leap of introspection. There is a strong need for user feedback to polish the design. In the case of children with Autism, this poses some special challenges, both because they are particularly vulnerable to user interface problems, and because they have difficulty articulating their difficulties to another. This argues for the help of Autism specialists and the teachers of the children in interpreting results.

Authoring Environment

The next step toward making the virtual shoe box tasks practical is to make an authoring environment to support creation of new tasks. I visualize this as a display similar to the existing VST setup, but with all the underlying structure made visible and manipulable. So in addition to the visible image elements, icons representing the invisible elements will also be present. When two elements are stuck together, this relationship is represented as a black line connecting the two elements with a label (see figure). Similarly, two elements that can be stuck together but that are not yet stuck will have a red labeled line connecting them. Such connections can be established by dragging the red or black square icon from one image to another. They can be removed by clicking on them.

Figure 3. Visualization of authoring tool.

In the authoring display, all elements are movable by clicking and dragging. The mobility of an element in the play environment is represented by a colored border around the element. Mobile elements will have a red border around them, immobile elements will have a black border. Clicking on the border toggles the element's mobility.

Double clicking on an element brings up a dialog that allows the editor to change the urls that point to the image and thread class associated with an element.

A menu item will add new invisible elements to the display, the delete key will delete elements.

Community Enhancing Web Page

A web page would provide an important future enhancement to the usability of this software. In addition to allowing users to download software, it could also provide reviews of software, technical support, a library of tasks, and a discussion group for help or mentoring among parents, teachers, and therapists interested in using the environment.

Other miscellaneous enhancements include some automated record keeping or data analysis, allowing authors to supply hints or prompts, allowing authors to place elements randomly, and more direct support for sequencing using some sort of state machine.


Once the authoring tool is complete a round of game design and user feedback can at last begin. Then this environment should provide a useful structured tool to the parents, educators, and therapists of children with Autism.


Hodgdon, L. A. (1995). Visual Strategies for Improving Communication. Troy, QuirkRoberts Publishing.

Jordan, R. and S. Powell (1995). Understanding and Teaching Children with Autism. New York, John Wiley & Sons.

Maurice, C., G. Green, et al., Eds. (1996). Behavioral Intervention for Young Children with Autism. Austin, pro-ed.

Quill, K. A., Ed. (1995). Teaching Children with Autism. New York, Delmar Publishers, Inc.

Siegel, B. (1996). The World of the Autistic Child. New York, Oxford University Press.

"Visually Structured Tasks: Independent Activities for Students with Autism and Other Visual Learners", Division TEACHH

More information on Division TEACHH is available at