What visual information is used during a simple task of picking up and putting down an object?
Virtual Reality Lab
Mary Hayhoe Principal Investigator
Jason Droll Graduate Student
Jochen Triesch Collaborator
Brian Sullivan Research Assistant
Keith Parkins Programmer
Goals
Most experiments in visual memory
and attention address the maximum capacity of storage or the limits of
human performance. Our interactive sorting task within virtual reality
is designed to mimic more natural behavior. We aim to reveal the
typical usage of memory, attention and motor movement during ordinary
behavior. How do the goals of a behavioral task interact with
mechanisms of attention?(The present paradigm builds upon other experiments described in Triesch, Ballard, Hayhoe & Sullivan, 2003)
Equipment
Subjects wear a Virtual Research V8 head
mounted display, displaying a stereo scene on two LCD screens with 640
X 480 resolution. Subjects' thumb and forefinger are placed in
thimbles, each attached to a Phantom-3
device from Sensable Technologies. The Phantom provides haptic
feedback, and the position of the subject's thumb and forefinger are
represented by small red spheres. The position of the left eye is
monitored with an ASL 501 tracker, and the velocity of the right eye is
monitored with an ASL 210 tracker (to allow sacccadic updating).
For more on saccadic updating in virtual reality, see Triesch, Sullivan,
Hayhoe &
Ballard, 2002.Task Paradigm
Subjects perform a brick-sorting task in which
one feature is used for pick-up selection, and either the same, or a
different, feature is used for the put-down decision. The pick-up
and put-down cues indicate which feature dimension is relevant (e.g.
color, width, height or texture) and what feature value to use on a
particular trial (e.g. red).
On a small fraction of trials (~10%), we make a change to one of the features of the brick being carried. We warn subjects before the experiment to expect these changes, and to place them in the "trash can," the black hole in the center of the table, if they detect the change.
General Results
Eye movements during the brick-sorting task
suggest that subjects are acquiring information on a "need-to-know"
basis. The order in which areas in the scene are fixated is
tightly linked to the immediate demands of the task.Rates of change detection suggest that subjects retain only the brick features that are expected to be relevant for the put-down decision. Despite being told to expect changes, subjects miss the majority of feature changes. However, subjects are more sensitive to changes in features relevant to the sorting task than features that are irrelevant. This suggests that the information acquired during a task can be as sparse as an indivudual object feature.
Subjects' sorting decision following a missed change suggests that another reason for why these changes are missed depends on the expected relevance of the changed feature. When subjects miss a change to a feature that they expect to use for the put-down decision, subjects appear to miss the change due to the failure to update the current brick feature (evidenced by the preference to sort changed bricks by the old, pre-change, feature). However, when subjects are instructed to use a feature they couldn't anticipate, subjects appear to miss the change due to a failure to retain the relevant feature before the change (evidenced by the preference to sort changed bricks by the new, post-change, feature).
Thus, selection and storage of visual information in naturalistic tasks depends on both task relevance and its predictability. Subjects dynamically trade off memory use and eye movements depending on their expectations of what information is needed for the task.
Example Trials
Predictable Put-Down Relevance (Droll et al, (accepted pending revision) JEP:HPP)
Normal Trials with No Feature Changes (32.0MB, 12Hz)
Several trials in the Two-Feature condition where height is relevant for pick-up and width is relevant for put-down. Fixations to the put-down cue are generally followed with a saccade to the belt, suggesting that the subject made the sorting decision based on information in working memory of the relevant feature (as opposed to re-fixating the brick before sorting). This behavior is most common when subjects can predict which feature is relevant for put-down.
Missing Color Change (18.1MB, 12Hz)
A subject is picking up bricks by their texture and sorting them also by texture (One-Feature condition). In the third trial, the blue brick changes to red but the subject fails to report the change (by failing to sort it in the "trash can").
Missing Color Change (11.8MB, 12Hz)
Similar to the above example, a subject is picking up bricks by their texture and sorting them also by texture (One-Feature condition). In the third trial, the red brick changes to blue but the subject fails to report the change (by failing to sort it in the "trash can").
Noticing Texture Change (13.3MB, 12Hz)
Again, a subject is picking up bricks by their texture and sorting them also by texture (One-Feature condition). However, the subject notices the relevant texture change in the second trial.
Unpredictable Put-Down Relevance (Work in progress...)
Normal Trials with No Feature Changes (16MB, 12Hz)
The subject is picking up bricks by their width and sorting them on the belts by any of the four features. The frequent fixations back to the brick after having fixated the sorting cue suggest that subjects are acquiring the relevant brick feature "just-in-time" for the put-down decision. This is a common phenomenon when the put-down relevant feature is not predictable.
Missing Color Change (15.7MB, 12Hz)
The subject is picking up bricks by their texture and sorting them on the belts by any of the four features. In the second trial, the red brick changes to blue but the subject fails to report the change (by failing to sort it in the "trash can").
Violating Scene Context (Work in progress...)
Noticing New Color Change (8.0MB, 12Hz)
Color changes are almost always noticed when the change introduces a novel color to the scene. This subject noticed the red to blue color change when blue had not been used earlier in the experiment. These same red to blue changes were usually missed when red and blue bricks were used throughout the task (see examples above).
Publications
Triesch, J., Sullivan, B., Hayhoe, M. & Ballard, D. (2002). Saccade contingent scene changes in unconstrained virtual reality. Proceedings, Eye Tracking Research & Application, 95-102.Triesch, J., Ballard, D., Hayhoe, M., & Sullivan, B. (2003). What you see is what you need. Journal of Vision, 3, 86-94.
Droll, J., Hayhoe, M., Triesch, J., & Sullivan, B. (2005) Task demands control acquisition and maintenance of visual information. Journal of Experimental Psychology: Human Perception and Performance. 31(6):1416-1438