Using New Information
Technologies in the Creation of Sustainable Afterschool Literacy Activities:
Evaluation of Cognitive Outcomes
Richard E.
Mayer, University of California, Santa Barbara (Principal Investigator)
Bill Blanton, Appalachian State University (Co-Principal Investigator)
Richard Duran, University of California, Santa Barbara (Co-Principal
Invistigator) Miriam Schustack, California State University, San
Marcos (Co-Principal Investigator)
June 30, 1999
Executive
Summary
The purpose of this project was to conduct an independent evaluation
of the cognitive consequences of children's participation in the
Fifth Dimension program, an informal afterschool computer club that
emphasizes literacy activities. In particular, the purpose of this
cognitive outcomes evaluation project was to assess changes in children's
literacy that can be attributed to their participation in the Fifth
Dimension program. In order to accomplish this goal, we developed
collaborations for evaluation at three active Fifth Dimension sites--Appalachian
State University (ASU), California State University at San Marcos
(CSUSM), and University of California at Santa Barbara (UCSB)--and
developed a variety of research methods and outcome measures. Our
primary research method was to assess relevant cognitive skills
of students before they began the Fifth Dimension program and after
they had extensive experience in the program, and to compare their
pretest-to-posttest changes with those of similar students who did
not participate. The outcome measures tapped changes in student
literacy, broadly defined, including changes in computer literacy,
language comprehension, problem-solving strategies, and academic
achievement.
The results of several years of testing at three different sites
yielded a collection of positive cognitive outcomes attributable
to children's participation in the Fifth Dimension. These cognitive
consequences of Fifth Dimension participation include increases
in: (a) computer literacy, as measured by a tests of performance
of computer operation, tests of factual knowledge about computer
operation, and tests of memory for computer terms; (b) comprehension
skills, as measured by tests of mathematics word problem comprehension,
tests of game instruction comprehension using a cloze method, and
tests involving the ability to follow a printed procedure; (c) problem-solving
skills, as measured by the speed and strategic efficiency of learning
a new computer-based math game or grammar game; (d) academic skills,
as measured by standardized tests of reading and mathematics. Some
results were not positive, however, including measures of writing
and verbal recall. In general, students improved in literacy skills
that were closely related to Fifth Dimension activities, namely,
learning to use educational computer games through reading instructions
and through oral communication with peers and mentors.
Overall, the cognitive evaluation provided converging evidence across
three different sites and using a collection of cognitive outcome
measures that participation in the Fifth Dimension resulted in improvements
in children's literacy. The evaluation instruments and methodologies
will be disseminated to the growing network of Fifth Dimension sites.
Products
With the encouragement of the Mellon Foundation, the cognitive evaluation
team produced the following published reports during the course
of the project. A copy of each is included with this report.
Mayer, R.
E. (1997). Out-of-school learning: The case of an after-school
computer club. Journal of Educational Computing Research, 16,
333-336.
Schustack,
M. W., Strauss, R. & Worden, P. E. (1997). Learning about technology
in a non-instructional environment. Journal of Educational Computing
Research, 16, 337-352.
Mayer, R.
E., Quilici, J., Moreno, R., Duran, R., Woodbridge, S., Simon,
R., Sanchez, D. & Lavezzo, A. (1997). Cognitive consequences of
participation in a Fifth Dimension after-school computer club.
Journal of Educational Computing Research, 16, 353-369.
Blanton,
W. E., Moorman, G. B., Hayes, B. A., & Warner, M. L. (1997). Effects
of participation in the Fifth Dimension on far transfer. Journal
of Educational Computing Research, 16, 371-396.
Mayer, R.
E., Schustack, M. & Blanton, W. (1999, March-April). What do children
learn from using computers in an informal collaborative setting?
Educational Technology, 39(2), 27-31.
Mayer, R.
E., Quilici, J. H., & Moreno, R. (in press). What is learned in
an after-school computer club? Journal of Educational Computing
Research, 19.
Consistent
with the charge to disseminate research results, members of the
cognitive evaluation team delivered the following convention papers
during the course of the project.
Blanton,
W. E., Moorman, G. B., Hayes, B., & Warner, M. (1996, April).
Effects of participation in the Fifth Dimension on far transfer
measures of reading and mathematics. Paper presented at the annual
meeting of the American Educational Research Association, New
York, NY.
Mayer, R.
E., Duran, R., Quilici, J. H., Moreno, R., Woodbridge, S., Simon,
R., Sanchez, D. & Lavezzo, A. (1996, April). Cognitive consequences
of participation in the Fifth Dimension after-school program.
Paper presented at the annual meeting of the American Educational
Research Association, New York, NY.
Schustack,
M. W., Strauss, R., & Worden, P. (1996, April). Learning to use
technology without formal instruction. Paper presented at the
annual meeting of the American Educational Research Association,
New York, NY.
Schustack,
M. W. (1997, March). Evaluating a voluntary participation program:
Who's the control group? Paper presented at the annual meeting
of the American Educational Research Association, Chicago, IL.
Mayer, R.
E., Quilici, J. L., Moreno, R., Simon, R., Duran, R. & Woodbridge,
S. (1997, March). The search for experimental control in evaluating
the cognitive consequences of participation in the Fifth Dimension
after-school program. Paper presented at the annual meeting of
the American Educational Research Association, Chicago, IL.
Blanton,
W. E. (1997, March). Product and process research and evaluation
on informal learning environments: Theoretical and methodological
issues. Paper presented at the annual meeting of the American
Educational Research Association, Chicago, IL.
Mayer, R.
E., Quilici, J. L., & Moreno, R. (1999, April). What is learned
in an after-school computer club? Paper presented at the annual
meeting of the American Educational Research Association, Montreal,
Canada.
The primary
goal of this project was to determine whether participating in the
Fifth Dimension improved children's literacy. The Fifth Dimension
participants were elementary school children who were given the
opportunity to use many different kinds of educational software
on a regular basis over the course of a school year. In the Fifth
Dimension program children worked with peers as well as adult volunteers
to learn to use a series of educational programs that have been
selected for appropriateness, appeal, and educational value. When
a child mastered one educational program, he or she moved on to
learn how to use a new one. The learning took place after school
and out-of-school in a computer club where the child's participation
was voluntary and self-paced. This was the situation that we have
studied for the past several years with hundreds of elementary school
children at three different sites--a Boys and Girls Club near Santa
Barbara, California (Mayer et al., 1997), a Boys and Girls Club
near San Diego, California (Schustack, Strauss, & Worden, 1997),
and public school after-school program in Boone, North Carolina
(Blanton, Moorman, Hayes, & Warner, 1997). Overall, our goal was
to determine what children learn from participating on a regular
basis in the Fifth Dimension program.
To create an
informal collaborative learning environment, each site had the following
characteristics based on a network of Fifth Dimension (5D) programs
developed by Cole (Cole, 1996; Nicolopolou & Cole, 1993): (a) The
participants were elementary school children, often from low-income
homes. (b) The program took place after school hours and was not
associated with the child's school. (c) Participation was voluntary
and there was no traditional teacher-student structure. (d) The
main activities were learning to use 30 to 50 off-the-shelf educational
programs; task cards explained each program and stated the criteria
for moving on to the next task. (e) Upon mastering a program, the
child selected the next activity from a set of options that depended
on the level of performance in the just-completed game. (f) There
were many opportunities for writing and creative expression such
as writing messages to a mythical Wizard using a word processor
and receiving written responses. (g) Learning was collaborative
with children often working together or with adult mentors called
Wizard's Assistants. (h) There was opportunity for students with
extensive experience to achieve the status of Young Assistant to
the Wizard. In each study, we compared students who participated
regularly in the Fifth Dimension over the course of an academic
year (e.g., more than 10 or 20 visits) to students with the same
characteristics who had not regularly visited the Fifth Dimension
club.
Educators have
long sought to determine the appropriate role of educational technology
for school-aged children. During the 1960's and 1970's the focus
was on programmed instruction, largely involving drill-and-practice
methods of instruction (Cognition and Technology Group at Vanderbilt,
1996; Mayer, 1999). During the 1980s, the emphasis was on learning
of computer programming, including experience in Logo environments
and programming in BASIC (Mayer, 1988; Papert, 1980), but in the
1990s the emphasis changed to learning to use software including
applications and educational games (Cognition and Technology Group
at Vanderbilt, 1996; Mayer, 1999). Although many strong claims have
been made for the power of educational technology throughout this
century, there has not been corresponding evidence documenting positive
cognitive consequences of exposure to computer-based learning environments
(Cognition and Technology Group at Vanderbilt, 1996; Cuban, 1986;
Mayer, 1988, 1999).
The
motivation for this project was to determine the cognitive consequences
of learning to use educational software in an informal, collaborative
environment using rigorous assessment methodologies. We were particularly
interested in whether exposure to educational computer programs would
promote literacy, that is, whether students with extensive experience
in the Fifth Dimension would be better able to perform literacy-related
tasks than students with little or no experience. Does learning to
use computers--or more specifically, learning to use educational software
on computers--improve a child's mind? What kinds of cognitive changes--if
any--occur over the course of a year in which computer-naive children
learn to use a series of educational programs in an informal and non-threatening
environment? In particular, we examined four kinds of possible cognitive
changes: computer literacy knowledge, comprehension skills, problem-solving
skills, and academic skills.
Cognitive
Evaluation Research Team
The first step in the project was to establish three cognitive evaluation
teams--at Appalachian State University (ASU), California State University
at San Marcos (CSUSM), and the University of California at Santa Barbara
(UCSB). The ASU team was headed by Bill Blanton and included Bobbie
Hayes, Gary B. Moorman, and Mark Warner. The team evaluated Fifth
Dimension sites that were operated as part of an after-school program
at four elementary schools in Boone, North Carolina. The CSUSM team
was headed by Miriam W. Schustack and included Rachelle Strauss and
Patricia Worden. The team evaluated a Fifth Dimension site operating
at the Boys and Girls Club in Escondido, California. The UCSB team
was headed by Richard Mayer and Richard Duran and included Amy Lavezzo,
Roxana Moreno, Jill Quilici, David Sanchez, Rebecca Simon, and Scott
Woodbridge. The team evaluated a Fifth Dimension site operating at
the Boys and Girls Club in Goleta, California.
The next step
was to develop research methodologies appropriate for evaluating
the Fifth Dimension's effects on children's literacy. The Fifth
Dimension is an innovative educational experiment in which children
learn in an informal technology-based after-school environment (as
described in the final report of "Using Information Technologies
in the Creation of Sustainable Afterschool Literacy Activities").
Anyone who has had the opportunity to visit a Fifth Dimension site
cannot help but come away with the impression that good things are
happening. However, more than good impressions are needed to convince
researchers, decision makers, and educational leaders who are engaged
in educational reform. When they see the Fifth Dimension, these
observers can legitimately ask, "Does it work?"
Educational reformers have recognized the role of non-school activities
in improving children's literacy. The informal learning experiences
afforded through out-of-school activities may be a valuable supplement
to traditional school curricula. As the demand for high-quality after-school
activities increases, it becomes increasingly important to examine
the effectiveness of well-designed programs for informal learning.
Our charge in this project was to provide a response to the deceptively
simple question, "Does it work?" In particular, the assignment given
the cognitive evaluation team was to conduct a traditional evaluation
of the cognitive consequences of participation in the Fifth Dimension.
Consequently, we focused our evaluation on the degree to which students'
activities in the Fifth Dimension would affect their performance on
academic literacy skills.
The search for problem solving transfer has a long and somewhat disappointing
history in educational and cognitive psychology (Mayer & Wittrock,
1996). Both in laboratory studies and field studies it is remarkably
difficult to find evidence that students who learn to solve problems
in one setting can transfer what they have learned to another setting
(McKeough, Lupart, & Marini, 1995). Given past difficulties in this
elusive search for problem-solving transfer, we accepted our assignment
to evaluate the Fifth Dimension with some trepidation.
In
order to answer questions about the effectiveness of the Fifth Dimension,
we found that it was first necessary to resolve five methodological
issues concerning the what, where, how, who, and when of the evaluation.
First, it was necessary to determine what to evaluate. In short, what
should be the dependent measure? We wished to select measures that
would tap the informal nature of the Fifth Dimension and at the same
time have some relevance to academic measures of literacy. Based on
the goals of the project--to promote literacy--we decided to examine
literacy skills that were related to Fifth Dimension activities. Given
the difficulties in demonstrating problem solving transfer in the
research literature, we invented or adapted tests to measure skills
ranging near to far transfer. The near transfer tests evaluated students'
learning of computer literacy knowledge, including being able to operate
a computer, being able to answer questions about how to operate a
computer, and remembering computer terms. These tasks tap skills that
are part of Fifth Dimension activities. The medium transfer tests
evaluated students' comprehension skills including the ability to
follow directions, to comprehend game instructions, and to comprehend
word problems, and their problem-solving skills, including the efficiency
with which they learned new math and grammar games. In the Fifth Dimension
students must understand and explain game instructions so these tests
tap literacy skills that are emphasized in the Fifth Dimension. The
far transfer tests evaluated basic reading and mathematics achievement.
Although these skills are not directly taught in the Fifth Dimension,
many of the educational games involved mathematics and reading activities.
Thus, the issue of what to evaluate was resolved by focusing on changes
in students' academic literacy skills that are related to Fifth Dimension
activities. These measures are described more fully in the results
section of this report.
Second, we needed to determine where to evaluate. In short, where
should we administer the dependent measure? Based on our interests
in examining a range of transfer, we decided to employ a range of
test settings. Some of the tests were embedded within authentic Fifth
Dimension activities such as playing an educational game at the site,
whereas other tests were administered as paper-and-pencil exercises
in a traditional classroom setting. Our goal was to determine whether
skills learned in the Fifth Dimension would influence not only similar
activities in the same setting but also traditional school tasks in
a classroom setting. Thus, this issue of where to evaluate was resolved
by focusing on both game-like activities embedded within a Fifth Dimension
setting and paper-and-pencil tests administered within a traditional
classroom setting. Third, perhaps the most challenging issue involved
the problem of how to create experimental control in an informal field
setting. This issue concerns the nature of the independent variable.
Among the challenges are that attendance in the Fifth Dimension is
voluntary. It is not enough to find that Fifth Dimension participants
produced a pretest-to-posttest gain in their performance, because
we need to compare their performance to the performance of a comparison
group that did not attend the Fifth Dimension. We developed five techniques
for creating comparison groups, each suited to particular situations
that arose at our sites. In method 1, we invited a group of students
to attend the Fifth Dimension (treatment group); for each treatment
student we identified a non-attending matched student from the same
grade level, teacher, level of English language proficiency, and gender
(comparison group). In method 2, we invited a group of students to
participate in the Fifth Dimension (treatment group); for each we
identified one or more non-attending students who attained the same
pretest score. In both method 1 and 2, each group took a pretest and
a posttest, so we could compare the pretest-to-posttest gains of the
two groups. In method 3, treatment and comparison students were matched
based on student characteristics as in method 1, but only a posttest
was administered. In method 4, treatment and comparison students were
matched based pretest score as in method 2, with the posttest score
as the major dependent variable. In both method 3 and 4, we compared
the posttest scores of the two groups. In method 5, when no non-attending
comparison group was available, we compared the test performance of
an experienced group of students (e.g., students who have attended
at least 15 times) and an inexperienced group (e.g., students who
are tested on their first visit but who eventually attended at least
15 times). Each method was adapted to field situations, and allowed
for some level of experimental control.
Fourth, the issue of who to evaluate was resolved by focusing on elementary
school children including a large proportion of language minority
students from the Latino community and a large proportion of girls.
We were particularly interested in including students who might not
otherwise have had access to computer technology in an informal setting.
Fifth, the issue of when to evaluate was resolved by testing students
after 10 to 20 sessions in the Fifth Dimension. This is a fairly short
treatment (at least, compared to many other educational activities),
but was necessitated by the voluntary nature of the program. Our primary
focus was on cognitive growth over the course of an academic year,
although in some cases our studies spanned a longer period (Mayer,
et al., 1997).
It is worthwhile to note that the research methodologies evolved over
the course of the project, through a process of pilot testing and
refinement. Several measures were not sensitive enough to produce
differences between the groups--including some measures of writing
and recalling spoken text. Other measures resulted in overwhelming
technical difficulties--such as computer glitches in trying to collect
on-line data as students learned a new computer game or staff errors
in administering an individual test of English language proficiency.
In addition, whenever possible we sought to replicate findings across
sites, so most measures were used at more than one site.
So what does it mean to say that the Fifth Dimension works or does
not work? Consistent with our mandate, we have constrained our evaluation
to a traditional examination of cognitive outcomes. However, even
in this seemingly straightforward evaluation scenario we had to adjust
our evaluations to fit the informal and voluntary nature of the Fifth
Dimension program. Each of the three sites that we studied intensively--ASU,
CSUSM, and UCSB--presented its own unique set of challenges and opportunities.
In the next part of this report, we summarize our major findings.
Cognitive
Consequences of Participation in the Fifth Dimension
The third major component of our research activities was to collect
data. Based on the methodologies described above, students who participated
in the Fifth Dimension were compared to non-participating students
on four major kinds of literacy measures--computer literacy skills,
comprehension skills, problem-solving skills, and academic skills.
Effects
of Fifth Dimension Experience on Computer Literacy
What do students learn from interacting with computers in the Fifth
Dimension? A straightforward answer is that students may learn specific
facts and procedures that are relevant to operating computers--that
is, what can be called computer literacy. Computer literacy represents
a form of near transfer--being able to solve problems and answer questions
that are similar to those experienced during learning. We tested this
hypothesis using three different measures of computer literacy--computer
term recognition test, computer knowledge test, and computer operation
test.
Skill
in remembering computer terms. In the word retention test students
viewed a series of 30 words on a computer screen with each word presented
for 1 second; some of the words were computer terms such as "cursor,
diskette, font, undo," and some of the words were neutral such as,
"winter, elephant, handle, develop." Then, the children were asked
to indicate which of two words had actually been presented for a set
of 34 word pairs (recognition test) and finally to write down all
the words they could remember (recall test). An aspect of computer
literacy is familiarity with computer terms, and memory research has
shown that students remember familiar words in a word list better
than unfamiliar words (Schustack, Strauss, & Worden, 1997). Thus,
superior retention of the computer terms is an indication of computer
literacy. At the one site where the study was conducted, students
showed improvement on this test after an average of 15 sessions of
experience in the computer-based after-school program whereas children
without experience in the program did not improve over the same time
period. These results demonstrate that part of what computer-experienced
students learn is key computer terms.
Skill
in remembering computer facts. The computer knowledge test is
a 40-item paper-and-pencil item test of basic computer information.
For example, one of the 40 items is: "To open a disk you position
the mouse pointer on the disk icon, and then: (a) drag the mouse while
holding it down, (b) double click mouse, (c) roll mouse around, and
(d) roll and then click mouse." The computer knowledge test is a school-like
test of computer literacy, covering the specific kinds of computer
facts that students were exposed to in the Fifth Dimension environment.
In the one site where a study was performed, students with extensive
computing experience showed an improvement on this test whereas equivalent
students who lacked computing experience did not. The results are
consistent with the conclusion that students gain basic computer knowledge
from their exposure to computers.
Skill
in operating a computer. The computer merit badge test consists
of a series of hands-on exercises that involve using software on a
computer. For example, one exercise is to "Find a Microsoft WORD document
called 'From the WIZ.' Read the letter, and then shut down the computer
and turn it off." To get credit for this exercise the student had
to find the file, open the file, quit the application, select shut
down, turn off the monitor, and turn off the computer. Thus, the computer
merit badge test represents a straightforward test of the specific
computer literacy skills that students were exposed to. In the one
site where a study was performed, students who had extensive computing
experience performed better than students with less or no experience
on successfully carrying out the exercises. These results show that
experience in a computer environment has a strong positive effect
on learning basic computer literacy skills.
Overall,
across three different ways of measuring computer literacy, there is
solid evidence that experience in the Fifth Dimension results in gains
in specific computer knowledge. Thus, part of the answer to the question
of "What is learned?" is that students learn facts and procedures about
computers. The next three sections explore the degree to which Fifth
Dimension experience results in learning that can be transferred to
new situations.
Effects
of Fifth Dimension Experience on Comprehension Skills
Learning to use educational software requires learning to follow instructions--including
both written and oral instructions. Each game comes with a task card
that contains a written explanation of how to set up and play the game.
In addition, students can ask their peers or mentors for oral instructions
on how to set up and play the game. Thus, over the course of figuring
out the written and oral directions for dozens of different games, students
may develop language comprehension skills particularly for the comprehension
of instructions. We tested this idea using a word problem comprehension
test, following procedures test, and directions comprehension test.
Comprehending mathematics
word problems. The word problem comprehension test consisted of
12 multiple choice items in which students were asked to select an equation
that matched a sentence in a word problem, to identify the numbers needed
to solve a word problem, or to specify the arithmetic operations needed
to solve a word problem (Mayer, et al., 1997). For example, one item
was:
Which number sentence is correct?
Ann and Rose have 20 books altogether.
a. Ann's books = Rose's books + 20
b. Ann's books + 20 = Rose's books
c. Ann's books + Rose's books = 20
d. Ann's books = Rose's books
Selecting
the correct answer (i.e., c) requires the ability to translate between
a formal and informal way of saying the same thing, a skill that may
be similar to understanding game instructions. At each of the three
sites, students who had extensive experience using educational computing
software in the Fifth Dimension scored higher on comprehending word
problems than did equivalent students who had not received this experience.
Comprehending
a procedure. The following procedures test consisted of 12 items
in which students where asked to follow a series of written instructions
involving moving within a matrix or the numbers of a calendar. For example,
one item consisted of an 8 x 8 matrix with a black dot in one of the
64 squares along with the following instructions: "Start at the black
dot. Go down 1. Go right 3. Where are you? Put an X in the block." Following
the procedure (i.e., placing an X in the seventh column and sixth row
if the dot began in the fourth column and fifth row) in this situation
is somewhat similar to following procedures for computer games. At two
of the three sites, students who had more exposure to learning the Fifth
Dimension scored higher on following directions than did equivalent
students who has less or no exposure.
Comprehending
game directions. The comprehending directions test was a cloze test
in which a portion of a passage describing how to play a new educational
computer game had a blank space instead of every seventh word. For example,
one version of the test consisted of a passage with 40 blanks as in
the following segment: "To jump to a scene, click __________ Henry's
shirt pocket. The note pad __________. Then, click on the options tab.
__________ options page appears. " Entering the correct words (i.e.,
"on," "appears," "the") indicates that the student comprehends the way
that game instructions are generally produced, a skill that educational
software users may develop. At two of the three sites, students who
had heavy exposure to the Fifth Dimension scored higher on comprehending
directions than did equivalent students who had less or no exposure,
and performance increased in relation to the amount of time students
attended the Fifth Dimension clubs.
Overall, across three different measures of language comprehension skill,
students who had Fifth Dimension experience in figuring out how to use
a wide variety of educational computer games tended to have developed
better comprehension skills than equivalent students who had not learned
to play educational computer games. These results encourage the idea
that Fifth Dimension experience can have a positive effect on certain
aspects of students' language comprehension skill. In short, in learning
to use educational software in the Fifth Dimension, students may learn
how to comprehend instructions.
Effects of Fifth Dimension Experience on Problem-Solving Skill
Learning to use educational software--particularly a wide variety of
educational games--may help students develop effective problem-solving
strategies. If students who master educational computing programs learn
generalizable skills, then they may be able to learn new programs more
easily than students who lack similar experience. To test this idea,
we examined improvements in students' game-playing skills in their ability
to learn a new math game (i.e., Puzzle Tanks Test) and a new language
arts game (i.e., Grammar Games Test).
Learning
to play a new math game. The puzzle tanks test is a paper-and-pencil
version of a mathematics computer game called Puzzle Tanks in which
students must pour juice among various-sized tanks in order to obtain
a specified amount of juice (Sunburst, 1996). For example, in one problem
students must use tanks that can hold 7 and 2 units respectively in
order to obtain 12 units. The solution is to fill the 7-unit tank and
then empty it into the final tank (to get 7 units); then fill the 7-unit
tank, send 2 units from that tank to fill the 2-unit tank, and then
empty the remaining 5 units from the 7-unit tank into the final tank
(to get 12 units). In our research (Mayer, Quilici, & Moreno, in press),
the student read written instructions for the game and then participated
in solving a series of three problems, with an experimenter giving immediate
feedback after each move. The test was administered individually to
students at their home schools as a dynamic assessment and none of the
students had ever been exposed to this game or one like it. This represents
a problem-solving transfer test because students must apply their skills
in comprehending game instructions and inventing solution strategies
to a new problem situation. Importantly, at the one site where a study
was conducted, students who had frequently attended the Fifth Dimension
computer club made fewer errors and generated more sophisticated problem-solving
strategies than did equivalent students who lacked educational computing
experience. This result demonstrates that experience in an educational
computing environment can promote skills that transfer to new problems.
At another site, analyses of video-taped episodes of students playing
Puzzle Tanks on-line showed that student learning goes beyond mastery
of problem-solving strategies (Blanton, 1997). The results reveal that
students learn to make rapid and appropriate changes in tool use, invent
tools, and use the same tool independently and simultaneously. Students
also learn to move back and forth among levels of joint activity, such
as collaborating with peers and providing and receiving guidance.
Learning
to play a new grammar game. Grammar Games is a children's educational
computer game in which students must copyedit several screenfulls of
text (Davidson & Associates, 1994). On each screen, several words were
shown with boxes around them. For each of these boxed words, the student
had to decide whether to change the word or leave it alone. On each
screen, several of the boxed words violated the rules of subject-verb
agreement. Students were given two passes at each screen. After the
first pass, the boxes around correct words disappeared, but boxes remained
around all the words that were still wrong. Learning to play this game
successfully involves problem-solving transfer because none of the students
had ever seen it before. At the one site where a study was conducted,
children with extensive experience in the Fifth Dimension scored higher
on the grammar games test than did equivalent students who had less
or no experience (Mayer, Schustack, & Blanton, 1999). This result provides
additional support for the idea that exposure to educational computing
environments can foster transferable problem-solving skills.
Overall, the results across two different games show that part of "what
is learned" in an educational computing environment is the ability to
learn to play a new game. Success at learning a new game requires both
the ability to comprehend the instructions and to devise an appropriate
solution strategy. Apparently, students who are exposed to a wide variety
of educational computing experiences develop skill in comprehension
and planning that can transfer to new situations.
Effects
of Fifth Dimension Experience on Academic Skills
Improving
reading and mathematics achievement. The reading and mathematics
achievement tests are statewide standardized tests of basic academic
skills administered at the end of each academic year (Blanton, Moorman,
Hayes, & Warner, 1997). These tests represent far-transfer measures
because the computer environment was not designed to provide systematic
instruction in reading and mathematics. However, using language and
mathematics skills were necessary for playing many of the educational
computer games that students learned to master, so students may have
acquired some content knowledge as well as some confidence. At the one
site where this study was conducted, students who had extensive Fifth
Dimension experience outscored their peers on improvement over their
prior year's score on both reading and mathematics achievement. These
results are consistent with the idea that participation in an educational
computing environment results in far transfer--that is, in learning
that goes beyond simple retention of specific computer facts and procedures.
Major
Accomplishments
It is easy to find strong claims for the power of computers to improve
students' minds but difficult to find scientifically valid research
testing these claims (Cognition and Technology Group at Vanderbilt,
1996; Mayer, 1999). This report summarizes some important cognitive
changes gained by children who used educational technology in informal,
collaborative learning environments. The results are strong, even though
the educational technology mainly involved using off-the-shelf programs
running on low-tech machines. Our research provides encouraging evidence
that appropriate experience with educational technology within a Fifth
Dimension environment can promote important cognitive changes in children,
including improvements in content knowledge about computing, strategies
for comprehending written instructions, strategies for devising problem-solving
plans, and even in basic academic skills.
In
short, this project allows us to provide an answer to question, "Does
the Fifth Dimension work?" When success is defined in terms of improvements
in children's literacy, we are able to point to positive cognitive changes
in several key areas that are attributable to children's participation
in the Fifth Dimension. The results are all the more compelling given
that they were obtained across a large collection of measures and at
three different sites.
Most projects aimed at improving cognitive growth in children are never
subjected to the kinds of controlled experimental evaluations employed
in this project, and those projects that are evaluated often fail to
generate sustainable cognitive effects (Zigler & Muenchow, 1992). In
contrast, this project joins the ranks of a small but growing collection
of scientifically valid evaluations of innovative educational programs
aimed at improving important cognitive skills (McKeough, Lupart, & Marini,
1995). Although this project does not prove that all Fifth Dimension
sites are effective in promoting all cognitive skills, it does provide
encouraging evidence: Methodologically rigorous studies at three different
sites yielded evidence of cognitive growth in literacy skills related
to communicating about educational computer games. Thus, the major accomplishment
of this project has been to document some areas in which the Fifth Dimension
promotes children's literacy.
As requested by the Mellon Foundation, we sought to communicate our
findings with educational researchers and practitioners. We are pleased
to report that we accomplished this goal mainly through publications
in peer-reviewed research journals (e.g., 5 articles in the Journal
of Educational Computing Research and 1 article in Educational Technology)
and presentations at peer-reviewed national conventions (e.g., 7 presentations
at the annual convention of the American Educational Research Association).
The full citations of these products are listed on pages 2 and 3 of
this report.
Limitations
and Future Directions
We did not compare one Fifth Dimension site against another, nor did
we systematically vary the features of any one site, so it is not is
not possible to pinpoint which features of the Fifth Dimension contribute
most strongly to the effects we obtained. Similarly, we did not compare
the Fifth Dimension against other types of after-school computer clubs
so it is possible that other types of clubs could be as effective or
even more effective in promoting cognitive growth in children. Further
research would be needed to pinpoint the features of an after-school
computer club that are most important.
Our findings are based solely on three Fifth Dimension sites--in Boone,
North Carolina; in Escondido, California; and in Goleta, California.
The generalizability of our conclusions are limited to the extent that
these sites may differ from others in the Fifth Dimension network. The
philosophy of the Fifth Dimension allows for each site to develop a
unique program that shares only some of the global features of other
Fifth Dimension programs, so it is not possible to conclude that the
same levels of cognitive growth would occur at all sites. Based on consultations
with the Mellon Foundation, it was determined that the most efficient
strategy for the cognitive evaluation was to focus intensively on three
Fifth Dimension sites that were most willing to be repeatedly evaluated.
The sites differed in their methods of operation, locations in the community,
and in the demographics of the attending children, but all were committed
to active and ongoing participation in the cognitive evaluation. Future
research would benefit from a broader participation of sites in the
evaluation, but this limitation is mitigated somewhat by the fact that
the current evaluation was based on a set of three diverse sites.
Although our research documents that cognitive growth can be attributed
to children's participation in the Fifth Dimension, we did not conduct
microgenetic studies of how that growth takes place. Similarly, although
our research documents that cognitive growth occurred in Latino students
who spoke Spanish as their first language, we did not conduct studies
directly investigating the role of language and culture. These two tasks
were the responsibilities of two complementary evaluation team, whose
reports should be seen as complements to this one aimed at evaluation
of cognitive outcomes.
The cognitive evaluation has yielded several evaluation instruments
and methodologies that could be used at Fifth Dimension sites that wish
to conduct self-evaluations. In order to disseminate these evaluation
instruments and methodologies to all Fifth Dimension sites, we plan
to develop a web site. The site will provide copies of the instruments,
descriptions of evaluation procedures, and back-up information on how
to conduct an evaluation of a Fifth Dimension site. These tasks will
be carried out as a follow-up project supported by the Mellon Foundation.
Blanton,
W. (1997, March). Product and process research and evaluation on
informal learning environments: Theoretical and methodological issues.
Paper presented at the meeting of the American Educational Research
Association, Chicago, IL.
Blanton,
W. E., Moorman, G. B., Hayes, B. A., & Warner, M. L. (1997). Effects
of participation in the Fifth Dimension on far transfer. Journal
of Educational Computing Research, 16, 371-396.
Cuban,
L. (1986). Teachers and machines: The classroom use of technology
since 1920. New York: Teachers College Press.
Cognition
and Technology Group at Vanderbilt (1996). Looking at technology
in context: A framework for understanding technology and education
research. In D. C. Berliner & R. C. Calfee (Eds.), Handbook of
educational psychology (pp. 807-840). New York: Macmillan.
Cole,
M. (1996). Cultural psychology: A once and future discipline.
Cambridge, MA: Harvard University Press.
Davidson
& Associates. (1994). Grammar games [Computer program]. Torrance,
CA: Author.
Mayer,
R. E. (Ed.). (1988). Teaching and learning computer programming.
Hillsdale, NJ: Erlbaum.
Mayer,
R. E. (1999). Instructional technology. In F. Durso (Ed.), Handbook
of applied cognition (pp. 551-570). Chichester, England: Wiley.
Mayer,
R. E., Schustack, M. & Blanton, W. (1999, March-April). What do
children learn from using computers in an informal collaborative
setting? Educational Technology, 39(2), 27-31.
Mayer,
R. E., Quilici, J. H., & Moreno, R. (in press). What is learned
in an after school computer club? Journal of Educational Computing
Research, 19.
Mayer,
R. E., Quilici, J. H., Moreno, R., Duran, R., Woodbridge, S., Simon,
R., Sanchez, D. & Lavezzo, A. (1997). Journal of Educational
Computing Research, 16, 353-370.
Mayer,
R. E. & Wittrock, M. C. (1996). Problem-solving transfer. In D.
C. Berliner & R. C. Calfee (Eds.), Handbook of educational psychology
(pp. 47-62). New York: Macmillan.
McKeough,
A., Lupart, J., & Marini, A. (1995). Teaching for transfer: Fostering
generalization of learning. Mahwah, NJ: Erlbaum.
Nicolopolou,
A. & Cole, M. (1993). The Fifth Dimension, its playworld, and its
institutional contexts: The generation and transmission of shared
knowledge in a culture of collaborative learning. In E. A. Foreman,
N. Minnick, & C.A. Stone (Eds.), Contexts for learning: Sociocultural
dynamics in children's development (pp. 283-314). New York:
Oxford University Press.
Papert,
S. (1980). Mindstorms. New York: Basic Books.
Schustack,
M. W., Strauss, R., & Worden, P. E. (1997). Learning about technology
in a non-instructional environment. Journal of Educational Computing
Research, 16, 337-352.
Sunburst
(1996). Puzzle tanks [Computer program]. Pleasantville, NY:
Author.
Zigler,
E. & Muenchow, S. (1992). Head Start: The inside story of America's
most successful educational experiment. New York: Basic Books.
|
