Improved Long-Term Retention for Identical
and Similar Material found on Chapter Quizzes

Bryan H. Hoyt, Ilie P. Vasilescu, Kristina Feeser, and James Horton
University of Virginia’s College at Wise

     Differences in the way instructors administer multiple choice quizzes affect student performance (Stiggins, 2001). Instructors use four common different administrations of multiple-choice tests: take-home, open-book, collaborative, and closed-book tests and quizzes. Researchers commonly compared these first three to the fourth, closed-book tests or quizzes (Hoyt, Vasilescu, & Feeser, 2006; Weber, McBee, & Krebs, 1983). Even though research on tests and quizzes extends back over several decades, important questions remained unanswered.
     Haynie (1997) explored how the testing with closed-book tests affects students’ retention of information. He conducted a study demonstrating that taking a test increased the retention of information. Haynie’s study compared three groups. Two groups received a study booklet and information about an upcoming test, but only one group received the initial test. The third group, the control group, received the study material and the information that they should study as if preparing for a test. The researcher collected the study materials from the students in the three groups. Three weeks later, the researcher gave the groups a surprise test. The researcher found that only the group which received an initial test immediately after studying the material performed significantly higher on a surprise test compared to the other two groups. Haynie concluded that the initial test produced the gain in retention of information.
     In addition to using tests to facilitate an increase in long-term retention, the type of questions on closed-book tests may influence retention of information as well. Haynie (1994) examined two closed-book testing methods: multiple-choice and short answer essays as aids to the retention of information. Again, as in the earlier study, the researcher found that participants, who received a test immediately after learning compared to those who studied the material without being immediately tested over the material, increased long-term retention of information (Haynie, 1997). In addition to the test’s effect on the retention of information, Haynie concluded that the testing method also influenced the retention of information. Those in the group that received a test immediately after learning were further divided into two groups based on the testing method, either essay or multiple-choice. Participants who received a multiple-choice test showed a significantly higher level of retention compared to those who received an essay test.
     Marsh (1984) tested college students across a variety of disciplines using either a closed-book or a take-home multiple-choice test. A surprise test given to all participants after the scheduled multiple-choice test measured how much information the students retained. The students who took the closed-book test exhibited significantly more retention of information than those who took the take-home test. In addition, the researcher found that the closed-book group obtained significantly higher scores on questions compared to the take-home test group (Marsh, 1984). Marsh’s findings seemed to concur with an earlier study by Gay and Gallagher (1976), who found that participants scored higher on a surprise test given toward the end of the semester when they received a closed-book exam compared to take-home exam, over the same material received earlier in semester.
     More recently Haynie (2003), seemingly supported the findings of Gay and Gallagher (1976) and Marsh (1984) concluded that closed-book tests facilitate retention of information more than take-home tests. Haynie found that testing students immediately after learning facilitated retention. The researcher found, based on the results of a surprise test given to both groups three weeks after learning that the students who received an earlier test immediately after studying the material, performed significantly better than the control group, who received no test following the studying of the material. In this study, Haynie also compared take-home tests and in-class closed-book tests. The group of students in his study who received the test immediately after studying received either a multiple-choice closed-book test or a take-home home essay test. Those who received the closed-book test obtained significantly higher scores on the second test given after a three-week interval. Haynie suggested this may be due to participants only looking for answers to questions, rather than trying to understand the material.
     One of the limitations of some of these studies involved the time interval between the first assessment and the follow-up assessment designed to measure retention. For example, Haynie (1994, 1997, & 2003) used only a three week interval and Marsh (1984) used a one week interval between assessments.
     A second limitation of some of these earlier studies involved a lack of control for the variable of question difficulty. Besides the long-term retention of information, other researchers showed interest in the test question’s level of cognitive domain (Weber, et al., 1983). Bloom’s taxonomy became the traditional benchmark for the levels of cognitive domain (Bloom, Englehart, Furst, Hill, & Krathwohl, 1956). From this taxonomy, the researchers in this present study considered three degrees of question difficulty. Ranging from low to high difficulty, the main process involved by the four degrees included recall, analysis, comparison, and inference. Recall questions asked students to remember and recite facts, ideas, definitions, or rules. In this paper, the researchers referred to them as factual questions. The other three types of questions asked students to analyze, to make comparisons, and to draw inferences. In this paper, the researchers referred to them as conceptual questions. Educational researchers criticized classroom-tests for relying heavily on the recognition of “knowledge” (the first level of the taxonomy) and not utilizing higher levels of cognitive domain such as comprehension, application, and analysis of information (Fleming & Chambers, 1983).
     A third possible limitation of some of these earlier studies involved the limited types of test administrative conditions. Some studies made comparisons using only closed-book tests (Haynie 1994, 1997). Other studies compared closed-book to take-home tests (Haynie, 2003; Marsh, 1984; Gay & Gallagher, 1976).
     In an attempt to address some of these issues, Hoyt, Vasilescu, and Feeser (2006) compared the retention of information on four conditions of administering multiple-choice chapter quizzes: take-home, open-book, collaborative, and closed-book. They found that the administration of take-home, open-book, and collaborative quizzes compared to closed-book quizzes significantly improved long-term retention on midterm and final closed-book exams, when the questions on the quizzes were identical to those on the exams. The researches also noted that information first tested in a conceptual type question on chapter quizzes, produced significantly higher levels of long-term retention on closed-book midterm and final exams compared to information represented by factual questions.
     As an extension of Hoyt, Vasilescu, and Feeser’s (2006) study, long-term retention of identical material could be used to prime similar material stored in memory. Collin and Loftus (1975) proposed a complex associated network of memories linked by relationships. Termed the spreading activation theory, Collin and Loftus suggested that the activation of one concept in memory may prime a related concept and benefit memory performance.
     This study aimed to address the conjoint effect of the method of quiz administration and three levels of similarity of conceptual questions on the long-term retention of material. Four types of quiz administration were utilized: take-home, open-book, collaborative, and closed-book. The long-term retention measurements consisted of a midterm exam given eight weeks into the semester. The questions occurred in three levels of similarity: completely similar questions repeated from chapter quizzes; partially similar questions on the same topic as the completely similar question but asked in a different way; and dissimilar questions on a new topic not found on chapter quizzes. In this study we attempted to determine whether or not a priming effect occurred between the completely similar questions and the partially similar questions that would facilitate long term retention of the material.

Methods

Participants
     Participants included 107 undergraduate typical college-aged students. Non-psychology majors made up the majority. These volunteers came from three Introduction to Psychology classes taught by two different instructors over a full semester at a small, rural, public liberal arts college. The three classes used the same textbook, syllabus, quizzes, tests, and covered the same material in the same order. The participants consisted of 55 females and 52 males of which non-Caucasians comprised approximately 10 percent of the group. Participants received extra credit as an incentive for participation in this study. The researchers offered an alternative method for earning extra credit for anyone not wishing to participate.
Materials
     The researchers constructed all quizzes in the same manner. Each chapter quiz contained 21 multiple-choice conceptual questions taken from the Hockenbury and Hockenbury (2006) test bank. Each of the multiple-choice questions offered four choices, one of them being the correct answer. The researchers took seven questions from each quiz and placed them on the midterm exam; these questions comprised the completely similar questions. Next, the researchers selected seven similar questions for the midterm from the test bank—labeled partially similar questions. The topic of this second group of questions matched the topic but not the wording of the completely similar questions. Finally, the researchers selected dissimilar questions from the test bank on a new topic not found on the chapter quizzes and placed them on the midterm.
Procedure
     After the completion of each chapter of the Introduction to Psychology text, participants completed a 21 item quiz. The instructors administered the chapter quiz as a take-home, open-book, collaborative, or closed-book. Each instructor randomly used one of these four types of quiz administration methods, but always a method different from the other two instructors. Students completed all quizzes during regular class time, with the exception of the take-home quiz. Instructors randomly assigned a partner for the collaborative type quiz and participants could discuss the questions and compare answers with their partner before choosing an answer, but could not use their textbook or notes. Answers to each question remained the responsibility of the individual participant and each individual recorded his or her response on a separate answer sheet. Instructors handed out the take-home quiz during a class period. They instructed participants that they may use their textbooks and notes to answer the questions and should return the quiz at the beginning of the next class period.
     At the midpoint in the semester (eight weeks), instructors administered a closed-book midterm exam using completely similar, partially similar, and dissimilar questions. Instructors collected all chapter quizzes from participants after each quiz and no viewing of questions occurred after a quiz. Finally, the researchers compared the responses from the four types of chapter quizzes and the three types of questions found on the midterm. They used the results from the midterm exam as a measure of information retention and whether or not this retention would extend to related material found on the partially similar questions.

Results

     The researchers conducted repeated measures ANOVA to compare the effect of the four types of administration of quizzes: take-home, open-book, collaborative, and closed-book. The comparison, also, included three types of questions: completely similar, partially similar, and dissimilar and the performance on midterm exam as a measure of long-term retention.
     A statistically significant difference in student performance occurred on the midterm F (3, 104) = 2.63, p = .05 among the four types of administration of quizzes. Next, a statistically significant difference in student performance occurred on the midterm F (2, 105) = 62.39, p = .00 in a comparison of the effects of the three types of questions.
     Subsequently, the researchers combined the four types of administration of quizzes into two categories. Open-book type administrations included open-book and take-home quizzes that allowed full access to the textbook and student notes during the quiz. Closed-book type administrations included closed-book and collaborative quizzes that did not allow any access to the textbook or notes during the quiz. In a comparison of the effects of open-book and closed-book type of quizzes, a statistically significant difference in student performance occurred on the midterm F (1, 106) = 8.73, p = .00 (see figure 1).
-------
Fig. 1
-------

     A statistically significant interaction occurred when comparing the four test administration type with the three question types F (6, 101) = 3.98, p = .00. An interaction occurred with the collaborative test administration between the partially similar and dissimilar questions (see figure 2).
-------
Fig. 2
-------
     The researchers used two tailed t-tests to further analyze differences in responses to the three types of questions. Students scored significantly higher on the completely similar questions compared to the partially similar questions on the midterm t (106) = 7.89, p = .00. Again, students scored significantly higher on partially similar questions compared to dissimilar questions on the midterm t (106) = 4.19, p = .00. Finally, students scored significantly higher on completely similar questions compared to dissimilar questions on the midterm t (106) = 10.10, p = .00.
Discussion
     This study compared the retention of information following four conditions of the administration of multiple-choice chapter quizzes: take-home, open-book, collaborative, and closed-book. The researchers contended that take-home, open-book, and collaborative quizzes, unlike closed-book chapter quizzes, provided students during the testing with a supplementary access to task-relevant information. Under these conditions, the chapter quiz could represent a learning opportunity and improve long-term retention of information. The take-home quizzes allowed students unlimited time and the use of both their textbook and notes. The open-book quiz format allowed students to use their textbooks, but students had a 30 minute time limit to complete the quiz. The collaborative quiz method allowed students to discuss each question with a randomly assigned partner, but in all other ways resembled a closed-book quiz.
     Unlike previous studies (Gay & Gallagher, 1976; Marsh, 1984; Haynie, 2003), this study showed that chapter quizzes that involved some form of supplementary access to task-relevant information during the testing represented learning opportunities that significantly improved long-term retention of information as compared to closed-book quizzes. The researchers compared open-book type quizzes, which included the open-book and take-home administration with closed-book type quizzes, which included closed-book and collaborative administration. Students performed significantly better the midterm exam on questions that were completely similar or partially similar that came from open-book type compared to the closed-book type quizzes. Again, the researchers suggest that the supplementary learning that takes place during an open-book type quiz improves a student’s long-term retention of information.
     The researchers also found that students performed significantly better on completely similar questions found on the midterm, those that came directly from chapter quizzes, compare to partially similar and dissimilar questions. The topic of partially similar questions found on the midterm matched the completely similar questions, but not the wording of the question. The researchers chose dissimilar questions from the test bank for the midterm that contained a new topic and wording of the question. Likewise, students performed significantly higher on similar questions compared to dissimilar questions.
     The questions on the midterm that originated from chapter quizzes, the completely similar questions, the researchers suggest may serve as prime for the partially similar questions. Collin and Loftus (1975) proposed a complex associated network of memories linked by relationships. Collin and Loftus in this spreading activation theory suggested that the activation of one concept in memory may prime a related concept and the partially similar questions matched the completely similar questions in terms of the topic covered.
     However, as may be noted in figure two, a surprising interaction occurred on the midterm with questions that originated from the collaborative administration of quizzes. Students performed higher on dissimilar questions as compared to partially similar questions. This interaction may be due to the set of questions, either their content or selection.
     Most studies failed to examine performance on conceptual questions (Haynie, 1994, 1997, & 2004), which differ in their relative difficulty from factual questions. Differences in the findings from this current study and Marsh’s (1984) study may be attributed to how the comparisons were made in the two studies. Marsh retested students with a surprise test one-week after the researcher’s first test and compared only a closed-book with a take-home test. The current study used four different conditions of administration of multiple choice quizzes. The students’ midterm exams comprised the measure of the long-term retention of information. The researchers gave the midterm exams eight weeks into the semester.
     For those who teach, effectiveness may be limited by looking at quizzes only as assessment instruments as opposed to supplementary learning opportunities. For a long time, instructors utilized closed-book quizzes as a favorite form of assessment and, if carefully constructed, they may give a relatively accurate appraisal of students’ learning. However, open-book type quizzes seem to actually facilitate learning, long-term retention, and perhaps a deeper understanding of the material. This study found that the increase in long-term retention seems to extend to related or similar material as did not originally appear on chapter quizzes.
                   Based on the results of this study, researchers need to replicate the findings of this study and also attempt to better understand how retention of information may extend to related information. Further study might yield a better understanding of how each type of quiz administration influences long-term retention of information and how the type of question asked best facilitates learning.

References

Bloom, B., Englehart, M., Furst, E., Hill, W., & Krathwohl, D. (1956). Taxonomy of Educational Objectives Handbook 1:
       Cognitive Domain. New York: David McKay Co.
Collins, A. M.,, Loftus, E. F. (1975). A spreading activation theory of semantic processing. Psychological Review, 82,
       407-428.
Fleming, M.., & Chambers, B. (1983). Teacher-made quizzes: Windows on the classroom. In W. E. Hathaway (Ed.),
       Testing in the schools: New directions for testing and measurement (pp. 29-38). San Francisco: Jossey-Bass.
Gay, L., & Gallagher, P. (1976). The comparative effectiveness of tests versus written exercises. Journal of Educational
       Research, 70, 59-61.
Haynie, W. J. (1994). Effects of multiple-choice and short answer tests on delayed retention learning [Electronic Version].
       Journal of technology Education, 6(1), 32-44.
Haynie, W. J. (1997). Effects of anticipation of tests on delayed retention learning [Electronic Version]. Journal of Technology
       Education, 9(1), 20-46.
Haynie, W. J. (2003). Effects of take-home tests and study questions on retention [Electronic Version]. Learning in Technology
       Education, 14(2), 6-18.
Hockenbury, D. & Hockenbury, S. (2006). Test Bank, NY: Worth.
Hoyt, B., Vasilescu, I. P., & Feeser, K. (2006). Evaluation of Assessment Methods and Long-Term Retention of Knowledge.
       The National Social Science Journal, 16(1), 57-63.
Marsh, R. (1984). A comparison of take-home versus in-class exams. Journal of Educational research, 78(2), 111-113.
Stiggins, R. J. (2001). The unfulfilled promise of classroom assessment. Educational Measurement: Issues and Practice,
       20(3), 5-15).
Weber, L. J., McBee, J. K., & Krebs, J. E. (1983). Take home quizzes: An experimental study. Research in Higher Education,
       18(4), 473-483.
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Retention of information from open-book type quizzes
compared to closed-book type quizzes

Figure 1: Student responses on the midterm showing retention of information of material originating
from open-Book type quizzes to closed-Book type quizzes.

Interaction of quiz-type and question type

Dissimilar

Completely Similar

Partially Similar

                       Figure 2: Interaction of quiz-type and question type on midterm.