Using the Quiz Tool in a Course Management System to Improve
Online Learning Effectiveness: An Empirical Analysis

Nghiep Nguyen, Al Cripps, and Hoang Nguyen
Middle Tennessee State University

Objective
     In the current information era, online learning is quickly assuming an important role in education. Hence, there is a critical need to render online learning as effective as possible. Our empirical study attempts to foster online learning effectiveness taking students beyond the rote learning level (based on memory) to a meaningful integrated learning level along with critical thinking.
     According to transformative learning theory (Cranton 1993, p. 3), students learn more if they are engaged in learning activities. Similarly when students are actively involved, they learn more than when they are passive recipients of instructions (Cross 1987, p. 4). Students learn what they care about and remember what they learn (Ericksen 1984, p. 51). Instructors should strive to find ways to encourage students to study the concepts, identify misunderstandings, and resolve confusion. In the classroom, instructors can work on problems at the end of the chapters and resolve any confusion that may arise or discuss some typical test questions to clarify possible misunderstandings that students may have. However, class time is limited and benefits of this strategy are also limited.
    Online courses have an appropriate electronic facility for reaching these levels of learning, the quiz tool. This paper discusses the logic and usefulness of the quiz tool, presents a theoretical framework, proposes hypotheses, discusses data and empirical tests of hypotheses for improving online learning effectiveness.
Logic
     Our study is based upon the three-step process implied in existing learning theories: active learning, constructive learning, transformative learning, and cognitive learning theories. Active learning (Meyers and Jones 1993) requires students to talk, read, write, apply, and reflect on the learned materials. In other words, active learning is defined as anything that “involves students in doing things and thinking about the things they are doing” (Charles Bowell & James Eison, 1993). Constructive learning is based upon the discovery process (Piaget, 1973). According to Piaget, “to understand is to discover, or reconstruct by rediscovery.” It involves finding something new and incorporates to one’s understanding. Transformative learning is defined as the process of learning through critical self-reflection derived from one’s experience and resulting from reformulation of knowledge (Jack Mezirow, 1990). It also involves seeing something new and incorporating into one’s understanding. Finally, “cognitive learning involves the acquisition and reorganization of the cognitive structures through which humans process and store information” (Good and Brophy, 1990, p. 187) where cognitive structure is an internal knowledge structure, called “schema” which may be combined, extended or altered to accommodate new information (Brenda Mergel, 1998, p.7). There is a three-step process implied by these theories: (a) use the existing schema to understand, to judge others, and to decide, (b) when interacting with others, identify or discover incorrect understanding of the world around or areas of distortions in our existing schema, and (c) to establish the “mental equilibrium”, we have to resolve the distortions in understanding. While active learning focuses on the activities that one has to perform, constructive learning looks at the three-step process itself, and transformative learning emphasizes the results of the process. Finally, cognitive learning validates and stores the learning results in the more permanent format as a newly altered schema. The three-step learning process is the volley of view of learning as referred by Pratt & Paloff (1999, p. 119). In interaction, the accelerated three-step process manifests in the form of “all of a sudden, a new idea, concept or thought emerges.” (Draves 2002, p. 140). As a result, the learner moves to higher levels of understanding. In general, if the learning process as described is promoted and facilitated, learning will occur. The more learning process cycles one goes through, the faster one learns. Practice quizzes promote interaction between students and materials. Appropriately used, practice quizzes also create interaction between students and instructor. Both interactions facilitate the three-step learning process discussed above. That is why Draves (2002, p. 31), stated that interaction is the heart and soul of online teaching (Draves 2002, p. 31). Simply put, interaction facilitates learning process and helps achieve more learning.
     Interaction can be between the instructor and students, among students, and students and instructional resources (Boettcher & Conrad 1999, p. 17). “The most significant feature of the networked computer is its ability to serve as a communication device, the very essence of teaching and learning processes” (Boettcher and Conrad 1999, p. 9). In order to interact meaningfully and productively, one has to promote and facilitate the three-step learning process. One must read the materials, reflect upon them, and apply them. While taking practice quizzes, there are ample opportunities for one to identify the deficiency in one's understanding of the subject matter. With help from classmates and instructor to resolve difficult questions, resolutions to conflicts easily emerge.
     In the traditional classroom, interaction is limited (Draves 2002, p. 159) and so is the effectiveness of active learning and the three-step learning process. In an online course, interaction is required and facilitated. Course management systems such as CourseInfo or WebCT provide online interaction as an important learning module.
Theoretical framework and hypotheses
     The quiz tool in a course management system (such as WebCT, D2L, etc.) can import a test bank from the text publisher, set up quizzes, and online exams to randomly select questions from the question database. In a course management system, instructors can provide feedback for each answer, edit a question, and add new questions of various categories (multiple choice, true and false, short question, essay question, fill-in-the-blank question, and matching question). In our study, we taught multiple sections of a course titled “Principles of Financial Management” using both an online course management system and a traditional classroom. Below describes what we have done to promote active learning in these sections during the Spring 2006 through Spring 2007.
     For each chapter, we setup a question database of 70 to 120 questions from which the corresponding practice quiz randomly selects eight questions anytime a student accesses and takes a practice quiz. Because quizzes are required and contribute to the final grade, students are obligated to practice. Because the quiz questions are randomly selected from the same question database as the exam questions, students are motivated to take the quizzes repeatedly to be exposed to as many questions in the database as possible. Besides, the score of a quiz is the average of all scores of all the times the quiz of the chapter is taken, students are motivated to do better the next time around. Hence they will need to review the materials to resolve any confusion before practicing the quiz again. With practice quizzes we expect students to perform better on each of the three exams for the semester.
Hypothesis 1: The more often students take practice quizzes, the better score of the corresponding exam they will have.
     However, we are concerned that our strategy will encourage students to memorize questions instead of learning and understanding the concepts. So we suspect students may perform worse if we don’t have exactly the same questions in the two question databases (one for quizzes and one for exams, both testing the same concepts).
Hypothesis 2: Students will perform worse if we don’t have exactly the same questions in the two question databases.
Data and empirical results
     Data are collected from both the traditional class and the online class. Each exam covers the materials of three to four chapters (hence three to four quiz grades). We recorded the total number of times a student takes quizzes for a corresponding exam and the score of that exam. We have data for three exams of the online course and only data for the first two exams of the traditional lecture course. We computed the correlation coefficients (between the total number of the times a student takes quizzes for a corresponding exam and the score of that exam) and performed the two-tailed significant test. Below are the empirical results.
Hypothesis 1 Results:

The traditional class exam 1: the correlation coefficient is 0.53109 with the number of observations being 35; there is a significant correlation with 99% confidence level.
The traditional class exam 2: the correlation coefficient is 0.309122 with the number of observations being 39; there is a significant correlation with 90% confidence level.
The online class exam 1: the correlation coefficient is 0.426455 with the number of observations being 43; there is a significant correlation with 99% confidence level.
The online class exam 2: the correlation coefficient is 0.330133 with the number of observations being 42; there is a significant correlation with 95% confidence level.

So hypothesis 1 being confirmed, we conclude that practice quizzes help students perform better in the exams.
Hypothesis 2:
     To test hypothesis 2, we used data from the online class with modified questions in the question database for exams. However, exam one didn’t have any modified questions, exam two had a substantial number of modified questions, and exam three had only one modified question. If students in fact tried to memorize quiz questions for test questions, then we expect the average scores of exam two and exam three to be significantly different. We performed the t-statistic test for the significant difference between two sample means (Mode, Elmer B., 1964, p. 164). With t=0.947 and degree of freedom of 74, the average scores are significantly different at the 80% level of confidence. So hypothesis 2 being confirmed, we conclude that students in fact tried to memorize quiz questions for test questions.
Improvements and Additional Hypothesis
     The confirmation of hypothesis 2 alarmed us that if we used quiz questions and exam questions from the same question databases, students can achieve rote learning. We must strive to achieve high-order learning. The three levels of learning (Brightman, 2003, p. 9) are rote learning (list, state, itemize, etc.), meaningful-integrated learning (explain, interpret, connect, compare and contrast, differentiate, integrate), and critical thinking (restructure, discover). Rote learning is based on memory. We need to move students beyond this basic level. During the summer of 2006, we implemented the following improvements:

- We set up another question database for exams. This database consists of questions very similar to those in the database for quizzes. They test the same concepts, sometimes exactly the same phrasing but the data are different. A question might look very much the same as in the quiz except for the key word, which if different will change the answer. The goal is to force students to study the concepts in depth and read the question very carefully.
- We set up a sample exam question showing an example of what we did to inform students about the difference between the two databases. This link in the “Course Menu” is made available to students at the beginning of the semester.
- We encouraged and constantly reminded students to send us quiz questions that they do not understand how the answers were obtained. In return, we made the questions and corresponding answers available to the whole class. If no one sent us questions, we occasionally picked a some sample questions, sent email to students, and follow up with answers.

We expected these improvements to help student learn more effectively and perform better on exams compared to those of Spring 2006. Below is an additional hypothesis to be tested.
Hypothesis 3: Students will perform better in similar exams compared to Spring 2006 even when quiz questions and exam questions are from different databases.
Empirical Results for Hypothesis 3
The following data is collected for the t test of significant difference between the average scores of each exam suring the Summer 2006, Spring 2007 online, and Spring 2007 traditional and that of Spring 2006.
Results of t-test of significant difference between average scores

-Average score of exam one of Summer 2006 is significantly different than that of spring 2006 with 95% level of confidence.
 -Average score of exam two of Summer 2006 is significantly different than that of spring 2006 with 90% level of confidence.
 -Average score of exam three of Summer 2006 is significantly different than that of spring 2006 with 99.5% level of confidence.
 -Average score of exam one of Spring 2007 online is significantly different than that of spring 2006 with 99% level of confidence.
 -Average score of exam two of Spring 2007 online is significantly different than that of spring 2006 with 98% level of confidence.
 -Average score of exam three of Spring 2007 online is significantly different than that of spring 2006 with 99% level of confidence.
 -Average score of exam one of Spring 2007 traditional is significantly different than that of spring 2006 with 99% level of confidence.
 -Average score of exam two of Spring 2007 traditional is significantly different than that of spring 2006 with 90% level of confidence.
 -Average score of exam three of Spring 2007 traditional is significantly different than that of spring 2006 with 99% level of confidence.

Empirical results confirm that students consistently performed better in all exams of all classes.
Conclusions
The quiz tool of a course management system can help implement active learning theory for online learning effectiveness. However, one should be aware of the possibility of encouraging rote learning (based on memory) from students. If one invests time to set up a question database for quizzes similar to the question database for exams as discussed, students can not only perform better in exams but also learn more effectively.

Table 1: Data for all exams of various semesters

Table 2: T-statistics Compared to Spring 2006

Note: “dof” is the number of degree of freedom. The computation of t-statistics and tests for significant difference between two sample means are from Mode (Mode, Elmer B. 1961, p. 164).

References

Boettcher, Judith; Conrad, Rita (1999). Faculty Guide for Moving Teaching and Learning to the Web.
       League for Innovation in the Community College.
Bowell, C. and J. Eison, (1993). Active Learning: Creating Excitement in the Classroom. Workshop,
       Nov. 18, 1993, University of Nebraska-Lincoln.
Brightman, H. (2003). Teaching and Learning Workshop. February 28, 2003, College of Business,
       Middle Tennessee State University.
Cranton, P. Tranformative Learning in Action: Insights from Practice. San Francisco: Jossey-Bass, 1993.
Cross, K. Patricia, 1987. “Teaching for Learning.” AAHE Bulletin 39: 3-7.
Draves, A. William (2002). Teaching Online. Second Edition, Wisconsin 2002.
Ericksen, Standford C. The Essence of Good Teaching. San Francisco: Josey-Bass, 1984.
Good, T. L., Brophy, J. E. (1990). Educational Psychology: A realistic approach (4th edition).
       White Plains, NY: Longman.
Mergel, Brenda (1998), “Instructional Design and Learning Theory,” Educational Communications and Technology,
       University of Saskatchevan.
       (http://www.usask.ca/education/coursework/802papers/mergel/brenda.htm.)
Mezirow, J. (1990). Fostering Critical Reflection in Adulthood: A Guide to Transformative and Emancicitory Learning.
       San Francisco: Josey-Bass.
Mode, Elmer B. Elements of Statistics, 3rd edition. New Jersey: Prentice Hall, Inc., 1961.
Myers, C. and T. Jones (1993). Promoting Active Learning: Strategies for the College Classroom.
       San Francisco: Jossey-Bass.
Paloff, Rena M. and K. Pratt (1999). Building Learning Communities in Cyberspace. Jossey-Bass.
Piaget, Jean (1973). To understand is To Invent. New York: Grossman.