Program Assessment: Ways to Make it Meaningful While Waiting For NCATE

Calvin F. Meyer
Marshall University

Introduction
     The challenge for Teacher Education programs is to prove their viability in a skeptical society. The key to surviving is finding common ground that insures collaborative efforts of assessments. This article seeks to demonstrate such ideas.
     The development and use of assessment processes across a university and by accreditation agencies offer many challenges. This article outlines annual and five year processes and compares these processes to the seven-year process of NCATE. Faculty members developed the process to include many variables and inputs that are key to proving program viability and productivity. The end result is one that speaks to society’s demand for accountability.
     To accomplish this an environment of assessment is critical. Such an environment must offer a viable approach of insuring all internal and external factors are addressed. The commentaries of employers, businesses and other community influences suggest they are stakeholders of the outcomes of effective teacher education programs. They depend on our success. While the demands on Teacher Educators is more and more burdened with tasks and responsibilities that often keep us from our first love of teaching, the need to streamline and insure our efforts are making an impact is critical to our survival. Assessment processes mandate lines of communication across constituencies. These constituencies enhance and support our efforts to prove Teacher Education is critical to .producing effective master teachers. But the key is insuring such a process is streamlined, on-going, all inclusive systematic, and done in a timely manner.
An Environment of Assessment
     APUS (2008) suggests that assessment today focuses on three major questions: “1). Have we made a difference?  2). How will we know?  and 3). How can we use assessment results for improvement?”(p. 1). This suggests that Teacher Education Programs, just like the counterparts in K-12 school, must be outcomes based, a method of evaluating the effectiveness of a program. Therefore the tools of assessment must demonstrate varying ways that the students have or have not achieved the intended outcomes of the program. Since this method of assessment is still a relatively new educational tool, and thus an inexact one, multiple means of assessment must be employed in which the weaknesses of any one means are offset by the advantages of others (Kenrick School of Theology, 2008).
To that end, universities have felt compelled to exercise layers of assessment beyond the national accreditation agencies that accredit both the university and the units. This necessitates an environment of assessment from the President’s Office to the faculty member’s office. Linn (2000) suggest that Educational Assessment is a barometer and lever of reform. As a barometer, assessments are expected to provide information on quality. It is also expected to serve as a mechanism of reforming educational practice. While this is true of K-12 schools, universities, are increasingly being held to the same accountability, thus they are seeking to find their way in designing assessment processes to accomplish this. The difficulty arises when higher education faculty fail to see this as an inherently tied to their role but rather that of the administration. This is most profoundly obvious with tenured faculty. The main thrust of this move to layers of assessment is increasingly becoming the mandate of university Board of Trustees, Governors or Regents. Additionally, colleges whose survival is dependent on national accreditation agencies are finding the requirements of internal assessments do not fit well with the external accreditations expectations. These inconsistencies create further barriers in developing a positive environment of assessment.
The university must find a way to articulate the mandate for assessment, the need for involvement by all and be open to modifications that better transcends the requirements of each layer. Faculty will be more prone to participate in an atmosphere that realistically serves the purpose of improving the program and aids in securing accreditation. At Marshall University an attempt has been made to insure voice and a pragmatic way to approach each layer. The university is seeking to develop a mindset that assessment is a building process to meet the most important needs for improvement as well as lead to accreditation, not just isolated sets of data that serve only to assure the approval of the Board of Governors.
At Marshall, the process is sequential and representative of each college. First, the university employed a full-time Director of Assessment. Second, a university wide Assessment Committee, represented by each college, the Faculty Senate and Graduate Council functions as the overseer of the process. Third, each assessment is sequenced with a rotation of review as needed. Fourth, the Assessment Committee is assigned the undergraduate programs for review and the Graduate Council the graduate programs. Each program is reviewed by three faculty members, insuring appropriate peer review. The peer review necessitates a close relationship between the reviewing team and the dean to insure mandated modifications are met prior to being sent to the Board of Governors. With that collaboration, it is affirmed the process is voiced by all the stakeholders. Finally, and most importantly, each college is allowed to add to the assessment process to insure transition and usability of all the layers. To that end, this process assumes that the various assessments build on one another, adds to the depth of each level of assessment and ultimately meets the intended purpose of the decision-makers upon which the survival of each program and college rests. Blanchard (2008) graphically demonstrates that assessment must start with the course, lead to the program, then to the college and ultimately the university. If we interpret this correctly, then assessment is a process that insures the course fits within the context of the program goals, the program goals form a conceptual framework for the college and the college fulfills the mission of the university.
Differences of Expectation
     In many universities, the layers of assessment include a Syllabus Review, an Annual Assessment Process, A Five-Year Assessment Process, and National Accreditation. For the purposes of this article, an analysis will be made of each assessment process in terms of expectation, using the NCATE SPA expectation as the National Accreditation process. The following chart graphically illustrates the differences of expectation:
 See Comparative Analysis of the Itemization of University Assessment Processes
Though some liberties were taken in interpreting the language of the items, the comparison of guidelines reveal that in some form the item language, where measured consistently, could be found in these processes. It must be observed that two most consistent processes are the NCATE SPA and the Five Year Review. In making this observation, one realizes NCATE is a seven year review and the Five Year Review, by its very name, obviously occurs every five years. It could thus be assumed that the Five Year Review could by its very nature serve as the foundation of the NCATE review. However with NCATE requesting the most recent three years of data, it is clear that the limited amount of information required in the Annual Reports and Syllabus Reviews, done each term, serve little purpose in providing a continuous and on-going data collection process for national accreditation. If Annual Reports and Syllabi Reviews are to serve a more significant purpose in leading to national accreditation, some modification is needed.
Finding Common Ground
The Graduate College of Education and Professional Development of Marshall University decided there was a need to utilize the Annual Report and Syllabi Review to annually serve as a data collection means to support the Five Year report and the NCATE  SPA process. With NCATE SPAs requiring three years of data collection, meaning this is an on-going, mandates the necessity of using your other assessment systems to support this process.
The thrust of this move is to insure a sustainable process which leads to the most important outcomes or purposes of assessment, improvement of program and accreditation. “Integration is a key problem for decision making that aims to foster progress towards sustainability. We must redefine the driving concept of conventional categories and focus on those items which lead to appropriate decision making” (Gibson, 2008, p. 60). To that end, the Annual Report should sustain the Five Years Review and NCATE SPA review but as Gibson suggested we must go beyond the limited categories of a typical Annual Review. Given this, a Modified Annual Report might include a process that follows this general outline:

1. Assessment Activities
1.  Program Goals
2. Data Collection
2. Feedback Loop (Generally the only requirement of the Annual Report)
3. Strategic Plan

Listed  is a chart illustrating a Modified Annual Report which includes all identifies the elements of the report and accounts for the common elements of Five Year Report and NCATE SPA report: 
Modification of the Annual Report to the Support NCATE  SPA and Five Year Assessment Process

One might surmise that this Modified Annual Report is simply incorporating all the Five Year requirements into the Annual Report. To some degree that is true but not completely. Each of the elements of the modified report are items that need to be collected annually regardless of when they are formally required by the university. Using this as a template for each annual report, the author of the report will find a developed process that collects the data in a timely manner thus providing the data base for the larger reports. The difficulty comes when the data process is not analyzed except when required for the major reports and thus the often stressful rigor of attempting to collect it in an abbreviated manner of time.
Final Thoughts
As the ole saying goes “Work smarter, not harder.” I found in my personal life collecting the information for my taxes is much easier if I do so systematically and continuously throughout the year rather than wait until April 1 to initiate the process. The same approach applies here. Additionally, we have personnel in place who collect the information needed on their computer data base. Knowing what we need from year to year becomes routine and thus when the data is needed it is often just a click away. I believe all concur “organization assessments are used for learning and improvement. Much like an annual health physical, this tool helps managers better understand what is working well and on what they should be focusing their improvement efforts. Assessments also are effective at helping communicate priorities with various stakeholders and with monitoring progress over time. But the real value of an assessment is its ability to improve performance – to improve outcomes and drive tangible organizational results” (Minnesota Council of Quality, 2008, p. 1). The key to a successful assessment process is doing it in a timely and sequential manner.

References

APUS Learning Outcomes Assessment (2008) Retrieved August 19, 2008, from:
http://www.apus.edu/Learning-Outcomes-Assessment/Resources/Degree-Program-Assessment_Reports/Program-Assessment-Reports_Overview.htm

Blanchard, K. (2008) Layers of assessment. Retrieved August 21, 2008 from:
https://www.mnsu.edu/humanres/profdev/handouts/databalnchard.pdf

Gibson, R. (2008) Integration through sustainable assessment. International Experience and Perspective. Retrieved August 25, 2008 from:
http://www.iaia.org/non_members/conference/SEA%20Prague/p%2059-82%20Stream%20C.pdf

Kenrick School of Theology (2008) Retrieved August 19, 2008, from
http://www.kenrick.edu/Assessment/orddivasses.html

Linn, R., Gronlund, N. (2000) Measurement and assessment in teaching. (8th ed.) Merrill/Prentice Hall, Saddle River, NJ.

Marshall University (August, 2007) Course syllabus review. Marshall University, Huntington, WV

Marshall University (February, 2008) Guidelines for undergraduate/graduate program assessment yearly report. Marshall University, Huntington, WW

Marshall University (April, 2008) Program review. Marshall University, Huntington, WV

Minnesota Council for Quality (2008). Retrieved August 27, 2008 from:
http://www.councilforquality.org/assess_org_benefits.cfm

Program Report for the Initial Preparation of Middle Level Educators (February, 2008)National Middle School Association, Reston, Ohio.

Comparative Analysis of the Itemization of University Assessment Processes
Key:  X = Item Required