Implementing Process Evaluation in a
School Violence Prevention Grant

Praphul Joshi
Susan Lyman
Charles Duncan
Lisa LeBlanc
Phil Caillouet
University of Louisiana at Lafayette

Abstract
     Many schools have implemented programs geared towards preventing school violence and promoting safety. Without systematic evaluation efforts, results of implementation and program outcomes in these programs are limited. Although several studies deal with evaluation results in the area of health promotion in general, very few document the protocols for conducting evaluation specifically in the field of school violence prevention initiatives.  The purpose of this paper was to provide a tool for implementing process evaluations for those involved in program planning, delivery, and evaluation. Steps that were used to implement process evaluation of a school violence prevention program in schools through a Safe Schools/Healthy Students grant initiative are discussed.  Evaluation of that school violence prevention program was implemented and continued throughout the life of the grant. Several strengths and challenges involved in evaluating the program and its components are discussed in the paper.
Background
     During the past two decades, the significance of school violence and safety has been studied in detail, and many programs have been implemented throughout the nation to address this issue. Violence prevention programs directed at reducing both suicide and fighting have value and relevance to school age youths (CDC., 2004). Similarly, strategies determined effective in reducing youth problem behaviors (e.g., skill and competence-building programs, positive youth development, and parent training) have the potential to reduce underlying risks and provide the skills and support that students need to avoid fighting and suicidal behavior (Trezza & Popp., 2000, Plutchik., 1995). However, many of these programs have failed to achieve their goals, in part, due to lack of planning and coordination with the people who implement, design, and evaluate them. To reduce failure, the efficacy of process evaluation has been examined in several health promotion programs and has been found to be very useful.
     Process evaluation is used to monitor and document program implementation. Its purpose is to aid in understanding the relationship between specific program elements and program outcomes. Process evaluation is used to describe how the program is implemented, what services are provided to whom and how they are provided. Process evaluation is used as feedback to assure that the program is being implemented as planned and to make any changes to improve the program (Gomby and Larson., 1992). The scope and implementation of process evaluation has grown in complexity as its importance and utility has become more widely recognized (Saunders, et al., 2005). Appropriate evaluation of a program encourages facilitators, coordinators, and the project director of the program to clarify their understanding of the goals. With a clearer understanding of the program goals those key individuals can function more cohesively as a team, focused on a common end (Schorr LB., 1997). Contrary to the outcome, a program’s lack of success can be attributed to any number of program-related reasons, including poor program design, poor or incomplete program implementation, or failure to reach sufficient numbers of the target audience (Steckler & Linnan., 2002).
     The purpose of this paper is to provide tools for implementing process evaluations in the field of school violence prevention programs and curricula in particular and any other aspect of health promotion in general. The methods in process evaluation that are being implemented in the school violence prevention program in the school system, through a Safe Schools/Healthy Students grant will be presented.
Background of LPSS*2 Grant
     The Lafayette Parish School System has recently completed the 3 years of funding for implementation of a several-pronged initiative for reducing violence and promoting safe environments in its schools through a grant awarded by the U.S. Departments of Education, Health and Human Services, and Justice. The grant initiative, which was referred to as Lasting Partnerships in School Safety in Lafayette Parish School System (LPSS*2), provided students, schools, and communities with a coordinated and comprehensive plan of activities, programs, and services that focus on promoting healthy childhood development, preventing violence and alcohol and other drug abuse.
     An evaluation plan was developed based on the seven required goals of the grant.  These required goals focus on behavior, policy, and environmental factors to provide a comprehensive approach to preventing school violence. Data sources for evaluation based on grant goals are indicated in Figure 1.
     The evaluation plan for this initiative was based on the six steps of process evaluation from Steckler and Linnan (2002) as illustrated in Figure 2. Discussion of these steps provides an overview of program evolution, and formulation of the evaluation plan.
Step 1. Describe the program.
     In large projects it is possible for independent aspects of the program to be overshadowed by the complexity of the program.  Expected outcomes were conveyed in a logic model that specified the theoretical constructs of interest, those expected to change, and mediators of the change process ((Steckler & Linnan., 2002; Scheirer et al., 1995).   A logic model can improve and provide focus for program direction. It also provides a frame of reference for one or more evaluations of the program.
     The LPSS*2 initiative provides students, schools, and communities with a coordinated and comprehensive plan of activities, programs, and services that focus on promoting healthy childhood development, preventing violence, and alcohol and other drug abuse. The theoretical framework of the program is based largely on Bandura’s Social Cognitive Theory (SCT) (Bandura., 1996) and PRECEDE-PROCEED (Green & Kreuter., 1992).  The primary constructs of SCT on which the LPSS*2 initiative focuses are self-efficacy, behavioral capability, situation, observational learning, knowledge, and attitudes.  The primary goals of the LPSS*2 initiative are to increase adolescents’ self-efficacy for preventing physical fights, abstaining from alcohol and tobacco consumption, and to influence the social norms within the school environment so that students and teachers can feel safe and secure. The LPSS*2 logic model is provided in Table 1.
Step 2. Describe complete and acceptable delivery of the program
     Elements that comprised the program in process evaluation planning; which include specific strategies, activities, media products and staff behaviors are described in more detail in Step 2 (Scheirer et al., 1995). The description of complete and acceptable delivery of the program is based on the details of the program (e.g., program components, theory, elements in logic model) and guided by an external framework, such as the recommended elements of a process evaluation plan. The ideally implemented LPSS*2 initiative had seven domain goals as indicated in Figure 1. Examples of strategies and curricula used are also indicated in the figure.
Step 3. Develop a list of potential evaluation questions
     In Step 3, the initial wish list of possible process evaluation questions, based on the program, and without full consideration of resources needed, was drafted. Through a series of meetings and discussions, the evaluation team developed a list of potential process evaluation questions. The following are examples of all-purpose (general) questions that might be used to evaluate other health promotion programs are listed, followed by some examples of the specific evaluation questions that were used with the LPSS*2 grant. Questions were categorized using the heading of fidelity, dose delivered, dose received, reach, recruitment, and context, based on the elements of process evaluation (Stufflebeam, et al., 1971; Steckler & Linnan., 2002).
Fidelity: (extent to which the intervention was implemented as planned)
General question: To what extent was each of the program elements implemented as planned?
Specific LPSS*2 questions:

• Did Second Step facilitators follow the standardized protocol for curriculum implementation?
• Did the facilitators follow standardized protocols for data collection during the implementation of Protecting You/Protecting Me curriculum?

Dose delivered: (extent to which components were received and valued by the facilitators)
General question: Were all intervention components delivered?
Specific LPSS*2 question:
Within the lessons of Second Step involving role play, what percentages of the components were actually implemented?

• Did Task Force meetings meet your expectations; why or why not?
• Dose received: (extent to which components were received and valued after program delivery)
• General: To what extent did the recipients of a program participate in it?
• Specific LPSS*2 questions:
• How many community partners participate in the Task Force Committee meetings?
• Did you enjoy the activities in Second Step curriculum?
• Reach: (achieving the goals and objectives it was intended to do)
• General: Did the program do what it was intended to do?
• Specific LPSS*2 question:
• Did Task Force Committee meetings recruit more partners to collaborate with implementing school violence prevention programs?
• Did student’s character improve after experiencing Principle Woods curriculum?

Recruitment: (obtaining staff and facilitators to carry out objectives of program)
General: What procedures were incorporated to recruit participants?
 Specific LPSS*2 questions:

• What procedures were followed to develop the Task Force Committee?
• What procedures were used to recruit community participants in the Communities Mobilizing for Change on Alcohol (CMCA) program?

Context: (organizational, situational, and other field issues that affected implementation)
General: What are the barriers or challenges that existed during the program?
Specific LPSS*2 question:

• What were the factors that existed concerning participation in the Leadership and Resiliency Program?
• What barriers or issues had a bearing on implementing law enforcement presence in your schools?

Step 4. Determine methods for process evaluation
     In Step 4, the evaluation team considered the methods that were used to answer each of the process evaluation questions. Resources are an essential aspect of effective process evaluation and were evaluated for impact. Considering the multiple facets of the LPSS*2 initiative, the data collection process was classified into predisposing, enabling and reinforcing factors. These factors are based on the PRECEDE PROCEDE model; PRECEDE: Predisposing, Reinforcing, and Enabling Constructs in Educational Diagnosis and Evaluation, and PROCEED: Policy, Regulatory, and Organizational Constructs in Educational and Environmental Development (Green & Kreuter., 1992). Predisposing factors are tools used to assess or influence knowledge, provide education, materials, and instruction. Enabling factors are the environmental supports to accomplish the targeted goals; availability and accessibility of resources. Reinforcing factors refer to assessments of incentives and potential benefits that can support and strengthen a particular project. The following are specific examples of data sources for process evaluation categorized with the same headings in Step 3 of fidelity, dose delivered, dose received, reach, recruitment, and context based on the elements of process evaluation (Stufflebeam, et al., 1971; Steckler & Linnan., 2002).
Fidelity:

• Monthly reports from facilitators implementing Second Step curriculum
• Community Mobilizing for Change on Alcohol (CMCA) monthly meetings agendas and minutes

Dose delivered:

• Completion Log Reports from facilitators that implemented the Second Step curriculum
•  Protecting You/Protecting Me facilitator completed online trainings

Dose received:

• Surveys completed by the school principals regarding law enforcement in schools
• Reports from administrators on environment changes made for school safety

Reach:

• Collaboration surveys completed by Task Force Committee members
• Completion Logs completed by Second Step facilitators

Recruitment:

• Community outreach to businesses for CMCA involvement
• Newspaper advertisements for nurses

Context:

• Open-ended questions related to barriers in implementing law enforcement in the schools, future considerations regarding improving school safety
• Task Force Committee networking to determine barriers and gaps in services

Step 5.  Consider program resources, characteristics, and context
     In Step 5, the team considered the resources needed to answer the potential process evaluation questions listed in Step 3 using the methods proposed in Step 4.  The LPSS*2 initiative faced several critical challenges and barriers for implementing many of its components in the first year. Effective communications among the partners was a critical aspect to address the challenges that occurred. Hurricanes Katrina and Rita created problems for individual participants, schools and the community, that necessitated several changes to the original list of programs for final implementation.
Step 6. Finalize the evaluation plan
     Upon assessing the feasibility of the programs, evaluators developed a final list of process evaluation elements to be used for implementation. The plan addressed all seven grant goals and it incorporated variables from all domains of process evaluation.
Discussion
     Initiatives such as LPSS*2 strive to address a wide variety of issues as they relate to planning and implementing school violence and safety measures. Considering the array of issues focused on in this initiative, as required in the seven grant goals, the evaluation efforts were challenging. A formative evaluation process was designed as per the grant goals and enabled the stakeholders to understand the relationships of the many facets of the program. Process evaluation is designed to measure the extent to which the programs are implemented and the degree of involvement of students, facilitators, and teachers in those programs. Incorporating a well designed process evaluation framework can facilitate accountability by providing quality control for the programs. Bi-annual reports were prepared for the grant period, which represented the results of process evaluations for all components of the initiative. These reports are sent to the U.S. Department of Health and Human Services to monitor grant progress. The evaluators met with the stakeholders on a monthly basis during the Task Force Committee meeting and discussed the evaluation results and the progress of the programs. These meetings also enabled the communication between those persons implementing the programs and curricula and the evaluation staff. To aid in the quality control measures Project Advisory Committee meetings were held quarterly to further discuss the goals, objectives, and the evaluation findings.
     To understand the extent of communication and work involvement with several partners in the initiative, a Collaboration survey was conducted in April 2006 and again in 2007 for key partners on the Task Force Committee involved in this initiative. The surveys provided feedback on whether an organization partnered with others, the perceived level of partnership, and any strengths or barriers associated with those partnerships. The results of the survey indicated a growth in the level of partnerships from year 1, and that strengthening of the partnerships should sustain the programs beyond grant funding.
     Many studies published in journals are data driven, but discussion seldom focuses on the plan for process evaluation. Few studies include an explanation of the implementation of evaluation strategies and its planning process. It is hoped that through this paper the essential steps that are outlined in process evaluation can provide guidance for program planners and evaluators who are involved in large multifaceted programs such as LPSS*2.
     Perhaps Stufflebeam et. al. (1971), stated the purpose of program evaluation best when they extolled us to remember “the purpose of evaluation is to improve, not to prove.”

References
Bandura, A. Social Foundations of Thought and Action: A Social Cognitive Theory. Englewood Cliffs: Prentice Hall. 1986
Centers for Disease Control and Prevention. (2004). Suicide Attempts and Physical Fighting Among High School Students
      - United States, 2001. MMWR 2004; 53(22):474-476
Green, L.W., Kreuter, M. W. (1992). CDC's Planned Approach to Community Health as an application of PRECEDE and
      an inspiration for PROCEED. Journal of Health Education 23: 140-147.
Plutchik R. Outward and inward directed aggressiveness: the interaction between violence and suicidality.
      Pharmacopsychiatry. 1995; 28(suppl 2):47-57.
Saunders RP, Evans MH, and Joshi P. Developing A Process Evaluation Plan for Assessing Health Promotion Program
      Implementation: A “How-To” Guide. Health Promotion Practice, 2005; 6: 134-147.
Scheirer, M. A., Shediac, M. C., & Cassady, C. E.  Measuring the implementation of health promotion programs - the
      case of the breast and cervical-cancer program in Maryland. Health Education Research, 1995; 10(1): 11-25.
Schorr LB. Common purpose: strengthening families and neighborhoods to rebuild America. New York, NY: Doubleday,
      Anchor Books, 1997.
Steckler, A. & Linnan, L. In: Steckler, A. & Linnan, L. (Eds.), Process evaluation for public health interventions and
      research. San Francisco: Jossey-Bass, 2002; pp. 1-24.
Stufflebeam, D.L., Foley, W.J., Gephart, W.J., Guba, E.G., Hammond, R.L., Merriman, H.O. and Provus, M.M. Educationa
      l evaluation and decision making. Itasca, Illinois: F.E.Peacock, 1971.
Trezza GR, and Popp SM. The substance user at risk of harm to self or others: assessment and treatment issues.
      Journal of Clinical Psychology 2000; 56: 193--205.

Table 1: Logic Model for LPSS*2 initiative 

Inputs	Immediate impacts	Short-term impacts	Behavioral impacts	Health outcomes
Providing training, materials, policies, consultation, to school administrators and community partners will	Result in the development of an LPSS task force and cross-disciplinary implementation of the LPSS initiative which will	Result in changes in the school environment, social norms, development of students’ self efficacy and skills, incorporating personnel for promoting safety and wellness, community mobilizing, which will	Result in reduced rate of school violence, and ultimately,	Improved health in students and better academic growth.

Figure 1. Data sources for evaluation of lasting partnerships for safe schools initiative, based on grant goals.

Figure 2.  Steps in the process evaluation process (adapted from Saunders et al, 2005, Steckler & Linnan, 2002