Evaluation is an ongoing process of asking questions, reflecting on the answers to these questions and implementing change based on the information received. Evaluation ­provides a systematic means of learning from experience. Organisations, like individuals, have a capacity for self-deception either through over- or undervaluing their achievements. The evaluation process introduces a dimension of objectivity.

Evaluation is critical at all levels of an organisation. At the macro level, it is necessary to ensure ongoing evaluation of the key policy and strategic directions. At the micro level, evaluation is more likely to be concerned with establishing whether or not a specific ­programme should be continued, modified or ceased. In the rehabilitation field, evaluation studies are frequently conducted to:

• contribute to decisions about the overall benefits of a programme – which clients tend to benefit from the programme and under what conditions?
  
• contribute to decisions about programme expansion, continuation, modification or ­termination – what works/does not work and why?
  
• obtain evidence to secure support for the programme – demonstrate effectiveness
  
• gain a better understanding of the processes (structural, financial, etc.) affecting the programme
  
• ensure that the programme remains responsive to the needs of the target group (to ­prevent ‘upmarket shift’ in service delivery, i.e. admits clients who were not intended for the programme)
  
• ensure programme fidelity – the programme continues to be provided as intended.

In effect, evaluation is concerned with determining whether or not a psychosocial ­rehabilitation intervention worked as it was supposed to work. Evaluation is strongly linked to quality assurance and generates information we can use to develop or sometimes even abandon programmes or interventions.

Approaches used in evaluating rehabilitation programmes

There are many approaches that can be used to evaluate rehabilitation interventions, each with different terms and language. Some of these employ experimental approaches using treatment and control groups while others use a quality improvement approach (such as collecting data on a single group and making improvements based on those data). It is important to discuss different approaches with service providers and other key players such as consumers prior to deciding on any one approach. It is clear that the approach required to evaluate a given programme will depend on a range of factors which must be considered by the evaluator or evaluation team at the time of evaluation. These are outlined in Table 18.1.

Having addressed the questions outlined in Table 18.1, the evaluator will be in a better position to select an approach to guide the evaluation. While a number of ­different approaches exist, most are based on the ‘structure’, ‘process’, ‘outcome’ model for assessment of quality of care proposed by Donabedian in his classic paper in 1966. Donabedian’s model is based on robust theory underpinning programme evaluation more generally. One approach closely aligned to that of Donabedian is the Context, Input, Process and Product (CIPP) Model developed by Stufflebeam (1971).

Table 18.1 Issues to consider prior to commencing an evaluation.

The Context, Input, Process and Product Model

The CIPP Model uses a systems approach to evaluation. A basic system requires input, process and output. Stufflebeam added ‘context’ and relabelled ‘outcome’ with the term ‘product’ to achieve the Context, Input, Process and Product (CIPP) approach to evaluation. The basic components of the model are described in Table 18.2.

Context evaluation

This component of the evaluation process examines the context of care provision. The focus here is the goals of service/programme provision. Are these aligned to policy and appropriate for the service? Data collected under this component of the evaluation could include:

• the goals and eligibility criteria for admission to the programme
  
• characteristics of clients admitted (age, gender, background, level of disability, etc.). Do these meet the eligibility criteria?

In effect, context evaluation explores the question: Are the clients being admitted to the rehabilitation programme the clients whom the program was established to treat?

Input evaluation

Input evaluation requires an examination of the services and interventions provided. It focuses on whether the ‘inputs’ (e.g. staffing levels, staff qualifications, costs, etc.) are appropriate to meet the objectives and goals of the service (described in the previous section on context evaluation). Data collected under this component of the evaluation could include:

• staffing levels, staffing mix, qualifications and experience
  
• details of interventions planned for clients
  
• overall cost of providing service.

The key question to be answered here is: Are the interventions provided capable of meeting the needs of the clients in the programme?

Table 18.2 The CIPP approach to evaluation.

Process evaluation

This component explores the extent to which planned interventions are actually ­implemented. In other words, it provides an examination of the interventions provided by different staff groups/members over a given period (e.g. 1 week). The issue here is programme fidelity (whether the programme was delivered in accordance with the ­programme manual, assuming there is some kind of manual). If the programme was delivered differently, the evaluation seeks to determine the reasons for the variation and what effect, if any, the variation had on the success of the programme. Data collected under this component of the evaluation could include:

• description of the activities provided, work practices, programme fidelity
  
• drop-out rates (attrition) are considered and the evaluation determines whether this rate is typical for the type of programme and target population. Other indicators of engagement are the average number of programme sessions completed and the proportion of programme tasks completed by participants
  
• how participants experienced the programme. This may include rating of enjoyment, usefulness and relevance, and impact on self-efficacy (capacity to do the things taught in the programme).

The key question to be answered here is: Are the interventions provided in keeping with those that were planned?

Product (outcome) evaluation

This is the final component of the evaluation process and examines whether the ­programme achieved the stated goals. For example, a healthy eating programme may aim to change eating habits, so that participants eat more fresh fruit and vegetables, less fat and less salt. Outcome is different from impact or process. A programme participant might find a ­programme enjoyable and might learn about the food pyramid (process) without changing actual eating behaviour (outcome).

While most rehabilitation programmes have a focus on client outcomes, staff outcomes (such as stress, job satisfaction, perceptions of achievements, team functioning, skills, etc.) can also be explored.

Assessment here is likely to focus on:

• outcomes for clients (which could include length of stay, changes in functioning, ­reduction in risk, reduction in substance misuse, etc.). In effect, this component of the evaluation will be concerned with determining whether participants acquired new knowledge or developed new skills during the course of the programme and if this has resulted in improved functioning
  
• outcomes for staff (which could include satisfaction with work, perceptions of team functioning, perceptions of individual performance and achievements, stress, burnout, etc.).

The key question asked here is: How well has the programme achieved its goals?

It is clear that an evaluation may focus on all or some of the components outlined. For example, a rehabilitation team may be more interested in the outcomes of a given programme (rather than the other components) and the evaluator will need to clarify this with the rehabilitation team prior to data collection. Indeed, it is recommended that the rehabilitation team provide terms of reference (ToR) for the evaluation or a written proposal detailing the questions to be addressed. This will ensure that the ‘deliverables’ from the evaluation will be as expected and address the questions posed by the rehabilitation team.

Evaluation methods

Two broad approaches to evaluation are commonly used: qualitative methods and quan­titative methods. Typically evaluations use mixed methods (both qualitative and quantitative).

Qualitative methods may include observation of programme implementation, review of programme documents and interviews with programme participants. Qualitative methods are systematic and are designed to extract themes or make judgements about matters such as programme fidelity. Quantitative methods use measurements to assign numerical values to programme impact and outcome. These measurements may take the form of satisfaction surveys, standardised outcome scales (see Section 1 of this book) or measures such as the number of days a person has worked or earnings (if, for example, a programme outcome was participant employment).

A worked example

To illustrate the assessment process we will consider how we might evaluate Sam’s ­substance misuse programme as it is described in Chapter 12.

This is an eight-session group programme. The brief manual set out in Chapter 12 is typical of the kind of manual you might expect to find in a practice setting. It is much less developed than a manual for a research programme: it simply sets out in summary form a week-by-week schedule of activities and tasks. The programme does include some ‘built-in’ evaluation processes, which will provide a useful starting point, but these will not be ­sufficient for the more rigorous evaluation we will develop here.

A programme evaluation would not just be done on one person, although evaluating Sam’s responses to it may provide indicative data on the potential issues that will arise when more people undertake a similar programme. The same logic can be applied to an individual or a group. However, the worked example below assumes that Sam is one of several clients who have undertaken the programme.

Developing a programme logic

A programme logic is a logical statement that sets out what a programme aims to achieve and identifies the resources and processes necessary to achieve the specified outcomes. It may be represented in the form of a diagram or a flow chart or it may be simply a ­statement or series of statements. When a programme is simply outlined as in this ­example, the starting point is to construct the programme logic.

The first step is to identify expected programme outcomes. In this case, the programme outcomes can be inferred from the title and content of the programme and from the kind of people who are referred into the programme. It is clear that the main purpose of the programme is to reduce substance misuse. It would therefore be expected that programme completers would show a reduction in substance misuse.

The second step is to identify programme inputs – the resources necessary to achieve the outcomes. In this case the inputs are one or more group facilitators, a suitable space for a group to meet, an activity schedule and various other resources such as printed exercises, questionnaires, information and resource handouts and completion certificates.

The third step is to set out the theory that explains how participation in the programme leads to the designated outcome. In this case, it is clear that the underlying programme logic is that people will reduce substance misuse when they:

• can manage cravings
  
• better understand the link between self-esteem and substance misuse
  
• can use cognitive strategies to overcome low self-esteem
  
• understand the relationship between stress and substance misuse
  
• can use problem-solving strategies and relaxation to deal with stresses and difficulties
  
• understand the relationship between social pressure and substance misuse
  
• can use assertiveness and other social skills to resist social pressure to use substances
  
• understand the importance of being able to substitute other enjoyable activities for ­substance misuse
  
• have identified and tried out enjoyable activities that are a safe alternative to substance misuse.

There are various psychological theories that sit behind this logic but they are not relevant to programme evaluation. However, programme evaluation may well be interested in the extent to which participants achieve the knowledge and/or skill specified in each of the bullet points above because the programme logic suggests that it is through acquisition of this knowledge and skill that the ultimate outcome of reduction in substance misuse is achieved.

Evaluating implementation of Sam’s substance misuse programme

• A thorough evaluation would require the evaluator to observe sessions or review ­videotapes of sessions to determine whether or not the programme was implemented as described in the manual.
  
• In some cases, psychosocial rehabilitation programmes have established fidelity scales that provide a checklist approach to recording evaluation of fidelity. For an example, see Teague et al. (1998). However, most programmes will not have a fidelity scale, and the evaluator will have to develop a simple tool that enables fidelity to be assessed. Checklists can be used to structure evaluations of videotapes, or they can be used by practitioners to confirm that they included specific elements in the session and ­implemented them correctly. The latter use of the scales relies on the practitioner’s grasp of the critical elements in the programme, and their ability to observe their own performance accurately.
  
• A less rigorous evaluation bases determination of fidelity on a structured interview with one or more programme facilitators plus review of any records or chart entries maintained in relation to the programme. The evaluator would note variations or omissions and explore reasons for these. The evaluator would also seek advice from the facilitator as to the expected impact of variations or omissions.

Evaluating engagement in Sam’s substance misuse programme

The evaluator will want to know:

• how easy it was to recruit participants into the programme and the pattern of ­attendance. In particular, the evaluator will want to know the numbers who agreed to participate and completed the programme, the mean number of sessions attended by each participant and the average number of participants in attendance at each session. Where possible, this information should be benchmarked against participation patterns for similar programmes
  
• the amount and quality of engagement and participation in programme activities, ­including exercises completed during sessions and homework activities between sessions
  
• (in a more thorough evaluation) reasons for dropping out of the programme or for ­missing sessions, obtained via interviews with participants. These interviews can also ­explore programme impact (see below).

Evaluating the impact/process of the programme on participants

Impact evaluation for a programme such as this is moderately complex.

• In part, it will mean finding out from participants how they experienced the programme. This will often make use of a standard satisfaction survey (see Chapter 5) but may also make use of more open-ended information obtained by means of a semi-structured interview.
  
• A more in-depth evaluation will explore the experience of each of the major ­components rather than comprising a global evaluation only. This level of evaluation is especially useful when planning changes to a programme.
  
• A thorough impact evaluation also requires a determination of the extent to which the programme improved participants’ knowledge, self-efficacy and skills. The programme logic provides the framework for identifying the relevant variables. When we developed the programme logic for Sam’s programme, we identified nine units of knowledge and skill that a participant might expect to develop as a result of participation in the programme. Ideally, an evaluation will use before-and-after measures to evaluate changes in knowledge and skill in each specified area.

Evaluating programme outcomes

This is the single most important part of any programme evaluation. The purpose of the programme is to achieve outcomes. No matter how good the implementation, programme participation or programme impact, if it does not achieve specified outcomes, it is failing. In this case, Sam and other participants enrolled in the programme to reduce the level of substance misuse. This means that the key outcome evaluation questions will concern changes in severity and frequency of substance misuse over the course of the programme and the extent to which these changes are sustained beyond programme participation. This means that it is important to obtain reliable data about substance misuse at baseline (prior to commencing the programme) and programme completions and at follow-up (3 and 6 months may be realistic). Chapter 2 contains examples of measures that can be used for this purpose.

When interpreting outcome, it is important to have reasonable reference points. Not all participants in a substance misuse programme can be expected to reduce their substance misuse. Reference to relevant outcome literature assists in determining reasonable and expected outcomes. Outcomes in real-world settings are typically somewhat weaker than those in research settings; while we strive for the best outcomes, some allowance needs to be made for the lower resources available to conduct the programme and the greater complexity in client problems that are often experienced in applications to routine practice.

Benchmarking rehabilitation services

The benchmarking of performance indicators is another form of programme evaluation. Although benchmarking has its origins in industry, the process is gaining currency in the mental health field as a means of improving service provision (Meehan et al., 2007). Bullivant (1994) defined benchmarking as an activity concerned with the ‘systematic process of searching for and implementing a standard of best practice within an individual service or similar groups of services’. Thus, benchmarking could occur within a ­single organisation with similar units (internal benchmarking) or between organisations with a similar focus (i.e. rehabilitation of individuals with severe disability) known as ‘collaborative’ benchmarking. Regardless of scope, having identified high-performing organisations, the task is to identify and emulate the clinical/administrative practices that lead to superior performance (Berg et al., 2005).

In practice, participating organisations agree to share information about their performance on a number of key domains such as efficiency, effectiveness and safety. The collection and reporting of performance data have been promoted as a means of improving service quality through increased accountability and transparency (Hermann & Provost, 2003). Performance data enable service providers, service users and funding bodies to monitor the performance of a given organisation relative to its peers on selected parameters. This motivates organisations to achieve higher performance and to strive for ­service provision that is of an acceptable standard (Shepherd et al., 2010).

The collection, reporting and investigation of benchmarking data follow a recognised procedure (Box 18.1). In the initial stage, benchmarking partners identify what indicators they wish to compare. Data are then collected on these indicators using the same data ­collection template to ensure consistency in the data collected. Services are then compared and activities leading to higher performance are investigated. These practices are then implemented in all services and the process is repeated.