Organizational interventions: A research-based framework for the evaluation of both process and effects

Organizational interventions: A research-based framework for the evaluation of both process and effects

Karina Nielsena* and Johan Simonsen Abildgaardb

aNorwich Business School, University of East Anglia, UK and National Research Centre for the Working Environment, Copenhagen, Denmark; bDepartment of Psychology, University of

Copenhagen, Denmark

(Received 4 June 2012; final version received 8 October 2012)

Organizational interventions are often recommended when organizations want to improve

employee psychological health and well-being. Research, however, has revealed inconsistent

results and reviewers have called for research on why interventions either bring about desired

change or fail to do so. Answering the ‘‘how’’ and ‘‘why’’ of intervention outcomes requires a

close examination of the elements that hinder or facilitate desired outcomes, thus moving

beyond evaluation of only the overall effects. In this paper, we present an evaluation

framework based on recent intervention research and process-oriented organization theory.

The framework offers suggestions for which elements to include when evaluating organiza-

tional interventions. Within the framework, elements crucial to intervention evaluation are

grouped into four overarching categories that we argue are crucial to evaluation over the

five phases of an intervention programme. These categories are: the organizational ‘‘actors’’;

the mental models of those actors; the context of the intervention; and intervention design and

process. Evaluation during the process as well as of the overall effects, as recommended by this

framework, should throw light on what works for whom, why, how and under which

circumstances.

Keywords: process evaluation; organizational interventions; effect evaluation; evaluation framework

Introduction

In recent years there has been an increasing interest in the use of organizational

interventions when aiming to improve employee psychological health and well-being,

and they have been widely recommended (ETUC, 2004; EU-OSHA, 2010; ILO,

2001). Organizational interventions can be defined as planned, behavioural, theory-

based actions that aim to improve employee health and well-being (e.g. Nielsen,

Randall, Holten, & Rial González, 2010c). While a design with a simple pre-and

post-measurement design with randomized controls has been considered the ‘‘gold

standard’’ for evaluating organizational interventions (e.g. Richardson & Rothstein,

2008), recent reviews have identified challenges in evaluating the total effects of

such interventions in the complex and multi-faceted context within which such

*Corresponding author. Email: K.Nielsen@uea.ac.uk

Work & Stress, 2013

Vol. 27, No. 3, 278�297, http://dx.doi.org/10.1080/02678373.2013.812358

# 2013 Taylor & Francis

interventions reside (Egan, Bambra, Petticrew, & Whitehead, 2009; Murta,

Sanderson, & Oldenburg, 2007). It has been suggested that organizational interven-

tions require elaborate evaluation frameworks due to their embeddedness in complex

social structures (Nielsen, Taris, & Cox, 2010d; Nielsen & Randall, 2012a) in order to detect what works for whom, why, how and under which circumstances (Pawson,

2006). Such a framework would require a consideration of the context within which

the intervention takes place, an examination of which intervention components are

effective, and an examination of how the design and implementation process of the

intervention helped to ensure a successful outcome (Murta et al., 2007).

In this article we present a framework of how organizational interventions may be

evaluated, taking into account the challenges of determining the total effects of such

interventions. We base this framework on an overview of the lessons learned from existing organizational intervention research. In their article, Grant and Wall (2009)

discussed the ‘‘why to’’ and ‘‘when to’’ of conducting intervention research. The

contribution of the present article is to provide insights into the ‘‘what to’’ look at

when evaluating organizational interventions. We present a research- and theory-

based framework for how to evaluate organizational interventions based on process

oriented organization theory (e.g. Tsoukas & Chia, 2002; Weick 1979). To our

knowledge, this is the first evaluation framework to combine both process and effect

evaluation and identify which elements should be looked at during each phase of an organizational intervention.

Based on existing organizational intervention research, we introduce the elements

that may have an impact on the outcomes of interventions, either directly through the

role they play in the implementation of the intervention and its activities or indirectly

through how they influence the behaviours of those involved. We also argue that the

impact of interventions must be evaluated at several levels: changes in attitudes,

values and knowledge, changes in individual resources, changes in organizational

procedures, changes in working conditions, changes in psychological health and well- being, changes in productivity and quality and changes in occupational safety and

management procedures.

Process organization theory as a theoretical framework

Our intervention framework is built on the foundation of the current focus on

‘‘organization as process’’ in the field of organization studies. In recent years scholars

have taken an increasing interest in viewing organizations as a continuous collective of processes that connect various players, or ‘‘actors’’ (Stengers, 2011). This is a

conceptual reformulation from the ‘‘social psychology of organizations’’ (Katz &

Kahn, 1978) to the ‘‘social psychology of organizing’’ (Weick, 1979). From a process

theoretical perspective, common constructs in intervention such as ‘‘resistance to

change’’, ‘‘managerial support’’ and ‘‘power relations’’ (Saksvik, Nytrø, Dahl-

Jørgensen, & Mikkelsen, 2002; Nielsen, Fredslund, Christensen, & Albertsen, 2006)

should be studied as on-going events occurring in the organization rather than seen

as generally stable intrinsic characteristics of the workplace (Hernes, 2008). When this perspective is applied to intervention evaluation it involves a shift in

focus from evaluating change as the certain movement from one fixed state (pre-

intervention) to another fixed state (post-intervention). The focus is instead directed

to the process aspects of change and specifically to how organizational interventions,

Work & Stress 279

as any other planned change activity, must be adapted to the routines and contextual

conditions within the organization (Tsoukas & Chia, 2002). There is a need for

organizational interventions to be understood as a collective of initiatives and change

activities, competing and intertwining with a multitude of concurrent events.

It has further been argued that organizational change should be seen as both a

ubiquitous given, and at the same time as an episodic occurrence in organizations

(Tsoukas & Chia, 2002). For example, when conducting an organizational interven-

tion, change becomes something specific and to some degree bounded in time and

space by the frame of the intervention, even though continuous adaptation and

change occurs before, during and after the change episode. This view of change as

both an episodic and a continuous phenomenon is an important factor in describing,

evaluating and understanding attempts to improve organizations. In our view, in

order to evaluate interventions aiming to achieve planned change we need to design

evaluation frameworks that are attentive to how change programmes are causing

effects in the organization. This can be done by such frameworks providing

an opportunity for action, but at the same time taking into account how intervention

activities are transformed and adapted to the contextual events and local culture in

the organization (Tsoukas & Chia, 2002).

We emphasize the link between the planned change of an organizational

intervention and concurrent changes within the context of the organization, as

these are in our view inseparable and a key element in understanding the complexity

and difficulties of intervention research. This would imply that an evaluation

framework should not focus on the activities set in motion by the intervention as

isolated events, but rather see them as situated in an environment containing forces

for both change and continuity. This leads us to include the mental models of actors

within the organization in our evaluation framework, because not only the

intervention processes themselves, but also the ongoing changing sentiments of

those actors towards the intervention programme and its elements play an important

role in influencing intervention process and outcome.

In summary, our framework, drawing on process organization theory, focuses on

documenting specific processes initiated by intervention programmes and on how

organizational actors and processes interact with the intervention activities to

influence intervention outcomes. To achieve a detailed understanding of change, it

should be borne in mind that concurrent events, such as budget cuts or mergers,

(shared mental) models, appraisals of intervention phases and events, management

and organization strategies, and the type of job can all influence an intervention, and

thus should be given attention in an evaluation.

The evaluation framework

We believe that process and outcome evaluation issues are irrevocably intertwined

(Nielsen et al., 2010d). Therefore, we should expand the view of the mechanisms that

can explain intervention outcomes (Pawson, 2006). Rather than merely focusing on

intervention activities (during the action planning and implementation phases) we

need to consider the phases and processes through which such activities are

developed and implemented. Pawson and Tilley (1997) developed the Context

Mechanism Outcome (CMO) model for realistic evaluation, emphasizing the need to

280 K. Nielsen and J.S. Abildgaard

look at what conditions in the Context are needed to bring about change, the

Mechanisms by which an outcome is brought about in the given context, and which

practical Outcomes are produced by these mechanisms. Our framework is structured

around four interlinked categories that are relevant to evaluation, as inspired by

the CMO model (Pawson & Tilley, 1997). However, our framework differs from the

Pawson and Tilley (1997) model in that we conceptualize Mechanisms differently.

Also, in contrast to previous proposals, we recommend that evaluation is carried out

at every stage of the process rather than only at the end of the intervention. The four

interlinked categories in our proposed framework are as follows. Firstly, as the

mechanisms to bring about change, we define the organizational actors who may

drive it (they include all key stakeholders who may influence the intervention process

and therefore the intervention outcomes, e.g. employees, line and senior managers,

researchers, Occupational Health and Human Resource consultants). The second

category is the mental models of those actors, which include cognitive schemata of

the organization, of working conditions and of the intervention including its purpose

and likely outcomes (Weick, Sutcliffe, & Obstfeld, 2005) that can help explain the

behaviours of the organizational actors. The third category is the contextual factors

surrounding the intervention activities that influence intervention outcomes, includ-

ing both the discrete contexts and the overall (omnibus) context. The fourth category

is the intervention design and processes. The framework is shown in Figure 1.

Figure 1. Proposed framework for the evaluation of organizational health interventions,

showing the various elements of an intervention, including both process and outcomes, that

need to be tracked and assessed within both the discrete contexts of individual phases and the

omnibus (overall) context of the intervention.

Work & Stress 281

In our framework we divide this last category, the evaluation of the intervention

design and process, into the phases commonly observed in organizational interven-

tions (Nielsen et al., 2010c). These are: initiation, screening, action planning,

implementation and effect evaluation. This latter category refers to Pawson and

Tilley’s (1997) Outcomes; however, based on the phased approach of organizational

interventions (Nielsen et al., 2010c), we suggest an approach where each phase can be

seen as the outcome of the previous phase, as shown in Figure 1. For instance,

the procedures for screening and the subsequent response rates are the result of the

planning that has taken place in the initiation phase; the outcomes of the action

planning phase depend on the quality of screening and the feedback provided based

on screening; and implementation depends on the level of detail in the action plans.

We believe that it is important to evaluate each of these phases separately to detect

how the decisions made and actions taken at one phase influence subsequent phases,

i.e. the mechanisms through which progress is made from one phase to the next

(Pawson, 2006).

In developing our evaluation framework we used four sources of information.

First, we reviewed frameworks from other disciplines, e.g. in public health,

participatory ergonomics and organizational development. In public health research,

the focus of evaluation is primarily on whether individuals change their own

behaviours (e.g. Rossi, Lipsey, & Freeman, 2004; Steckler & Linnan, 2002). While

this research is not directly transferable, useful information on implementation

fidelity can be transferred to intervention evaluation frameworks. In organizational

development, the focus is primarily on intervention outcomes. Valuable information

may be obtained about different levels of outcomes and targets of interventions (e.g.

Anderson, 2012; Cummings & Worley, 2009). From the participatory ergonomics

literature we can get valuable information on the phases of participatory intervention

and the factors needed to ensure successful implementation (Wells, Norman, Frazer,

& Laing, 2001). Second, we identified two reviews focusing on process factors (Egan

et al., 2009; Murta et al., 2007) and from here we identified relevant elements. Third,

we identified a number of papers discussing intervention implementation and

outcomes (Cooper, Dewe, & O’Driscoll, 2001; Guastello, 1993; Lipsey, 1996; Nytrø,

Saksvik, Mikkelsen, Bohle, & Quinlan, 2000; Pettigrew, 1990; Semmer, 2011;

Shannon, Robson, & Guastello, 1999; Vedung, 2006; Nielsen & Randall, 2012a).

Fourth, we conducted a thorough review of the existing research on organizational

interventions that includes information on the three areas of focus: the mental

models of the various actors, context and/or intervention design and implementa-

tion. (Full information on the systematic literature search can be obtained on request

from the authors). These are the papers that form the basis for the elements included

in our framework. Organizational interventions are a particular class of intervention,

but lessons learned from evaluating other sorts of interventions might be applicable

to them. However, this paper is focused on the knowledge derived from organiza-

tional intervention research. In the following sections we first introduce the three

categories that play a role in moderating and mediating the link between the

intervention and its outcomes � organizational actors, mental models and context � and then move on to describe the phases that should be considered in intervention

evaluation.

 

Looking for a Similar Assignment? Let us take care of your classwork while you enjoy your free time! All papers are written from scratch and are 100% Original. Try us today!

0 replies

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply

Your email address will not be published. Required fields are marked *