Self Assessment: Use at Operational Level to Promote

Continuous Improvement.

The European Foundation for Quality Management (EFQM) Excellence 2000

Model is widely adopted by organisations as a means of Self-Assessment to

enhance performance. This Self-Assessment has mainly focussed on the

company or strategic level, but it is argued here that the Self-Assessment process

can be modified to more appropriately suit the different needs of continual

improvement at an operational or departmental level. This paper reports a

generic process, applicable to the majority of situations, whereby Self-

Assessment can be developed to meet the continual improvement needs of

individual departments. The development process involves the application of

Grounded Theory Learning Activities to generate a set of departmentally

relevant constructs or issues which are then reviewed against a simple five step

model of Self-Assessment to determine the most appropriate departmental

approach. The process is illustrated through a case application showing the use

of the methodology.

By GC MacKerron, R Masson & Kevin McGlynn

Napier University, Craighouse Campus, Craighouse Road, Edinburgh, EH10 5LG

Key words: Self-Assessment, Operational Level, Continuous Improvement.

1
1.0 Introduction

Quality professionals will be familiar with the concept of Self-Assessment (SA).

Developed in the USA as far back as 1987 in the form of the Malcolm Baldrige

Award, SA has gained international recognition as a methodology for identifying

strengths and weaknesses broadly across an organisation's activities and performance

as part of their total quality management (TQM) activity. It might be argued that one

of the methodology's strengths is its holistic overall evaluation of activities and

performance, thus avoiding the lack of focus often apparent in the myriad of other

different operational or process level tools and methods that are available. The

European Foundation for Quality Management (EFQM) actively promote the use of

SA by stating that the ‘process allows the organisation to discern clearly its strengths

and areas in which improvements can be made and culminates in planned

improvement actions which are monitored for progress’ (EFQM 1998). The SA

criteria are directed toward almost all aspects of organisational management, which

may well be suited to companies or business units with a clear strategic focus utilising

financial figures – however this has not always been so suitable at the operational

levels where much more hands-on, non-financial and short term information is used,

especially with regard to continuous improvement. Current literature on SA

approaches (EFQM 1999, Arcelay 1999, Sneddon 1998, Oakland 1999) relates to

implementing the SA process over large business units, or at a corporate level. There

is a surprising lack of reference to implementing SA at an operational or departmental

level. Whether the process of SA using the EFQM Excellence Model (EM) is

unsuitable at operational levels, or whether the recognised and documented

approaches are equally applicable, has not been addressed in any depth, if at all.

2
The study reported here has researched SA methods, through literature review, case

studies of organisations and analysis of an operational department, and has concluded

that operational or departmental SA is not only possible, but it can deliver true

benefits to those organisations wishing to embrace SA below the strategic level.

Through this, there has been the development of a new holistic approach to

operational SA, using the principles of Grounded Theory. The developed approach

was rigorously tested by applying it to the operational level Customer Services

Department of a medium sized organisation in Central Scotland. The Customer

Services Department had 30 staff with responsibility relating to external relations

throughout the supply chain.

2.0 Self-Assessment and Continuous Improvement.

It is reasonable to argue that in order for organisations to survive and grow they must

implement and sustain a robust strategy of continuous improvement. (Dyason and

Kaye (1995), for example, identified competitive continuous improvement as a new

business performance requirement, in addition to the other criteria, statistical quality

control, quality assurance, inspection and strategic quality management, identified by

Bounds and York (1994). Continuous improvement in an organisation is more likely

to be sustained by assessing, on a regular basis, what activities are performing well,

which have stagnated, and what still needs to be improved. SA provides this

necessary feedback about the organisation’s approach to continuous improvement.

By utilising SA, the organisation is encouraged to take an in-depth look at its entire

operations, processes and customers, and to compare them to a best-practice

excellence model, such as the European Foundation for Quality Management’s

Excellence Model (Van der Wiele 2000).

3
3.0 Self Assessment using the EFQM Excellence 2000 Model.

The familiar EM, figure 1, is based on the concept that organisational results are

achieved through enablers, both being enhanced through the feedback and use of

innovation and learning. The model is structured on nine main criteria which

organisations can use to assess and measure their own performance with a view to

benchmark against the practices and performance of leading companies or ‘best in

class’. Through this process, the organisation identifies and exploits its strengths,

recognises weaknesses and applies improvement initiatives to eliminate them

(Azhashemi 1999).

Figure 1: the EFQM Excellence 2000 Model (EM)

There are a number of different approaches to carrying out SA against the EM. These

approaches range from relatively straightforward and simple opinion-based SA

approaches, like the Matrix and Questionnaire approaches, to more the rigorous and

time consuming evidence-based methods like the Workshop, Proforma and Award

Simulation approaches, see figure 2.

4
Figure 2: Approaches to Self Assessment.

DATA

Supported by Evidence
AWARD SIMULATION

PROFORMA

Process WORKSHOP
Rigour
LOW HIGH
MATRIX

QUESTIONNAIRE

Based on Opinion
© EFQM Self-Assessment Training Course 1997

While there is no ‘best approach’ (Porter and Oakland 1998) there are advantages and

disadvantages to each method, often related to the maturity of using the SA

methodology in the organisation and their circumstances. It has been found, Finn

(1999), that when organisations become more experienced in SA, they are more

likely to use a range of methods to obtain a more accurate assessment.

To demonstrate that any enhancement methodology actually leads to improved

business performance is a notoriously difficult task, although research carried out at

the European Centre for Business Excellence (Oakland 1999) has highlighted that

organisations adopting SA against the EM achieve sustained high levels of

performance in key areas including customer satisfaction, market share, revenue

growth and operating profit. Such results are also consistent with other research

carried out by Finn (1994), Longbottom (1998) and McAdam (1999).

5
4.0 The Experience of Users of Self Assessment.

Conti (1993), Hillman (1994) identified key good practice elements for success

within any SA process, viz., commitment of all those involved, and how it can be

developed, (Oldfield 1998); planning the SA process; justifying the method to be

used and designing appropriate formats for recording the data (if applicable);

achieving a common understanding of the process and the involvement and training

of all assessors; obtaining consensus on strengths and areas for improvement obtained

after assessment; establishing a plan of action resulting from the assessment; and

implementation of the action plan. Experience in the use of SA reveals that the

majority of organisations embark on SA utilising their own personnel; very few

employ external consultants. This is useful in ‘owning’ the process of assessment, but

it is important that those carrying out the SA process are aware of the factors

important for success. (Longbottom 1998, McAdam 1999, Zink 1998) however

suggest that too often the process of SA is simply delegated to internal quality

mangers, leading to the process being run as a distinct and separate activity from

those actually making policy decisions, and hence becoming merely a monitoring

exercise. Surprisingly the research identified that there is little involvement of

employees in the review process; SA has tended to be a management driven process

rather than a participative exercise across the organisation. It is also important that

line management are involved because the results will be more readily acted upon

through process improvement and they will be able to actually drive the process

themselves. This criticism is even more applicable as the SA process moves towards

the operational level, where increasingly, all staff need to engage in the process,

necessitating a more hands-on approach with the use of non-financial and shorter

term information, which can often require a different terminology.

6
If the EFQM Excellence Model is to be adopted as a framework for managing,

analysing and improving organisations’ performance, it must, in the long term, be

adopted by all levels in the organisation (EFQM 1998, Lascelles 1999).

5.0 Methodology.

Qualitative research was conducted into the application of SA at an operational level.

The research method chosen was based on the principles of Grounded Theory (Glaser

1990). This work involved, through use of gathered evidence, the development and

application of a procedure to provide a customized SA tool, for application within a

department of a business unit. The procedure was tested and validated by applying it

to the case department. Prior to this study, the case department had not attempted SA,

but had tried to use a range of different unrelated improvement-related techniques.

These were all short lived, with no cohesiveness or policy focusing or linking them to

strategic objectives. There had been no means of assessing the effectiveness of the

work to drive continuous improvement, nor the contribution of each improvement

technique's input to the department's enhancement. Previously applied techniques had

also been criticized in the case department for not containing enough emphasis on

‘softer’ management issues, something the customized SA method would address.

6.0 Preliminary Work

A set of broad acceptability criteria, table 1, against which the overall suitability of

the new assessment method would be tested were generated through a series of

meetings with the case department management. These were largely based on

management's experience of SA and their use of other improvement methodologies.

Table 1: Acceptability criteria for the new methodology.

7
1. Relevant and easy to apply to the department, but remain consistent to the EM.

2. Not be time-consuming to apply.

3. Must clearly identify strengths and areas for improvement, to facilitate action

planning.

4. Should be an evidence-based, not a perception-based, approach.

Research using Grounded Theory requires the use of Learning Activities. These

Learning Activities included a literature review by the authors, and interviews in two

further case organisations. The interviews were used to obtain information on the

perceptions and practices of SA in these case organisations, these acting as

benchmarks. This work enabled the establishment, and validation, of an emergent

theory from the research data obtained. Neither case organisation actively used the

EM at departmental level, however both saw this as something they aspired to and

were willing to discuss their progress towards doing this and address what they

thought were salient issues.

To ensure all staff within the case department in which the methodology was to be

developed were engaged or allowed to comment on the research, a series of

departmental meetings, involving all staff, were held to communicate the aims and

work of the research. The expected research outputs and recommendations were

viewed as catalysts to enable the formation of business improvement initiatives

necessary to enhance the department's operation.

The data and results from the learning activities described above, were collated and

distilled into a set of positive or negative constructs, or issues relevant to the case

department, in order to determine their impact on the approach taken. The case

department constructs are shown in table 2, below.

8
Table 2: Constructs of Departmental Self-Assessment.

Positive Constructs:

 Familiarity of the department with the concepts of TQM. Maximise the benefits

of the assessment, interpretation of the model and the assessment’s findings.

 Strong internal motivation for improvement; this enables SA to direct and

motivate continuous improvement.

 Trained Internal Assessors; facilitators to lead programmes ensuring a valid and

consistent assessment.

 Employee participation; enhances the process by increasing the amount of

evidence contributed and integrating the process into the department / operation.

 Customised Model criteria; ensures SA approach is relevant to department /

operation.

 Factual Assessment Approaches; delivers better results and allows action to be

taken on the basis of objective evidence.

 Identify Strengths and Areas for Improvement; assists improvement process.

 Use of RADAR; encourages a more scientific approach to assessing

performance.

 Prioritisation of Actions & Recommendations; provides structure to the

improvement process.

 Outcomes Linked to Business/ Operating Plans; creates stronger links.

 Communication of Results; creates awareness of process and ownership of

results.

Negative Constructs:

 Assessment Driven by External Consultants; reduces ownership of the approach.

9
 EFQM Questionnaire Booklet; does not provide a factual assessment and does

not assist the process of identifying areas for improvement.

 Poor Planning; creates confusion, de-motivates and reduces chances of success.

 Excessive List of Improvement Recommendations; de-motivates and obscures

important issues.

7.0 The Stepped Development of a Holistic Operational Self-Assessment

Approach.

The holistic operational level SA approach was developed through 5 simple stages or

considerations. These 5 stages are:

1. How relevant is SA to the departmental culture ?

2. Who is best suited to do the assessment ?

3. How can we customize the model criteria to suit our particular needs ?

4. What approach shall we use for the assessment ?

5. How can we use the outcomes to best advantage ?

In answering each of the above considerations, relevant elements from the positive

and negative constructs from table 2 for the case department were grouped together

for each consideration. The output or development resulting from this consideration

process at each stage was continually subjected to a verification process to assess its

suitability by comparison against the established acceptability criteria developed in

table 1.

The development of the departmental SA holistic approach through each of the 5

stages are shown below, including their related constructs and a description of the

work undertaken. For example in step 1, ‘TQM culture’ is seen as a critical element;

10
departments must review how and why they wish to do the assessment as well as their

maturity in relation to TQM.

Step 1. TQM CULTURE - How relevant is SA to the departmental TQM culture ?

Relevant constructs: Familiarity with concepts of TQM.

Internal motivation.

To assess the appropriateness or relevance of conducting SA, gauge the

organisation’s TQM maturity and its motivation for such an exercise. A review of

the department’s historical quality initiatives suggested that the department was

very familiar with the concepts of TQM, and would benefit from an assessment.

Motivation of staff for the procedure was found to be strong.

Step 2. ASSESSORS.- Who is best suited to do the assessment ?

Relevant constructs: Trained internal assessors.

Good employee participation.

For the purpose of the exercise, training provided to internal staff by a licensed

assessor course was deemed wholly appropriate. The trained assessor carried out

the assessment with inputs from key personnel.

Step 3. CUSTOMIZED MODEL - customizing the model criteria to suit our needs.

Relevant constructs: Customised model criteria.

Preliminary research was used to ascertain the most suitable SA model criteria for

the department. This was achieved by conducting two pilot exercises, one using

the EFQM (2000) Questionnaire booklet, ‘Determining Excellence’ (perception-

based) and the other using an example of a Pro-Forma approach (evidence-based)

on Leadership. Both methods were then evaluated and compared against the

acceptability criteria in table 1, to gauge suitability and applicability. This stage

included the development of a customised SA approach, adapted to fit the

11
requirements of the case department while retaining the integrity of the EM, and a

recognised implementation and resource plan. This aspect is further developed in

the following step 4. An important aspect, when carrying out the review of the

criteria, was the need to utilise phraseology or terminology appropriate to the

personnel involved in the exercise, thus avoiding confusion and making the model

user-friendly.

The need to re-define the terminology used to describe partners and customers was

also identified: a distinction was made between external and internal customers

and partners. This was an improvement, as it extended the benefits of the model

and make it more applicable to the case department. By amending the assessment

criteria, the approach was deemed to be relevant to the case department and

accordingly satisfied the first of the established acceptability criteria in table 1.

The assessment approach format chosen consisted of a collection of pro-forma

assessment sheets contained within an EFQM-type scorebook. This approach was

chosen to ensure factual data was gathered to aid measured improvement. Each

pro-forma sheet represented one of the 32 sub-criterion contained in the 9 main

criteria of the EM, figure 1. It was necessary to make some modifications to the 32

EFQM standard sub-criteria to suit the organisation under review and to meet all

positive constructs contained in table 2. For the case organisation, the EM sub-

criteria modifications, and the reason for these modifications, are shown in table 3.

This step is of paramount importance to the methodology and must be carried out

separately for any operational area to be assessed. Changes to the EM sub-criteria

shown for this organisation are relatively minor, but subsequent changes are

expected to increase or be enhanced once staff become more familiar with the

12
 concept; this in itself will form part of the continuous improvement effort. For

some departments, changes are likely to be more extensive.

Table 3: Sub-criteria modification for the case department

EM Model Adaptations/ amendments Reasons for adaptations/

Criterion amendments
Leadership.  Sub-Criterion1a.  Health, safety & environment is a

Inclusion of a reference very important issue for the

to the promotion of department.

health, safety &

environment issues.
Policy and  Sub-criteria refer to the  To avoid confusion and to make

Strategy. department rather than the model appear more relevant

the organisation. and user friendly.

People.  Sub-criteria refer to the  To avoid confusion and to make

department rather than the model appear more relevant

the organisation. and user friendly.

Partnership  Partners were defined as  Internal partners; teams, or

and either external, or functions, part of the

Resources. internal, to the organisation, but were outside the

organisation. department, for example, IT

support, Technical Teams.

Processes.  Sub-Criterion 5d was  Sub-criterion 5d was believed to

replaced by 5e. be not applicable to the

department.
Customer  Customers were defined  Internal customers were deemed

Results. as either external, or to be Teams, or functions, that

internal to the were part of the Organisation but

organisation. were outside the Department, for

13
 example, Manufacturing, Sales

Teams.
People  Sub-criteria refer to the  To avoid confusion and to make

Results. department rather than the model appear more user

the organisation. friendly.

Society  Sub-Criterion 8a was  Sub-Criterion 8a was thought to

Results. removed. be too impractical to measure.

Key  Sub-criteria refer to the  To avoid confusion and to make

Performance department rather than the model appear more user

Results. the organisation. friendly.

Step 4. ASSESSMENT APPROACH - Which assessment approach ?.

Associated constructs: Factual assessment approach.

Identification of strengths and areas for improvement.

Use of RADAR

An effective plan prior to assessment.

The assessment approach was developed, incorporating information from training

undertaken, and the pilot approaches described in step 3. It was decided that to

meets the needs of the department, an objective and factual evidence based

approach assessment was best suited. Central to achieving this, was the

incorporation of the comprehensive assessment process employed in the EFQM

Assessment Scorebook (EFQM 1999). This encouraged a reasoned evidenced

based assessment, through data collection, categorisation of strengths and areas for

improvement, and a judgement of excellence through the application of the EFQM

scoring mechanism, RADAR.

It was deemed necessary to rigorously test the new approach and ensure it was

comprehensible and applicable in context, and that it provided a basis for action

14
 (Glaser 1990). A project plan was formulated to identify the activities and resource

required to undertake a successful assessment. This plan also identified key

personnel, within the organisation, that would be involved in the process to solicit

input. After communicating and ensuring the plan was understood by all

concerned, the assessment programme was then implemented. The developed

approach was practical, comprehensible and user-friendly. The prescribed need to

gather evidence before assessing performance augmented the process integrity and

provided an objective foundation for decision-making. The link between the

assessment process and the systematic identification of strengths and areas for

improvement provided a fair, effective and transparent assessment. The use of the

EFQM scorebook enhanced the process by providing a clear and structured

approach to the assessment programme, whilst the concise and prioritised list of

recommendations was found to aid the improvement action planning process.

The outcome of the use of the above steps, when implemented, successfully

generated a series of recommendations to enable the department to focus on

improvement activities and assist its progression to excellence.

Step 5. OUTCOMES - Using the assessment outcomes to best advantage

Relevant constructs: Concise list of recommendations.

Prioritisation of actions/ recommendations.

Communication of results.

Outcomes linked to business plans.

Through use of the approach, key themes emerged requiring improvement,

subsequently prioritised, in order to create the maximum impact and use resources

productively. In addition, recommendations for future improvement were set out in

a structured manner, agreeable to all. Communication of the key issues and

15
 subsequent recommendations were found best delivered at group meetings,

complete with a formal report for reference. It was also believed the inclusion of

graphical representation of each criterion score would assist the reporting of the

assessment findings; this is to be used in future assessments for performance

measurement. In order to feedback assessment outcomes to business plans, it was

envisaged that a review of the assessment recommendations would be carried out

during department objective setting/ reviews, to ensure there was a tie-up of the

new process and longer term objective setting.

8.0 Case Department Reflections.

The case department sought to apply the SA process at an operational level, where

there was a need to undertake assessment to identify and prioritise improvements

through the use of positive feedback within the model and to sustain their strategy of

continuous improvement. It was clear that the use of the developed approach reported

here proffered a high level of understanding of operational SA by participating staff,

it was easily applied to the department, and facilitated the learning of best practices

and aided the transfer of ideas throughout the case department. Existing improvement

tools and methods could also be linked to aspects of the process and it generated

relevant and appropriate recommendations conducive to the department’s continuous

improvement programme, successfully detailing key themes and ensuing

recommendations that could be used to improve departmental operations. For this

situation, evidence-gathering SA methods were seen as having delivered more

accurate assessments and better results, when determining strengths and areas for

improvement, compared to opinion based approaches. The act of scoring was not

viewed as an important factor of the process, but thought to be of use in offering a

16
benchmark for future SA exercises. In addition, in the early stages of development,

use of a process mapping exercise should be undertaken to identify key processes, and

links to internal / external customers, suppliers and partners. It is also recommended

that the department recognises the need for internal measures for its key results,

ensuring these are understood. It is recommended that once awareness of the

usefulness of the mechanism at the operational level grows, more emphasis be placed

on measurement, assessment and review of strategy and processes to focus attention

on the means of achieving better operational performance. Using the developed

holistic approach provided the necessary drive and direction to the department’s

continuous improvement campaign. The formulation of recommendations to enhance

business performance was an integral part of this approach. It is the intention to carry

on with the approach throughout other departments within the organisation.

9.0 Conclusions and Recommendations.

SA against the EM remains a popular and practical way forward for many

organisations. For departmental or operational level use we have argued that in some

ways the accepted use of SA methodologies could be modified to be more appropriate

to the needs of individual departments. The work reported here describes a generic

process through which an individual department can go about determining its SA

requirements and evaluating and developing all aspects of SA to meet these needs. If

SA is to be performed at operational level, some experimentation and customisation

is needed before it can be applied. The project investigated the suitability and

applicability of operational SA by reviewing current techniques and then through the

development of a new holistic approach to operational SA, to help sustain a

programme of continuous improvement, through rigorous testing by applying it to the

17
department. The work reported here illustrates this methodology as it was applied to

a particular organisational department but the process can of course be considered as

generic and applicable to any department.

SA, against the EFQM’s Excellence 2000 Model, will deliver true and tangible

benefits to organisations. This statement is just as accurate when assessing at the

strategic level as it is at the operational level, where there is a true need to assess in

order to ensure prioritised improvements are being effective through the use of

positive feedback within the model and to sustain their strategy of continuous

improvement.

18
10.0 References.

ARCELAY, A., SANCHEZ, E. and HERNANADEZ, L. et al, 1999, Self-assessment

of all the health centres of a public health, International Journal of Health Care

Quality Assurance, Vol.12, No.2, pp.54-58.

AZHASHEMI, M.A. and HO, S.K.M., 1999, Achieving Service Excellence: A new

Japanese approach versus the European framework, Managing Service Quality, Vol.9,

No.1, pp.40-46.

BOUNDS, G and YORKS, L et al, 1994 Beyond TQM Toward the Emerging

Paradigm, (London: McGraw-Hill).

CONTI, T., 1993, Building Total Quality: A Guide to Management, (Chapman and

Hall, London).

DYASON, M.D. and KAYE, M.M., 1995, Is there life after total quality

management?(II), Quality World Vol1, pp23-31.

EFQM, 1999, Model Scorebook, (EFQM Publications, Brussels).

EFQM Press, 2000, Determining Excellence, (EFQM Publications, Brussels).

EFQM, 1998, Self-Assessment Guidelines for Companies, (EFQM Publications,

Brussels).

ESKILDSEN, J.K., DAHLGAARD, J.J. and NORGAARD, A., 1999, “The Impact of

Creativity and Learning on Business Excellence, Total Quality Management,

Abingdon, Vol 6(3), pp43-51.

FINN, M. and PORTER, L. J., 1994, TQM Self-Assessment in the UK, The TQM

Magazine, Vol.6, No.4, pp.56-61.

GLASER, B.G. & STRAUSS, A, 1990, The discovery of grounded theory, (Aldine,

Chicago).

19
Hillman, P.G., 1994, Making Self Assessment Successful, The TQM Magazine, Vol. 6

No.3, pp29-30

LASCELLES, D. and PEACOCK, R, 1999, Self-Assessment for Business Excellence,

(McGraw-Hill, London), 1st Edn, pp103-127.

LONGBOTTOM, D., 1998, Self-Assessment Game Over?, Total Quality

Management, Abingdon.

MCADAM, R. and O’NEILL, E., 1999, Taking a critical perspective to the European

Business Excellence Model using a balanced scorecard approach: a case study in the

service sector, Managing Service Quality, Vol.9, No.3, pp.191-197.

OAKLAND, J., 1999, Winning Performance Through Business Excellence, Credit

Control, Hutton, Vol 3, pp43-56.

OLDFIELD, H., 1998, Quality Awards, Quality World, pp. 22-23.

PORTER, L., OAKLAND, J. and GADD, K., 1998, Unlocking Business

Performance With Self-Assessment, Management Accounting, London, Vol 5, pp55-

72.

SNEDDON, J., 1998, The Business Excellence Model: Will it Deliver?, Management

Services, Vol2, pp56-70..

VAN DER WIELE, T., DALE, B. and WILLIAMS, R., 2000, Business Improvement

Through Quality Management Systems, Management Decision Vol.38, No.1, pp.19-

23.

ZINK, K.J., and SCHMIDT, A., 1998, Practice and Implementation of Self-

Assessment, International Journal of Quality Science, Vol.3, No.2, pp.147 - 170.

20