Received: 15 February 2020 Revised: 17 August 2020 Accepted: 17 August 2020

DOI: 10.1002/cpp.2510

ASSESSMENT

A short scale to evaluate supervision and supervisor

competence—The SE-SC8

Craig J. Gonsalvez

School of Psychology, Western Sydney

University, Penrith, New South Wales, There is an urgent need for instruments to evaluate supervisor competence. The cur-
Australia
rent study describes the development of the SE-SC8, an eight-item version of a previ-
Correspondence ously published Supervision Evaluation and Supervisory Competence scale that has
Craig J. Gonsalvez, School of Psychology,
adequate reliability and validity. The current study analysed data from the sample
Western Sydney University, Locked Bag 1797,
Penrith 2751, New South Wales, Australia. (N = 142) used in the initial validation of the full scale to identify the best set of items
Email: c.gonsalvez@westernsydney.edu.au
for a short scale before testing the scale's psychometric properties in a new study
(N = 122). The SE-SC8 demonstrated good reliability and adequate convergent and
divergent validity. The SE-SC8 has two overall items respectively measuring supervi-
sor effectiveness and supervision satisfaction and six items that represent supervi-
sory competencies, namely, (1) Openness, Caring and Support, (2) Supervisor's
Knowledge and Expertise as Therapist, (3) Supervision Planning and Management,
(4) Goal-Directed Supervision, (5) Restorative Competencies, and (6) Insight into and
Management of Therapist–Client Dynamics and Reflective Practitioner Competencies. As
with other subjective ratings, SE-SC8 scores should be supplemented by other assess-
ments for a holistic appraisal of supervisor competence.

KEYWORDS

psychology instruments, psychometric properties, short tests, supervision evaluations,

supervision scales, supervisor competence

1 | I N T RO DU CT I O N expert committees have formulated an authoritative description and

classification of benchmarks that could guide psychology practitioner
The last two decades have witnessed the ascendancy of competency- education and training (Fouad et al., 2009; Rodolfa et al., 2013). The
based models for practitioner training in psychology and across other model differentiates between foundational competencies (e.g., ethical
health professions. In the words of Watkins and Milne (2014a), “there and legal foundations, individual and cultural diversity, relationships,
has perhaps never been a more sweeping and all-pervasive singular and reflective practice) and functional competencies (e.g., assessment,
change to so rapidly impact the entirety of the supervision enterprise intervention, consultation, and supervision) and has described bench-
than the advent of the competency movement itself” (p. 674). marks for both sets of competencies at three different developmental
Although understandable differences between training (e.g., Hatcher stages (readiness for practicum, readiness for internship, and readiness
et al., 2013) and practitioner groups (e.g., Rodolfa et al., 2005, 2013) for entry to practice; Fouad et al., 2009).
persist with regard to the specific domains and the number and com- Important contributions by regulatory and professional bodies
plexity of the competencies within each domain, significant progress nationally and internationally (e.g., American Psychological
has been achieved and expert consensus reached with regard to the Association, 2014; Pilling & Roth, 2014) alongside contributions by
relevance and usability of a three-dimensional, cube-based model of experts in competence pedagogies have helped extend the critical
practitioner competence (e.g., Hatcher et al., 2013; Rodolfa examination of practitioner competence to the nature of supervisor
et al., 2005, 2013). On the basis of the competency-cube model, competence (e.g., Falender & Shafranske, 2007; Neuer Colburn,

452 © 2020 John Wiley & Sons, Ltd. wileyonlinelibrary.com/journal/cpp Clin Psychol Psychother. 2021;28:452–461.
GONSALVEZ 453

Grothaus, Hays, & Milliken, 2016). These initiatives have led to the
formulation of principles and best-practice guidelines to promote the Key practitioner messages
development and maintenance of supervisory competence
(Falender & Shafranske, 2007, 2014; Gonsalvez & Calvert, 2014).
• Recent advances have led to a rich and nuanced concep-
Using a similar approach to the conceptualization of practitioner com-
tualization of supervisor competence, but there is a
petence, frameworks for supervisory competence have been formu-
dearth of short, reliable, and valid instruments to estimate
lated by a stepwise listing of individual competencies, clustering them
competence in regular supervisory practice.
into pedagogically meaningful groups, and then organizing competen-
• The SE-SC8, a brief, eight-item, supervision evaluation
cies into a hierarchical or multidimensional framework. Within this
and supervisor competence scale, has relatively good
meta-theoretical model, “Supervision that applies a competency-
psychometric properties and captures key elements
based approach entails the creation of an explicit framework and
included in recent conceptualizations of supervisor
method to initiate, develop, implement, and evaluate the process and
competence.
outcomes of supervision. A competency-based approach is predicated
• Because the SE-SC8 is designed to provide a quick evalu-
on supervisors having the knowledge, skills, and attitudes regarding
ation, it is ideally suited for regular and ongoing monitor-
the provision of quality supervision and professional psychology
ing of supervision, whereas the full scale is better suited
models, theories, practices.” (APA, 2014, p. 4).
for a more comprehensive evaluation at the end of place-
The framework for supervision, approved by the APA Council of
ments, internships, or supervision contracts.
Representatives, is built around seven domains: Diversity; Supervisory
• The SE-SC8 provides a subjective estimate of compe-
Relationship; Professionalism; Assessment/Evaluation/Feedback; Pro-
tence and may be used by a supervisee, a peer-supervi-
fessional Competence Problems; and Ethical, Legal, and Regulatory
sor, or by the supervisor themselves. These evaluations
Considerations (APA, 2014). In their analysis of supervisor compe-
should be complemented by other objective competence
tence frameworks designed by professional and regulatory bodies
assessments.
from the United States, the United Kingdom, and Australia, Watkins
• The SE-SC8 does not specifically measure some impor-
and Milne (2014b) found striking similarities. They identified six fun-
tant supervisor competencies (e.g., adequate oversight of
damental and common areas of supervision competency, namely,
ethical and legal issues and providing effective summa-
(i) knowledge about and understanding of supervision models,
tive assessments). Should specific measures of these
methods, and intervention; (ii) knowledge about and skills in attending
competencies be required, a 12-item version (see Appen-
to matters of ethical, legal, and professional concern; (iii) knowledge
dix A) that includes the eight items from the SE-SC8 and
about and skills in managing supervision relationship processes;
four additional items should be considered.
(iv) knowledge about and skill in conducting supervisory assessment
and evaluation; (v) knowledge about and skill in fostering attention to
difference and diversity; and (vi) openness to and utilization of a self-
reflective, self-assessment stance in supervision (p. 11). All frame- Borders, 2016). A further problem is that because most measures
works recognize that competencies are multifaceted in that they have have been used in only one study, there has been a less than adequate
knowledge (e.g., knowledge of cognitive therapy and knowledge of opportunity to examine the scope of these instruments, track their
educational principles), skills (e.g., technical and relationship capabili- effectiveness, or to develop and refine their potential (Ellis, D'Iuso, &
ties to enhance these competencies in supervisees), attitude-value Ladany, 2008; Gonsalvez & Milne, 2010; Reiser, Cliffe, & Milne, 2018;
(e.g., commitment to the welfare of client and trainee and to Wheeler & Barkham, 2014).
evidence-based practice), and metacompetency components
(APA, 2014; Gonsalvez & Calvert, 2014; Pilling & Roth, 2014; Psychol-
ogy Registration Board, 2018). 1.1 | The need for instruments to evaluate
In an extensive review of measures used to evaluate supervision supervisor competence
processes, styles, satisfaction with supervision, and effective supervi-
sion, Wheeler and Richards (2007) identified as many as 59 measures. An emphasis on the ongoing monitoring and ecologically valid assess-
They also noted that valid and accurate measurement in supervision ment of competence is of inherent importance to competency-based
was compromised by serious inadequacies. Few measures had pedagogies. Consequently, the commitment to competency-based
established psychometric properties, and some aspects of the supervi- models for practitioner training and supervision has led to rigorous
sory process were emphasized to the neglect of others. For instance, scrutiny of competency assessment procedures, which, in turn, has
the supervisory relationship was studied quite extensively, with revealed an urgent need for reliable and valid instruments to measure
11 instruments designed to capture a construct that remains broad, supervisor competence (Falender & Shafranske, 2007, 2014;
nuanced, and somewhat elusive with its conceptual boundaries over- Gonsalvez & Milne, 2010; O'Donovan & Kavanagh, 2014; Pilling &
lapping with related constructs such as transference phenomenon and Roth, 2014; Reiser et al., 2018). There are compelling reasons to prior-
parallel process (Palomo, Beinart, & Cooper, 2010; Tangen & itize this issue. First, increased acceptance of competency-based
454 GONSALVEZ

approaches to supervision has underlined the requirement that instru- effectiveness, an important discrimination within competency
ments should map well onto new conceptual frameworks. A close approaches (O'Donovan & Kavanagh, 2014).
alignment with these frameworks is an essential first step within the
larger design to chart growth trajectories across competency domains,
types, and diversity of supervisory approaches and practices 1.2 | The need for short scales to measure
(Falender & Shafranske, 2014; Watkins, 2012). Second, most supervisor competence
established instruments focus on some (e.g., the supervisory relation-
ship or supervisory styles) but not all aspects of supervisory compe- In parallel with emphasizing the need for measures of supervisor com-
tence believed to be important for supervisory effectiveness petence, recent reviews have highlighted the need for short scales.
(Gonsalvez & Milne, 2010; Ladany, Mori, & Mehr, 2013; O'Donovan & This gap is particularly significant in the context of converging
Kavanagh, 2014; Pilling & Roth, 2014; Reiser et al., 2018). For research demonstrating better client outcomes following routine out-
instance, mapping the elements of the Supervisory Working Alliance come monitoring and feedback through the use of short scales. Such
measure onto current conceptualizations of supervisor competence research uses short scales following completion of each/alternate cli-
reveals that the SWAI focuses on some competencies (e.g., skills in ent session(s) to evaluate symptom change and therapist performance
managing supervision relationship processes) but not on other impor- (e.g., Barkham et al., 2010; Brattland et al., 2018; Shimokawa, Lam-
tant domains (e.g., attention to difference and diversity, ethical and bert, & Smart, 2010). These results have led researchers to hypothe-
legal competencies, and expertise in and skills to facilitate therapist– size that similar routine outcome monitoring and feedback applied to
client competence). Also, the SWAI focuses on relationship skills and the supervisor–supervisee dyad would improve training outcomes and
processes but pays inadequate attention to knowledge and attitude- supervision effectiveness (Wheeler & Barkham, 2014; Wheeler &
value competencies (e.g., respect for scientific methods and empirical Richards, 2007). To progress such research, the development of easy-
evidence, commitment to client care, respect for diversity, and com- to-use short scales that can be completed by supervisors and trainees
mitment to one's professional development). Finally, to evaluate the on a regular basis across a range of situations would be an essential
comparative effectiveness of supervisor competence on the growth step. Full and short scales of the Supervision: Adherence and Guid-
trajectories of trainee competence and on client outcomes, we require ance Evaluation (SAGE) have been derived and subjected to psycho-
longer instruments that can provide a comprehensive profile of com- metric evaluation (Milne & Reiser, 2011). The SAGE was specifically
petence at pivotal times during a supervision period (e.g., end of designed for cognitive-behavioural therapy (CBT) supervision when
internships) alongside short scales that regularly assess ongoing the supervisor has access to the observation of a supervisee–client
changes during these placements. session (e.g., through an audio/video record), so its application in
A recent review (Reiser et al., 2018) identified four scales of supervision would be restricted to these situations.
supervisor competence, namely, a competence framework for the As far as we are aware, a short scale that evaluates the broad
supervision of psychological therapies (Newman-Taylor, Gordon, range of supervisor competencies is yet to be published, and the cur-
Grist, & Olding, 2012; Pilling & Roth, 2014), the Supervisory Compe- rent study addresses this need. Given the merits of the SE-SC, a set of
tence Scale (Rakovshik, 2015), the Supervision Evaluation Scale eight-items was selected (SE-SC8) on the basis of the data originally
(Corrie & Worrell, 2012), and the Supervision Evaluation and Supervi- used in the development of the full scale. A confirmatory study was
sory Competence scale (SE-SC; Gonsalvez, Hamid, Savage, & then conducted to test the psychometric properties of the SE-SC8
Livni, 2017). Regrettably, the psychometric properties of two of the with a new dataset. Because the SE-SC full scale and a discussion of
instruments, the Supervisory Competence Scale (Rakovshik, 2015) its psychometric properties have been published, the primary aim of
and the Supervision Evaluation Scale (Corrie & Worrell, 2012), remain the current study was largely limited to the determination whether a
unpublished in peer-reviewed journals, making a comprehensive and shorter scale could reliably represent the full scale and be fit for use
reliable appraisal of their scope and utility difficult to establish. The for ongoing and routine estimates of a range of supervisor competen-
Roth and Pilling framework () comprises a list of 18 generic and spe- cies. This aim was tested by two hypotheses: Hypothesis (1): reliability
cific supervisor competencies, and, although the framework has been measures, including internal consistency measures for the SE-SC8 and
used to evaluate the effectiveness of a programme of supervisor correlations between means scores for the short and full scale, would
training, the psychometric properties of the scale itself was not exam- be high; and Hypothesis (2): items within the SE-SC8 representing the
ined in the publication (Newman-Taylor et al., 2012). The SE-SC scale six clusters would correlate highly with mean scores of the clusters
(Gonsalvez et al., 2017) has several merits. It has good psychometric from the full scale.
reliability and validity, there is a good conceptual alignment with cur- An additional aim of the study was to examine convergent and
rent competency-based training models with items covering a range divergent validities for the short scale. In the absence of a psychomet-
of competency domains and competency facets (e.g., knowledge, rically valid and established gold-standard instrument of supervisor
technical skills, relationship skills, attitude-values, and meta-compe- competence, we predicted that mean scores of the SE-SC8 would cor-
tencies), it is easy to administer, and it can be used across different relate modestly with other established measures of supervision evalu-
theoretical and supervisory approaches. Additionally, the scale differ- ation, such as the SWAI and SSI. In other words, convergent validity
entiates between supervision satisfaction and supervision would be demonstrated by a pattern of higher correlations between
GONSALVEZ 455

similar constructs measured by the SE-SC8 and other instruments clusters. Gonsalvez et al. (2017) report details of the sample, the
(SWAI/SSI), and divergent validity would be demonstrated by lower measures used, and the psychometric properties including internal
correlations between dissimilar constructs. Specifically, we predicted consistency measures for its six subscales, test–retest reliability, and
that scores on (Hypothesis 3a) Cluster 1, Openness, Caring and Sup- convergent validity, but made no attempt to derive a short scale in
port, would be more highly correlated with Rapport (SWAI) and Inter- the original study.
personal Sensitivity (SSI) than it would with Client-Focus (SWAI) and
Task-Orientation (SSI); (3b) Cluster 4, Goal-directed Supervision, would
be more highly correlated with Task-Orientation than it would with 2.3 | The SE-SC8
Interpersonal Sensitivity on the SSI; (3c) Cluster 5, Restorative Compe-
tencies, would have a stronger relationship with Interpersonal Sensitiv- In the original study, Gonsalvez et al. (2017) subjected the 22 specific
ity than it would with Task-Orientation on the SSI; and (3d) Cluster items of the SE-SC to a hierarchical cluster analysis that produced six
6, Insight into and Management of Therapist-Client Dynamics, would first-order clusters that represented different supervisor competen-
have a stronger relationship with Client-Focus (SWAI) than it would cies: (1) Openness, Caring, and Support, (2) Supervisor's Knowledge and
with Rapport (SWAI) or Task Orientation (SSI). Expertise as Therapist, (3) Supervision Planning and Management,
(4) Goal-directed Supervision, (5) Restorative Competencies, and
(6) Insight into and Management of Therapist–Client Dynamics and
2 | METHOD Reflective Practitioner Competencies. These results justified the forma-
tion of an eight-item short scale, the SE-SC8, that incorporated six
2.1 | Participants in Dataset A items capturing the six independent supervisor competencies derived
from the original cluster analysis and two additional items measuring
The sample in Dataset A was the same sample that formed the basis overall satisfaction and overall supervision effectiveness. To select the
of the psychometric validation of the full scale in the original study items that best represented each of the clusters, Pearson's correla-
(Gonsalvez et al., 2017) and comprised 142 supervisees who were tions were computed between the mean scores of each of the clusters
completing their clinical placements (rotation/externship) as part of an and each of the items that were contained within the cluster. The item
accredited psychology-training programme in Australia. At the end of that correlated highest with the cluster mean was included in the
their placements, to evaluate the supervision they received, super- short scale.
visees completed a battery of tests that included (i) Demographic and
training information, (ii) the SE-SC scale (31 items), (iii) the 19-item
Supervisory Working Alliance Inventory (SWAI; Efstation, Patton, & 2.4 | Participants in Dataset B
Kardash, 1990), and (iv) the Supervisory Styles Inventory (SSI,
33-items; Friedlander & Ward, 1984). Both the SWAI and the SSI have 2.4.1 | Supervisees
established psychometric properties. For the SWAI, adequate internal
consistency measures (α = .90 for Rapport, and α = .77 for Client Dataset B contains unpublished data from 122 supervisees, all of
Focus) and acceptable factorial validity for the two subscales are whom were completing their clinical placements (rotation/externship)
reported (Efstation et al., 1990). The psychometric properties of the as part of a clinical psychology training programme accredited by the
SSI were established over a series of studies demonstrating good con- Australian Psychology Accreditation Council. Supervisees received a
tent and factorial validities and good reliabilities for the three sub- weekly 90-min group supervision session for 12–14 weeks. Super-
scales (α values ranging from .76 to .93; Friedlander & Ward, 1984). In visees were enrolled in either a Master's degree in clinical psychology
the current study, of 142 participants, 131 completed the SE-SC, or a doctoral programme in clinical psychology (PsyD or PhD) and
70 participants completed both the SE-SC and the SWAI, and 51 partic- were in their first, second, third, or fourth clinical placement. At the
ipants completed the SE-SC, the SWAI, and the SSI. end of their placements, supervisees anonymously evaluated the
supervision received from their respective group supervisors by com-
pleting the full version of the SE-SC either online or in hard copy.
2.2 | The SE-SC full scale Group size varied from four to six supervisees.

The full version of the SE-SC scale comprises 31 items. Three items
(e.g., summative assessments of supervisee competence) were 2.4.2 | Supervisors
excluded from the analyses because a large number of supervisees
considered that these competencies were not addressed in supervi- Because supervisees completed the evaluation about their supervi-
sion and rated these items as “not-applicable.” Therefore, 28 items sors, and because it was important to reassure supervisees and super-
were analysed: (i) three items that measured overall satisfaction with visors that their identity would not be obvious to programme
supervision, (ii) three items that measured overall effectiveness of coordinators, information about the supervisor's age, gender, and
supervision, and (iii) 22 items that mapped onto six independent practitioner and supervisory experience was not solicited. In
456 GONSALVEZ

accordance with requirements for accreditation, all supervisors held a supervisee competence (e.g., through live or recorded sessions)
current Board registration as clinical psychologists and had a minimum (6%), (iii) role play (7%), and (iv) other methods including
of two or more years of practitioner experience following completion discussions/exercises regarding supervision process, ethics, and
of their registration. In addition, mandatory training introduced by the reflective practice (16%). In terms of supervision time allocated to the
licencing authority in Australia required supervisors to complete the various objectives/goals of supervision, 83 of 122 (68%) supervisees
equivalent of a minimum of 18-hours of supervisor training imparted provided the information requested to indicate that supervision time
through skills training workshops that covered content including was allocated to the following goals: (i) knowledge and knowledge-
supervision theory and models, supervision process, ethics, compe- application (29%), (ii) skills training (27%), therapist-client relationship
tence assessment, and evaluation. Supervisors demonstrated compe- (14%), supervisor–supervisee relationship (8%), self-awareness and
tence by satisfactory completion of knowledge and skill exercises reflective practice (19%), and other objectives/goals (3%). The study
embedded in their training and, following their training, submitted a was approved by the University's Ethics committee.
videotape of their supervision to pass an objective skills-assessment.
On the basis of information provided by supervisees, the theoretical
approach adopted by supervisors in their practice was (i) CBT (50%), 3 | ANALYSES AND RESULTS
(ii) Behaviour Therapy (32%), (iii) Acceptance and Commitment Ther-
apy (8%), (iv) Psychodynamic (6%) and other approaches (4%). The 3.1 | Internal consistency and reliability
popularity of the CBT approach or one of its variants in the current
sample is consistent with data reported for accredited clinical psychol- Internal consistency measures (Cronbach's α) were computed for the
ogy programmes in Australia (Gonsalvez, Deane, Terry, Nasstasia, & SE-SC Full scale and the SE-SC8. To test the reliability of the short
Shires, 2019). Supervisees were also requested to provide information scale, we computed correlations (Pearson's r) between the mean
about the percentage of group supervision time that supervisors typi- scores of the SE-SC (full scale) and the SE-SC8. Consistent with
cally devoted to commonly used supervisory methods or techniques. Hypothesis 1, for the SE-SC8, reliability measures were high (α = .89)
Ninety-three of 122 (76%) supervisees provided these data that indi- and SE-SC8 mean scores were strongly correlated with full
cated that supervisors spent the most time in (i) case presentation, scale means in both datasets, in each instance, r = .97, p < .001
discussion, and feedback (75% of supervision time), (ii) observation of (See Table 1).

TABLE 1 Correlations between SE-SC8 items and SE-SC Full Scale overall and cluster scores

SE-SC Full Scale

Data A Data B
Overall scores: Item description (N = 140) (N = 122)
Overall scores SE-SC8: Mean score (eight items) .97*** .97***
Overall satisfaction Overall, my expectations of supervision were .86*** .83***
matched or exceeded
Overall effectiveness Overall, supervision significantly enhanced my .68*** .71***
competence as a practitioner and professional
Cluster number and name Clusters: item description
1. Openness, caring, and support The supervisor was approachable, caring, and .82*** .72***
supportive
2. Supervisor's knowledge and expertise as a The supervisor impressed me as a skilled and .68*** .68***
therapist effective therapist
3. Supervision planning and management Supervision goals and competencies were designed to .77*** .77***
match my developmental needs
4. Goal-directed supervision Supervision sessions were thoughtfully structured, .60*** .58***
and supervision activities were goal-driven
5. Restorative competencies Supervision provided a safe place for emotional .73*** .65***
ventilation and support.
6. Insight into and management of therapist–client The supervisor helped me understand my patterns of .76*** .72***
dynamics and reflective practitioner emotional responding in client work, including
competencies when appropriate and schema/transference-type
reactions
*
p < .05.
**
p < .01.
***
p < .001.
GONSALVEZ 457

3.2 | Representation of cluster and overall scores the correlations were marginally lower (r = .68, p < .001 for Dataset A
by individual items within the SE-SC8 and r = .71 for Dataset B).

To determine the extent to which individual items within the short

scale adequately captured the clusters they represented in the full 3.3 | Convergent and divergent validity
scale, we computed correlations between the mean cluster scores
derived from the full scale and the corresponding items in the SE-SC8. Convergent and divergent validity for the SE-SC8 were determined by
To prevent inflated correlations, we excluded the item included in the computing correlations between SE-SC8 items and sub-scale scores of
short scale before computing the cluster mean. For instance, Cluster the SWAI and SSI. The hypotheses predicted that correlations
1, comprising a set of five items in the full scale, was represented by a between similar constructs would show high correlations, whereas
single item (Item 7) in the SE-SC8. We therefore computed the corre- correlations between dissimilar constructs would show low or modest
lation between Item 7 and the mean derived from the remaining four correlations. These results are presented in Table 2.
items. A similar analysis was conducted for overall satisfaction and All four predictions concerning convergent and divergent validity
overall effectiveness. The results for both datasets are presented in were supported. Specifically, Cluster 1, Openness, Caring and Support,
Table 1. was more highly correlated with Rapport (SWAI, r = .77, p < .001) and
As predicted by Hypothesis 2, the individual SE-SC8 items reliably Interpersonal Sensitivity (SSI, r = .85, p < .001) than with Client-Focus
predicted cluster scores derived from the full scale. Correlations (SWAI, r = .41, p < .001) and Task-Orientation (SSI, r = .35, p < .05;
(Pearson's r) ranged from .68 to .82 for Dataset A, and from .65 to .72 Hypothesis 3a). Similarly, Cluster 4, Goal-directed Supervision, was
for Dataset B, with all correlations significant at p < .001. Of the six more highly correlated with Task-Orientation (r = .54, p < .001) than it
clusters, goal-directed supervision (Cluster 4) elicited a modest but was with Interpersonal Sensitivity on the SSI (r = .37, p < .05; Hypothe-
somewhat weaker correlation, r = .60, p < .001 for Dataset A and sis 3b); Cluster 5, Restorative Competencies, had a stronger relationship
r = .58, p < .001 for Dataset B. Overall satisfaction was reliably cap- with Interpersonal Sensitivity (SSI, r = .69, p < .001) than it did with
tured by the SE-SC8 item, as demonstrated by strong correlations for Task-Orientation (SSI, r = .12, p > .05; Hypothesis 3c); and Cluster
both datasets (r = .86, p < .001 for Dataset A and r = .83, p < .001 for 6, Insight into and Management of Therapist-Client Dynamics, had a
Dataset B). Overall effectiveness was also reliably captured, although stronger relationship with Client-Focus (r = .60, p < .001) than it did

TABLE 2 Correlations between the SE-SC8 items and subscales of the SWAI and the SSI

SWAI (N = 70) SSI (N = 51)

SE-SC8 Rap Client At IP Task

SE-SC8 mean score .70*** .71*** .63*** .78*** .35*
Cluster number and name Item description
1. Openness, caring, and support The supervisor was approachable, caring, and .77*** .41*** .62*** .85*** .35*
supportive
2. Supervisor's knowledge and expertise as a The supervisor impressed me as a skilled and .53*** .54*** .64*** .76*** .36*
therapist effective therapist
3. Supervision planning and management Supervision goals and competencies were designed .66*** .55*** .56*** .72*** .41**
to match my developmental needs
4. Goal-directed supervision Supervision sessions were thoughtfully structured, .55*** .64*** .44*** .37** .54***
and supervision activities were goal-driven
5. Restorative competence Supervision provided a safe place for emotional .65*** .53*** .50*** .69*** .12
ventilation and support
6. Insight into and management of therapist– The supervisor helped me understand my patterns .26* .60*** .39** .52*** .05
client dynamics and reflective practitioner of emotional response in client work (including
competence when appropriate and
schema/transference-type reactions)

Note: Correlations are based on N = 140 for SE-SC; N = 70 for the SWAI; N = 51 for SSI. Rap = Rapport subscale and Client = Client Focus subscale;
At = Attractiveness, IP = Interpersonal Sensitivity, and Task = Task Orientation.
Abbreviations: SSI, Supervisory Styles Inventory; SWAI, Supervisory Working Alliance Inventory.
*
p < .05.
**
p < .01.
***
indicates p < .001.
458 GONSALVEZ

with Rapport (SWAI, r = .26, p < .05) or with Task-Orientation (SSI, from .54 to .85 with a mean of .69) are comparable, if not higher, than
r = .05, p > .05; Hypothesis 3d). patterns reported by other well-established instruments. For instance,
convergent validity correlations ranging from .21 to .65 (Pearson's r)
were reported in the development and validation of the SSI and the
4 | DISCUSSION SWAI (Friedlander & Ward, 1984).

The primary aim of the current study was to design a short scale of
supervisor competence and to test its reliability. On the basis of data 4.2 | Subjective evaluations versus objective
originally used in the full-scale, an eight-item scale, the SE-SC8, was competence assessments
developed. A confirmatory study was then conducted to test the psy-
chometric properties of the SE-SC8 with a new dataset. The SE-SC8 It is important to differentiate between subjective estimates and
demonstrated excellent internal consistency (α = .89) and high reliabil- objective assessments of competence. The SE-SC and its shorter ver-
ity evidenced through high correlations between short- and full-scale sion, the SE-SC8, were designed to be used by untrained supervisees,
means (Datasets A and B in Table 1). Additionally, we examined the peer-supervisors, and supervisors themselves (e.g., for self-evalua-
reliabilities with which individual items in the short scale predicted the tions) for impressionistic evaluations of supervisory competence dur-
respective cluster means they represented in the full scale. Relatively ing and at the end of placements/internships/supervision contracts.
high correlations (mean r = .70) were obtained for five of the six clus- Therefore, these scales provide supervisors, training programmes, and
ters in the confirmatory study, thereby demonstrating an acceptable other stakeholders a quick, inexpensive, impressionistic, and experien-
level of reliability for the items chosen to represent the clusters in the tial rating of competence across a range of pertinent domains. An
SE-SC8. Modest correlations for Cluster 4, Goal-directed Supervision important principle of competence assessments is that competence
(r = .60, and r = .58 p < .001 for Datasets A and B), suggest that, in this should be evaluated by objective methods in ecologically valid con-
instance, the nuances of this competency may not be accurately cap- texts and against clearly specified criterion-based standards
tured by the specific item on the short scale. Item-based analysis is (Gonsalvez & Calvert, 2014; Kaslow et al., 2007). Consequently, sub-
warranted to determine whether a new item specifically crafted to jective evaluations of competence, be it by a supervisee, a peer-super-
capture the essence of the cluster (in lieu of choosing an existing item visor, or by the supervisor themselves, must be complemented by
from the cluster) would improve the item's reliability. Further, the cor- other objective assessments (Gonsalvez & Crowe, 2014; Kaslow
relations for the six items across the two datasets are very similar et al., 2007; O'Donovan & Kavanagh, 2014). Valid objective assess-
(Table 1), demonstrating robustness of the scale's measurements, ments typically involve the development of specific and standardized
good replicability across the two studies, and generalizability across scenarios for each targeted competency, manuals for training, and
individual and group supervision contexts. customized rubrics for assessment that have to be tested for reliability
(see Gonsalvez et al., 2019). Systematic use of such competency
assessments have occurred in medicine through the development and
4.1 | Convergent and divergent validity evaluation of the Objective Structured Clinical Examinations (OSCE;
see Newble, 2004). Similar competence assessments in the area of
The concurrent validity of a new instrument is best demonstrated by supervision are currently lacking, although an initiative assessing com-
high correlations, indicating strong levels of agreement between petence in CBT supervision shows good progress (Milne &
scores obtained by the new instrument and those obtained by an Reiser, 2011; Reiser et al., 2018).
established, gold-standard measure. In the area of supervisor compe- Both subjective evaluations and objective competence assess-
tence, an instrument that can serve as a benchmark is yet to be ments have their strengths and weaknesses. Although the supervisee
established. In the absence of a credible anchor, the current study cor- may lack the credentials, experience, and expertise to make an
related scores of the SE-SC8 with subscales of other established mea- informed judgement about a supervisor's competence, they have
sures such as the SWAI and SSI, predicting that similar constructs direct experience of the supervisor's behaviour over an extended
across the new and established scales would yield high correlations time. In contrast, although an expert's assessment of a supervisor's
and dissimilar constructs would yield lower correlations. Four predic- performance on an objective test may appear both credible and con-
tions concerning Clusters 1, 4, 5, and 6 were formulated, and the vincing, their judgements are reliant on a very limited sample of the
results supported these predictions, indicating good convergent and supervisor's behaviour (often each competency is assessed by a single
divergent validity for the SE-SC8 (see Table 2). Competency-based standardized scenario), often in a simulated context. Consequently,
supervision emphasizes the need for supervisors to be systematic, the validity of their judgements stands or falls on the ability of the
structured, and outcome-driven (Cluster 3) and also to be competent standardized scenarios to elicit supervisory behaviours that will accu-
supervisors and therapists (Cluster 2; APA, 2014). These competen- rately predict real-life competence. Moreover, the development and
cies were not adequately represented in the SWAI or the SSI and were validation of objective competency assessments are costly and labour
not tested in the study. The pattern and magnitude of the correlations intensive and so will take considerable time to establish. In interim
obtained for the SE-SC8 for convergent validity comparisons (r ranges period, best-practice guidelines for competence assessments
GONSALVEZ 459

recommend that evidence for competence be obtained using multiple therapies. In M. Barkham, G. E. Hardy, & J. Mellor-Clark (Eds.), Devel-
methods, multiple tasks, and from multiple sources (Gonsalvez oping and delivering practice-based evidence: A guide for the psychologi-
cal therapist (pp. 175–219). Chichester, UK: Wiley.
et al., 2019; Kaslow et al., 2007).
Brattland, H., Koksvik, J. M., Burkeland, O., Gråwe, R. W., Klöckner, C.,
In effect, although investigation into the extent of agreement Linaker, O. M., … Iversen, V. C. (2018). The effects of routine out-
between subjective and objective evaluations of supervisory compe- come monitoring (ROM) on therapy outcomes in the course of an
tence remains both an important and interesting priority for future implementation process: A randomized clinical trial. Journal of
Counseling Psychology, 65(5), 641–652. https://doi.org/10.1037/
research, it was outside the scope and feasibility of the current study.
cou0000286
Additionally, less than high correlations between subjective and objec-
Corrie, S., & Worrell, M. (2012). The Supervisor Evaluation Scale.
tive measures may not necessarily imply that one/both instruments Unpublished instrument, available from: sarah.corrie@nhs.net
lack/s validity. The supervisee's ratings might be a reliable and valid Efstation, J. F., Patton, M. J., & Kardash, C. M. (1990). Measuring the work-
estimate of their appraisal of supervisory competence, an important ing alliance in counselor supervision. Journal of Counseling Psychology,
37(3), 322–329. https://doi.org/10.1037/0022-0167.37.3.322
consideration in supervision regardless of whether or not this judge-
Ellis, M. V., D'Iuso, N., & Ladany, N. (2008). State of the art in the assess-
ment is confirmed on an objective test. At a critical transition time ment, measurement, and evaluation of clinical supervision and evalua-
towards competency-based training and supervision, there is a need tion of clinical supervision. In A. K. Hess, K. D. Hess, & T. H. Hess
for credible objective and subjective instruments because they fulfil (Eds.), Psychotherapy supervision: Theory, research, and practice
(pp. 473–499). Hoboken, NJ: Wiley.
different purposes and have different applications.
Falender, C. A., & Shafranske, E. P. (2007). Competence in competency-
based supervision practice: Construct and application. Professional Psy-
chology: Research and Practice, 38(3), 232–240. https://doi.org/10.
4.3 | Limitations and future research directions 1037/0735-7028.38.3.232
Falender, C. A., & Shafranske, E. P. (2014). Clinical supervision: The state
of the art. Journal of Clinical Psychology, 70, 1030–1041. https://doi.
Confirmatory empirical validation of the cluster structure of the full org/10.1002/jclp.22124
scale, including the six lower-order clusters by other independent Fouad, N. A., Grus, C. L., Hatcher, R. L., Kaslow, N. J., Hutchings, P. S.,
studies, is required. In that respect, the adequacy of the short scales Madson, M. B., … Crossman, R. E. (2009). Competency benchmarks:
A model for understanding and measuring competence in profes-
in representing important supervisor competencies rests on the repli-
sional psychology across training levels. Training and Education In
cability of the cluster structure outlined for the full scale. A further Professional Psychology, 3(4, Suppl), S5–S26. https://doi.org/10.
limitation of the full SE-SC scale is that it covers most but not all com- 1037/a0015832
petencies listed in current frameworks (e.g., APA, 2014), and this limi- Friedlander, M. L., & Ward, L. G. (1984). Development and validation of
the Supervisory Styles Inventory. Journal of Counseling Psychology, 31
tation extends to the SE-SC8. A revision of the SE-SC addresses this
(4), 541–557. https://doi.org/10.1037/0022-0167.31.4.541
inadequacy and better maps onto the array of supervisor competen-
Gonsalvez, C. J., & Calvert, F. (2014). Competency-based models of super-
cies but is yet unpublished. A 12-item version that includes the eight vision: Principles and applications, promises and challenges. Australian
items of the SE-SC8 and four additional on-trial items is included in Psychologist, 49, 200–208. https://doi.org/10.1111/ap.12055
Appendix A. Gonsalvez, C. J., & Crowe, T. P. (2014). Evaluation of psychology practi-
tioner competence in clinical supervision. American Journal of Psycho-
In summary, the SE-SC8 meets a current and urgent need for
therapy, 68(2), 177–193. https://doi.org/10.1176/appi.psychotherapy.
short scales of supervisor competence and supports important ongo- 2014.68.2.177
ing research on supervision and training and their effects on client Gonsalvez, C. J., Deane, F. P., Terry, J., Nasstasia, Y., & Shires, A. (2019).
outcomes. The results of the current study indicate that the SE-SC8 is Innovations in competence assessment: Design and initial validation of
the Vignette Matching Assessment Tool (VMAT). Training and Educa-
fit for the purpose, although additional research to corroborate these
tion in Professional Psychology, Advance online publication. https://doi.
results is indicated. org/10.1037/tep0000302
Gonsalvez, C. J., Hamid, G., Savage, N. M., & Livni, D. (2017). The supervi-
DATA AVAI LAB ILITY S TATEMENT sion evaluation and supervisory competence scale: Psychometric vali-
dation. Australian Psychologist, 52(2), 94–103. https://doi.org/10.
The data that support the findings of this study are available from the
1111/ap.12269
corresponding author upon reasonable request.
Gonsalvez, C. J., & Milne, D. (2010). Clinical supervisor training in
Australia: A Review of current problems and possible solutions.
ORCID Australian Psychologist, 45, 233–242. https://doi.org/10.1080/
Craig J. Gonsalvez https://orcid.org/0000-0002-3294-6314 00050067.2010.512612
Hatcher, R. L., Fouad, N. A., Grus, C. L., Campbell, L. F.,
McCutcheon, S. R., & Leahy, K. L. (2013). Competency benchmarks:
RE FE R ENC E S Practical steps toward a culture of competence. Training and Education
American Psychological Association. (2014). APA Guidelines for Clinical in Professional Psychology, 7(2), 84–91. https://doi.org/10.1037/
Supervision in Health Service Psychology. Retrieved from http://apa. a0029401
org/about/policy/guidelines-supervision.pdf Kaslow, N. J., Rubin, N., Bebeau, M., Leigh, I., Lichtenberg, J., Nelson, P., …
Barkham, M., Mellor-Clark, J., Connell, J., Evans, R., Evans, C., & Smith, L. (2007). Guiding principles and recommendations for the
Margison, F. (2010). The CORE measures & CORE system: Measuring, assessment of competence. Professional Psychology: Research and Prac-
monitoring, and managing quality evaluation in the psychological tice, 38, 441–451. https://doi.org/10.1037/0735-7028.38.5.441
460 GONSALVEZ

Ladany, N., Mori, Y., & Mehr, K. E. (2013). Effective and ineffective super- Rodolfa, E. R., Bent, R. J., Eisman, E., Nelson, P. D., Rehm, L. R., &
vision. The Counseling Psychologist, 41, 28–47. https://doi.org/10. Ritchie, P. (2005). A cube model for competency development: Impli-
1177/0011000012442648 cations for psychology educators and regulators. Professional Psychol-
Milne, D. L., & Reiser, R. P. (2011). Observing competence in CBT supervi- ogy: Research and Practice, 36(4), 347–354. https://doi.org/10.1037/
sion: A systematic review of the available instruments. The Cognitive 0735-7028.36.4.347
Behaviour Therapist, 4(3), 89–100. https://doi.org/10.1017/ Shimokawa, K., Lambert, M. J., & Smart, D. (2010). Enhancing treatment
S1754470X11000067 outcome of patients at risk of treatment failure: Meta-analytic and
Neuer Colburn, A. A., Grothaus, T., Hays, D. G., & Milliken, T. (2016). A mega-analytic review of a psychotherapy quality assurance system.
Delphi study and initial validation of counselor supervision competen- Journal of Consulting and Clinical Psychology, 78, 298–311. https://doi.
cies. Counselor Education and Supervision, 55, 2–15. https://doi.org/10. org/10.1037/a0019247
1002/ceas.12029 Tangen, J. L., & Borders, D. (2016). The supervisory relationship: A concep-
Newble, D. (2004). Techniques for measuring clinical competence: Objec- tual and psychometric review of measures. Counselor Education and
tive structured clinical examinations. Medical Education, 38(2), Supervision, 55, 159–181. https://doi.org/10.1002/ceas.12043
199–203. https://doi.org/10.1111/j.1365-2923.2004.01755.x Watkins, C. E. (2012). Psychotherapy supervision in the new millennium:
Newman-Taylor, K., Gordon, K., Grist, S., & Olding, C. (2012). Developing Competency-based, evidence-based, particularized, and energized.
supervisory competence: Preliminary data on the impact of CBT Journal of Contemporary Psychotherapy, 42, 193–203. https://doi.org/
supervision training. The Cognitive Behaviour Therapist, 5(4), 83–92. 10.1007/s10879-011-9202-4
https://doi.org/10.1017/s1754470x13000056 Watkins, C. E. Jr., & Milne, D. L. (2014a). Clinical supervision at the inter-
O'Donovan, A., & Kavanagh, D. J. (2014). Measuring competence in super- national crossroads: Current status and future directions. In The Wiley
visees and supervisors. In C. E. Watkins, Jr., & D. L. Milne (Eds.), The international handbook of clinical supervision (pp. 673–696). West Sus-
Wiley international handbook of clinical supervision (pp. 458–467). West sex, England: John Wiley & Sons Ltd.
Sussex, England: John Wiley & Sons Ltd. Watkins, C. E. Jr., & Milne, D. L. (2014b). Defining and understanding clini-
Palomo, M., Beinart, H., & Cooper, M. J. (2010). Development and valida- cal supervision. In The Wiley international handbook of clinical supervi-
tion of the Supervisory Relationship Questionnaire (SRQ) in UK trainee sion (pp. 3–19). West Sussex, England: John Wiley & Sons Ltd.
clinical psychologists. British Journal of Clinical Psychology, 49(2), Wheeler, S., & Barkham, M. (2014). A core evaluation battery for supervi-
131–149. https://doi.org/10.1348/014466509x441033 sion. In C. E. Watkins, Jr., & D. L. Milne (Eds.), The Wiley international
Pilling, S., & Roth, A. D. (2014). The competent clinical supervisor. In handbook of clinical supervision (pp. 365–385). West Sussex, England:
C. E. Watkins, Jr., & D. L. Milne (Eds.), The Wiley international handbook John Wiley & Sons Ltd.
of clinical supervision (pp. 20–37). West Sussex, England: John Wiley & Wheeler, S., & Richards, K. (2007). The impact of clinical supervision on
Sons Ltd. counsellors and therapists, their practice and their clients: A system-
Psychology Board of Australia. (2018). Guidelines for supervisor training atic review of the literature. Counselling and Psychotherapy Research, 7
providers. Retrieved from: https://www.psychologyboard.gov.au/ (1), 54–65. https://doi.org/10.1080/14733140601185274
Standards-and-Guidelines/Codes-Guidelines-Policies.aspx
Rakovshik, S. (2015). The supervisor competence scale: Development and
psychometric properties. Paper presented at the annual conference of
the BABCP, Warwick, UK. How to cite this article: Gonsalvez CJ. A short scale to
Reiser, R. P., Cliffe, T., & Milne, D. L. (2018). An improved competence rat- evaluate supervision and supervisor competence—The
ing scale for CBT Supervision: Short-SAGE. The Cognitive Behaviour SE-SC8. Clin Psychol Psychother. 2021;28:452–461. https://
Therapist, 11, 1–16. https://doi.org/10.1017/S1754470X18000065
doi.org/10.1002/cpp.2510
Rodolfa, E., Greenberg, S., Hunsley, J., Smith-Zoeller, M., Cox, D.,
Sammons, M., … Spivak, H. (2013). A competency model for the
practice of psychology. Training and Education in Professional
Psychology, 7(2), 71–83. https://doi.org/10.1037/a0032415
GONSALVEZ 461

APP E NDIX A. : SUPERVISION EVALUATION AND SUPERVISOR COMPETENCE—SHORT SCALE (SE-SC8)

Supervisee form: Use the following Likert scale to evaluate the supervision you received from your supervisor.

Not at all Strongly disagree Moderately Neutral Very much so Strongly agree
1 2 3 4 5 6 7
Item
no. Items Score
1 Overall, my expectations of supervision were matched or exceeded.
2 Overall, supervision significantly enhanced my competence as a practitioner and professional.
3 The supervisor was approachable, caring, and supportive.
4 The supervisor impressed me as a skilled and effective therapist.
5 Supervision goals and competencies were designed to match my developmental needs.
6 Supervision sessions were structured, and supervision activities were goal-driven.
7 Supervision provided a safe place for emotional ventilation and support.
8 The supervisor helped me gain an understanding of my emotional response in client work (including when appropriate, schema,
and transference-type reactions).
9 The supervision experience has given me a richer and more accurate appraisal of myself as a therapist and professional.
10 The supervisor improved my awareness and analyses of how ethical issues affected client care and professional activities.
11 The supervisor helped me gain a deeper appreciation for the value of using scientific methods and principles to shape professional
practice.
12 The supervisor improved my awareness and analyses of how sociocultural values affected the processes and outcomes of my
professional work.

Note: Research on the SE-SC8 supports the use of the first eight items (Items 1–8). The psychometric properties of the new items (Items 9–12) are yet to
be established but may be used for a more comprehensive evaluation of supervisor competence.
Copyright of Clinical Psychology & Psychotherapy is the property of John Wiley & Sons,
Inc. and its content may not be copied or emailed to multiple sites or posted to a listserv
without the copyright holder's express written permission. However, users may print,
download, or email articles for individual use.