See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/344283972

Critical Thinking Performance Increases in Psychology Undergraduates

Measured Using a Workplace-Recognized Test

Article in Teaching of Psychology · September 2020

DOI: 10.1177/0098628320957981

CITATIONS READS

10 874

1 author:

Mike van Duuren

University of Brighton
15 PUBLICATIONS 570 CITATIONS

SEE PROFILE

All content following this page was uploaded by Mike van Duuren on 21 September 2020.

The user has requested enhancement of the downloaded file.

Science of Teaching and Learning Corner
Teaching of Psychology
1-11
Critical Thinking Performance Increases ª The Author(s) 2020
Article reuse guidelines:
sagepub.com/journals-permissions
in Psychology Undergraduates Measured DOI: 10.1177/0098628320957981
journals.sagepub.com/home/top
Using a Workplace-Recognized Test

Merce Prat-Sala1 and Mike van Duuren1

Abstract
Background: Higher education institutions and universities aim to provide students with a range of transferable skills that enable
them to become more thoughtful and effective employees, citizens, and consumers. One of these skills is critical thinking.
Objective: The aim of the present research was to examine whether taking a psychology degree is concomitant with students’
increase in critical thinking skills when students are not explicitly taught critical thinking. Method: Study 1 utilized a cross-
sectional design and Study 2 a longitudinal design. The Watson and Glaser Critical Thinking Appraisal (WGCTA, UK) was used to
measure critical thinking. Results: For both studies, the overall scores of WGCTA, as well as scores of the subtest of Recognition
of Assumptions, were significantly higher for final-year than for first-year students. Conclusion: From the findings, we conclude
that the levels of critical thinking by final-year psychology students may be enhanced. Teaching implications: We propose that
teaching other aspects of critical thinking such as Evaluation of Arguments and Interpretation, as measured by this test, could be
beneficial in further developing psychology students’ overall critical thinking performance.

Keywords
critical thinking, transferrable skills, psychology undergraduates, WGCTA

There are many skills that are required for the world of work empirical studies have focused on whether the learning,
including information management, oral and written commu- research, and practice that students normally engage in lead
nication, collaboration, critical and analytical thinking, self- students to improve these skills (e.g., Liu, Liu, et al., 2016;
regulation, integrity, and adaptability (e.g., Naufel et al., Pascarella & Terenzini, 2005).
2018). Given the importance of these skills not only in the From a teaching perspective, two issues are relevant with
workplace but also to support almost any significant learning respect to the teaching and assessing of critical thinking: (1)
opportunity in life generally, many higher education institu- teaching: whether critical thinking skills should be taught in the
tions list these skills as part of the learning outcomes for their context of discipline-specific matter and focus on cognitive
students (Roohr et al., 2019). skills relevant to that discipline or whether universities should
There is an argument that among these aforementioned teach critical thinking skills with general vignettes with
skills sets, critical thinking is central for shaping effective glo- nondiscipline-specific problem arguments and statements and
bal workforce roles (Liu et al., 2014). However, in addition to (2) assessment: whether the tests used to assess students’ crit-
that in a world where the opportunity for the consumption of ical thinking should be discipline-specific or whether a generic
information and misinformation is ubiquitous, critical thinking critical thinking test should be used. The ability of students to
becomes a valued skill for students to develop in order to transfer critical thinking skills to the workplace is important
participate as responsible members of their communities (Dam because if critical thinking skills are in the main more subject-
& Volman, 2004; Paul & Elder, 2019). Moreover, it is also a specific, then the transferability of critical thinking skills to
valued skill for differentiating between real and the so-called graduate employment might be more limited. Furthermore,
“fake news” (Musgrove et al., 2018; Paul & Elder, 2019). given that many employers use generic tests (Watson & Glaser,
Although interest in students’ development of critical think-
ing has been longstanding, it is only relatively recent that
1
empirical research has been focused on what factors might University of Winchester, Hampshire, United Kingdom
increase critical thinking skills. The majority of this work has
Corresponding Author:
focused on examining the effectiveness of critical thinking Merce Prat-Sala, University of Winchester, Sparkford Road, Winchester,
interventions (e.g., Barnett & Francis, 2012; Bensley et al., Hampshire SO22 4NR, United Kingdom.
2010; Cloete, 2018; Solon, 2007). However, very few Email: merce.prat-sala@winchester.ac.uk
2 Teaching of Psychology XX(X)

2018), perhaps this is an opportunity for universities to Jones & Morris, 2007; see also Huber & Kuncel’s, 2016, meta-
demonstrate that the rhetoric that promises improved employ- analysis). However, when including participants from a range
ability and transferable skills has actual relevance in relation of disciplines, Roohr et al. (2017) found a significant difference
to some of the criteria (occupational testing) used in the actual in students’ critical thinking scores after 4/5 years in higher
world of work. education, but not after 1, 2, or 3 years, suggesting the need to
Critical thinking is widely assumed to be an important part measure critical thinking skills over a longer period of time.
of psychology graduate, as reflected in the British Psycholo- Using a cross-sectional design and students from different
gical Association (BPS, 2019) Standards for the accreditation American colleges and different disciplines, Liu, Mao, et al.’s
of undergraduate, conversion and integrated masters pro- (2016) findings suggest that scores from final-year students were
grammes in psychology in the UK and the American Psycho- significantly higher than the first years’ scores. Similar findings
logical Association (2013) Guidelines for the Undergraduate were obtained by Mines et al. (1990) who include mathematics
Psychology Major in the United States. Furthermore, and social science students from a U.S. University.
psychology degrees include courses on research methods
which promote rules and values of science such as objectivity,
valid evidence, falsifiability, and operationism; the use of a
Subject-Specific Versus Generic Critical Thinking Tests
wide range of methods and statistical analyses; and appropri- Over the past decades, many definitions of critical thinking
ate conclusions derived from empirical evidence and analysis have been put forward (e.g., Ennis, 1989; Facione, 1990; see
(Bensley, 2009; Yanchar et al., 2008). also Griggs et al., 1998; Hitchcock, 2018; Liu et al., 2014, for
In this article, we present research that examines the extent, reviews of definitions). Following a panel of experts on crit-
if at all, students taking a psychology degree, without any ical thinking, Facione (1990, p. 3) offered the following broad
additional intervention or explicit teaching of critical thinking definition of critical thinking with its emphasis on cognitive
skills, further develop their critical thinking. In what follows, skills: “We understand critical thinking to be purposeful, self-
we review critical thinking literature that focuses on two regulatory judgment which results in interpretation, analysis,
aspects: the time span across assessment of students’ critical evaluation, and inference, as well as explanation of the evi-
thinking skills and the specific tests used to assess critical dential, conceptual, methodological, criteriological, or con-
thinking skills. We then present two studies. textual considerations upon which that judgment is based.”
One of the controversies in the critical thinking literature is
whether the test used to measure critical thinking skills should
Empirical Evidence of Students’ be subject-specific or a generic critical thinking test. Some
Development of Critical Thinking researchers have developed subject-specific critical thinking
in Education tests that focus on the content and cognitive skills related to a
particular discipline (e.g., Bensley et al., 2010; Lawson, 1999;
Time Span Between Testing Lawson et al., 2015; Wentworth & Whitmarsh, 2017). For
The focus of a considerable number of studies has been on the example, Lawson et al. (2015) developed further the Psycho-
effectiveness of interventions to develop students’ critical logical Critical Thinking Exam (PCTE) proposed by Lawson
thinking skills in higher education, using a “pretest to posttest” (1999), which they claim taps into the critical thinking cog-
design where the same participants are tested once at the begin- nitive skills of Evaluation of claims (see also Wentworth &
ning and again at the end of a semester (e.g., Bensley et al., Whithmarsh, 2017, for another psychology specific test with
2010; Burke et al., 2014; Haw, 2011; Lawson, 1999; Lawson similar characteristics to the PCTE). However, one of the
et al., 2015; Stark, 2012). In these studies, the time frame where main limitations of these studies is that the extent to which
critical thinking skills are expected to develop is relatively the skills developed for subject-specific discipline tap into
small ranging between 12 and 16 weeks. One of the main nondiscipline-specific critical thinking is not clear.
limitations of these studies is that they do not provide evidence Alternatively, some studies have focused on students’ level
on whether the gain in critical thinking continues after post- of critical thinking using generic critical thinking tests such as
testing at the end of a semester. Watson and Glaser Critical Thinking Appraisal (WGCTA) or
In contrast, the focus of other studies has been on possible Cornell Critical Thinking Test (CCTT). Tests such as WGCTA
student critical thinking gains over larger time spans than a are called generic because they assess the individual’s critical
single semester (e.g., Bauwens & Gerhard, 1987; Behrens, thinking skills applied to statements that reflect the wide vari-
1996; Cloete, 2018; Liu, Liu, et al., 2016; Liu, Mao, et al., ety of arguments from many everyday life situations such as
2016; Roohr et al., 2019) using either cross-sectional (i.e., data newspapers, magazines, conversations, and media material in
from two different cohorts of students, e.g., first-year vs. final- general (Watson & Glaser, 2018).
year students) or longitudinal designs (i.e., data from the same The findings of the studies using WGCTA or CCTT are
group over time). mixed. Some show that following an intervention, students’
Using a longitudinal design, the authors of several studies on performance in critical thinking skills increases (e.g., Barnett
nursing training programs have found a lack of gain in critical & Francis, 2012; Cloete, 2018; Solon, 2007) while other stud-
thinking skills (e.g., Bauwens & Gerhard, 1987; Behrens, 1996; ies show no gain between pretest and posttest (Renaud &
Prat-Sala and van Duuren 3

Murray, 2008). Furthermore, Cloete (2018) showed that both Table 1. Prequalifications of Participants, by Year Group.
the control and the experimental group improve their perfor-
Year of Study/Prequalifications Year 1 Year 3
mance on the WGCTA test, albeit that the improvement in the
control group was more limited, thereby showing that students A levels 75 82
were able to improve their critical thinking performance with- HND/BTEC a 2 2
out intervention. Burke et al. (2014) also used the WGCTA test Access b 3 1
to assess students’ critical thinking skills. The aim of their IB c 8 1
Combination 3 3
study was to compare directly psychology and philosophy
Other 3 1
undergraduates’ performance (cf. Ortiz, 2007). Findings
a
revealed that only the philosophy students improved their crit- Higher national diploma/ordinary national diploma. b Access to higher educa-
ical thinking between pretesting and posttesting. These authors tion diploma. c International Baccalaureate.
argue that the lack of improvement in critical thinking skills for
the psychology students might have been due to psychology 90 were third-year students (17 male, 73 female) with mean age
students learning mainly inductive reasoning (statistical and 21.30 (range 20–33). Deprivation index3 (DEPI) based on the
methodological), while WGCTA is mainly (but not exclu- zip code of first-year student’s home indicated that 23.9% came
sively) based on deductive reasoning. Nevertheless, what is from deprived areas (Scales 2–4) while 76.1% came from non-
unclear from their study is whether there is any long-term gain deprived areas (Scales 0–1). For third-year students, the index
in critical thinking performance as a result of their degree indicated that 20.2% came from deprived areas (Scales 2–4)
experience, as participants were tested after (only) 15 weeks. while 79.8% came from nondeprived areas (Scales 0–1). We
did not collect information on race/ethnicity of the participants;
Aim of Study 1. The aim of Study 1 was to examine whether however, the student population at the University was 90%–
students taking an accredited psychology degree1 without addi- 92% European White.
tional explicit teaching of critical thinking skills (e.g., Bensley The entry qualifications of participants included A levels,
et al., 2010; Haw, 2011; McLean & Miller, 2010; Stark, 2012) Business and Technology Education Council (BTEC), higher
gain critical thinking skills. We used the critical thinking test national diploma (HND), International Baccalaureate (IB),
WGCTA, UK principally because (1) it is a recognized psycho- and access routes, with a predominance of A levels (see Table
metric test with psychometric properties (Watson & Glaser, 1). A level is a qualification offered by education institutions
2002, 2018) and (2) it assesses the individual’s critical thinking in the UK and is used by universities to assess a student’s
skills applied to statements that reflect the wide variety of eligibility for an undergraduate degree course. International
material encountered in many everyday life situations (Watson students are more likely to study an IB as a form of access to a
& Glaser, 2018). UK University, and mature students take the access route that is
Study 1 compared a group of first-year psychology students a qualification that prepares individuals without a traditional
at the beginning of their degree with a group of third-year qualification for study at University. BTEC and HND are diplo-
students at the end of their degree.2 We had two research ques- mas in further education and vocational qualifications.4 We had
tions: (1) Under the assumption that psychology courses enable full details of students’ entry points for 89.5% Year 1 and 96%
students to develop their critical thinking, it was hypothesized Year 3 students. An independent t test between these two groups
that Year 3 students would show higher scores in the WGCTA showed no significant differences on level points between Year
critical thinking test than the Year 1 students, when at the same 1 (M ¼ 87.01, standard deviation [SD] ¼ 15.33) and Year 3
time controlling for potential confounds such as age, entry (M ¼ 90.49, SD ¼ 13.15); t(169) ¼ 1.57, p ¼ .118, CI [7.739,
qualification, and socioeconomic background and (2) Burke 0.877, Cohen’s d ¼ .24, small effect size difference; Cohen, 1988).
et al. (2014) argue that psychology degrees address mainly
inductive critical thinking. Furthermore, as far as we know,
to date, research has not included WGCTA subtest analysis. Measures
With the second research question, we aimed to explore Students’ critical thinking was measured using the WGCTA,
whether Year 3 performance scores on each of the subtest UK (Watson & Glaser, 2002), a measure of verbal reasoning
differed from Year 1 performance scores. the content of which relates to a wide range of contemporary
sociopolitical and everyday scenarios which are presented in the
Study 1: A Cross-Sectional Study form of problems, statements, arguments, and interpretations.
This 80-item instrument consists of five different subtests, each
consisting of 16 items, measuring the following aspects of crit-
Method ical thinking: inference, recognition of assumptions, deduction,
interpretation, and evaluation of argument (Watson & Glaser,
Participants 2002, 2018; see Appendix for full definitions of each subtest and
Participants were 184 psychology students from a University in example items).
England, UK. Ninety-four participants were first-year students The test as a whole measures a candidate’s critical ability to
(15 male, 79 female) with mean age 19.47 (range 18–44), and correctly identify answers that are correct in an absolute,
4 Teaching of Psychology XX(X)

logical way, as well as those for which only a more probabil- Table 2. Means and Standard Deviations of Total WGCTA and Subt-
istic judgment can be given, given the sufficiency or otherwise est for First- and Third-Year Students.
of evidence provided. Year 1 Year 3
The WGCTA, UK raw scores range from 0 to 80. The data
a
reported below are reported in terms of raw scores adjusted for Year of Study/(Sub)Test Score Type M SD M SD
chance5 (see Wagner & Harvey, 2006) with higher scores indi- b
Overall WGCTA, UK 122.64 51.67 150.36 55.19
cating a greater facility for critical thinking, or if indicated, in Inference 5.30 3.19 6.67 3.02
terms of standardized scores which are required for any com- Recognition of assumptions 5.57 4.92 8.13 5.34
parison between different sample data previously obtained Deduction 4.72 4.41 5.75 3.52
(WGCTA, UK, [Watson & Glaser, 2002]). Interpretation 7.89 3.85 8.80 3.95
Evaluation of arguments 7.17 3.79 7.62 3.96
Procedure Note. WGCTA ¼ Watson and Glaser Critical Thinking Appraisal.
a
Subtest scores adjusted by chance range 0–16. b Scores adjusted by chance
To ensure that testing took place in conditions commensurate range 0–320.
to psychometric testing for job recruitment, the WGCTA, UK
was administered strictly in accordance with the procedure With the Research Question 2, we aimed to explore whether
outlined in the manual (Watson & Glaser, 2002, 2018) by the performance scores on the WGCTA subtests were different
qualified BPS occupational test administrators. One of the test between Year 3 and Year 1 students. Table 2 shows the descrip-
administrators was responsible for the test score conversion. tive statistics (means and SDs) of subscale scores for both
On arrival, participants were seated and were asked to read groups. To analyze whether any of the differences were statis-
the briefing instructions and asked for written consent. The tically significant, further ANCOVAs were carried out, one for
test instructions were then read out with an opportunity for each of the WGCTA subscales, using the subscales scores as
asking questions. Participants were given a maximum of the dependent variable and year group (Year 1 vs. Year 3) as
50 min to complete the test. The test was administered in the fixed factor. Students’ age, scores on entry qualification,
paper-and-pencil form. and DEPI scores were entered as covariate. As can be seen
First-year students completed the WGCTA, UK during the from Table 3, there was a significant difference between Year
first 5 weeks of their first semester at university, which meant 1 and Year 3 on both the measures of Inference and Recogni-
that they had not completed any credits at the university, tion of Assumptions. There was no significant difference
whereas third-year students did so during the last 4 weeks of between the year groups on the other three WGCTA subscales.
their final semester. These third-year students had all com- The covariate age was not significant for any of the subscales.
pleted 285 credits in psychology.6 To encourage participation, The covariate entry scores were significant for the subscale of
first-year students were offered credit participation points Recognition of Assumptions only and the covariate DEPI was
toward their required psychology study participation quota. All significant for the subscale Interpretation only.
final-year students, in recognition of their time commitment,
were offered a small financial incentive (£10.00) to take part.
Study 1 Discussion
The findings of Study 1 support the hypothesis that final-year
Results students evidence a higher level of critical thinking skills than
Concerning Research Question 1, it was predicted that overall students at the beginning of their degree course using an
WGCTA, UK scores would be higher for Year 3 than Year 1 industry-standard critical thinking test. These findings are com-
students. We present descriptive analyses in Table 2. To exam- parable with other research which included students from other
ine whether the performance scores from students in Year 3 disciplines and utilized a cross-sectional design (e.g., Liu, Mao,
were significantly higher than from Year 1 students, we carried et al., 2016; Mines et al., 1990) and builds on previous research
out an analysis of covariance (ANCOVA) with overall that has also used the WGCTA test (e.g., Behrens, 1996; Burke
WGCTA as the dependent variable, year group as the fixed et al., 2014; Cloete, 2018; McLean & Miller, 2010; Mines
factor, and age, A-level score, and DEPI score as covariate. et al., 1990). These findings suggest that participating in a
The main effect of year group was significant, F(1, 149) ¼ psychology degree at university could increase students’ criti-
3.947, p ¼ .049, Z2p ¼ .026, CI [52.943, 0.142]), indicating that cal thinking skills. The findings relating to specific subscale
students in Year 3 had significantly higher scores than students scores also extend previous findings that have focused only on
in Year 1.The covariate of age, F(1, 149) ¼ 0.371, p ¼ .543, the overall WGCTA scores (e.g., Behrens, 1996; Burke et al.,
Z2p ¼ .002, CI [11.173, 5.906], entry qualification, F(1, 149) ¼ 2014; Cloete, 2018; McLean & Miller, 2010; Mines et al.,
3.614, p ¼ .059, Z2p ¼ .024, CI [0.026, 1.367], and levels of 1990) to provide more details on the subscale scores for psy-
deprivation were not significant, F(1, 149) ¼ 0.092, p ¼ .763, chology students. In particular, it shows that only Inference and
Z2p ¼ .001, CI [7.882, 5.788]. Cohen (1988) indicates that the Recognition of Assumptions were significantly different
effect size benchmark for a partial Z2 is defined as small ¼ .01, between the two groups, partly confirming the argument made
medium ¼ .06, or large ¼ .14. by Burke et al. (2014).
Prat-Sala and van Duuren 5

Table 3. Results of the Analysis of Covariance With Year 1 and Year 3 as the Fixed Factor and Age, A-Level Scores, and DEPI Scores as
Covariates.

Main Effect of Year Covariate: Covariate: A-Level Covariate:

Scales Group F (p) Z2p Age F (p) Z2p Scores F (p) Z2p DEPI F (p) Z2p

Inference 4.055 (.046) .026 1.301 (.256) .009 1.659 (.200) .011 0.189 (.665) .001
Recognition of assumption 4.467 (.036) .029 0.480 (.490) .003 4.226 (.042) .028 0.009 (.923) .0001
Deduction 0.006 (.938) .0001 1.252 (.265) .001 0.168 (.683) .001 0.545 (.461) .004
Interpretation 0.439 (.509) .003 0.128 (.722) .001 3.803 (.053) .025 6.119 (.014) .039
Evaluation of arguments 0.559 (.456) .004 0.374 (.542) .003 0.350 (.555) .002 0.076 (.783) .001
Note. The degrees of freedom for each analysis was 1, 149. DEPI ¼ deprivation index.

However, there are some important limitations to the above took part in Study 1, 63 took the WGCTA, UK again when they
study that lead us to treat the results with caution. For this were in Year 3 (10 males and 53 females).
study, we followed a cross-sectional design comparing two
groups in two different stages of their degree. Although, here, Measures and Procedure
we include age, entry qualification scores (A-level scores), and
deprivation information (DEPI scores) as covariates to take The measures and procedures were the same as for Study 1. As
into account possible confounding variables affecting the main these students were now in Year 3 of their study, as alternative
variables, the two groups of students were nevertheless not to course credits as those would not have been of benefit to
matched for other variables that could have affected the results, them at the end of their degree, they received £10.00 for taking
such as for example, motivation, intellectual skills, cognitive the time to take part in the study. The same qualified BPS
ability, academic performance, or critical thinking disposition occupational test administrators were employed for data col-
(cf. Bensley et al., 2010; Facione, 1990; Solon, 2007). lection and score transformation.
This led us to carry out Study 2 using a longitudinal design
where we compare the scores of students in Year 1 with the Results
scores of the same students at the end of their studies, in Year 3.
The longitudinal design allowed for some potentially confound- The first aim of this second study was to examine whether the
ing variables (such as, e.g., entry point and DEPI, as well as students’ scores for Year 1 differed from the scores of the same
intellectual skills, cognitive ability, and critical thinking disposi- students in Year 3. Table 4 shows the descriptive statistics
tion) to be controlled for by including the same participants. (means and SDs) of scores obtained during Years 1 and 3. As
The aims of this second study were twofold: (1) to examine can be seen, the overall WGCTA, UK scores were higher for
whether we could replicate the findings of Study 1 using a long- Year 3 than for Year 1.
itudinal design. Under the assumption that undergoing a psychol- To assess whether these differences were statistically signif-
ogy curriculum enables students to develop their critical thinking, icant, we carried out a repeated measure ANCOVA with age at
it was hypothesized that the performance scores of the students in Year 3 as covariate and total WGCTA scores as repeated mea-
their final year would be higher than the scores they obtained at sures. From the results, we found that there was a significant
the beginning of their study. Furthermore, following the results of difference in the overall critical thinking performance between
Study 1, it was predicted that the scores of the subtests of Recog- students in their first and final year, F(1, 61) ¼ 6.029, p ¼ .017,
nition of Assumptions and Inference would be higher in Year 3 Z2p ¼ .090. The covariate of age was not significant, F(1, 61) ¼
than in Year 1 and (2) in order to be able to provide some possible 3.821, p ¼ .06, Z2p ¼ .059.
recommendation regarding teaching to further enhance students’ In addition, to examine whether the scores of each of the
performance in critical thinking, we examined the contribution of five subtests reliably differed between study years, further
each subtest of the WGCTA to the final overall WGCTA scores. repeated measure ANCOVAs were carried out, with age at
To this aim, we carried out a hierarchical regression analysis. Year 3 as covariate and the scores on the WGCTA subtest as
repeated measures. From the findings it transpired that there
was a significant difference between Year 1 and Year 3 for
Recognition of Assumptions, F(1, 61) ¼ 3.989, p ¼ .050, Z2p
Study 2: A Longitudinal Perspective ¼ .061. The covariate of age was not significant, F(1, 61) ¼
2.600, p ¼ .112, partial Z2p ¼ .041.
There was no significant difference, however, for the
Method remaining subtests of critical thinking included in the
WGCTA: Inference, F(1, 61) ¼ 0.020, p ¼ .888, Z2p ¼ .001;
Participants the covariate age, F(1, 61) ¼ 0.099, p ¼ .754, Z2p ¼ .002;
The same participants who were Year 1 students at the time of Deduction, F ¼ (1, 61) ¼ 1.966, p ¼ 1.66, Z2p ¼ .031; covariate
Study 1 took part in Study 2. From the initial 94 students who age, F(1, 61) ¼ 0.099, p ¼ 1.696, Z2p ¼ .027; Interpretation,
6 Teaching of Psychology XX(X)

Table 4. Means and Standard Deviations of Total WGCTA and Subt- included, F(5, 49) ¼ 57.013, p ¼ .0001, R2Adj ¼ .838, account-
est for First and Third Year. ing for a further 17.8% of the variance. When Recognition of
Year 1 Year 3
Assumptions was entered as independent variable, a significant
model also emerged, F(6, 48) ¼ 115.624, p ¼ .0001, R2Adj ¼
a
Year of Study/Test Score Type M SD M SD .927, accounting for a further 8.9% variance, as was the case
b when Evaluation of Argument was included, F(7, 47) ¼
Overall WGCTA, UK 131.14 54.86 162.75 59.73
Inference 5.52 3.17 7.92 3.47
169.286, p ¼ .0001, R2Adj ¼ .956, accounting for a further
Recognition of assumptions 6.19 5.06 9.24 4.68 2.9% of variance. Finally, when Interpretation was included,
Deduction 5.30 4.83 6.06 4.89 a final significant model emerged, F(8, 46) ¼ 1049.889, p ¼
Interpretation 8.35 3.69 8.95 3.71 .0001, R2Adj ¼ .995, accounting for a further 3.8% of variance.
Evaluation of arguments 7.43 3.58 8.60 3.96
Note. WGCTA ¼ Watson and Glaser Critical Thinking Appraisal.
a
Study 2 Discussion
Subtest scores adjusted by change range 0–16. b Scores adjusted by change
range 0–320. The findings of Study 2 are similar to those of Study 1. Students
in Year 3 reported higher scores on the overall WGCTA scores
and hence overall higher levels of critical thinking skills than
Table 5. Results of the Hierarchical Multiple Regression Analysis. when they were at the beginning of their degree and had yet to
complete any credits at university. The results are also similar
Variable B SE B b t R2 DR2
to those found for Study 1 with respect to the measures of
Step 1 Recognition of Assumptions.
Constant 353.583 159.132 2.22* .049 .007 Furthermore, the findings of Study 2 extend to psychology
Age 11.843 7.758 .215 1.527 students’ previous results, which utilized a longitudinal design
DEPI 0.781 6.286 .017 0.124 with nursing degree students (e.g., Bauwens & Gerhard, 1987;
A level 0.616 0.724 .120 0.851
Behrens, 1996; Jones & Morris, 2007) or students from arts,
Step 2
Deduction 6.934 0.819 .546 8.462*** .685 .660 humanities, and science disciplines (e.g., Roohr et al., 2017).
Inference 8.799 1.173 .508 7.495*** .853 .838 Finally, the results of the hierarchical regression analysis
Recognition of 4.407 0.565 .358 7.797*** .935 .927 provide us with an overview of how much each individual
assumption subtest contributes toward the overall WGCTA score. Looking
Evaluation of 2.888 0.505 .187 5.721*** .962 .956 at the literature, research on critical thinking which employs
arguments the WGCTA as a measure of critical thinking focuses on the
Interpretation 3.644 0.219 .233 16.618*** .995 .994
overall scores without considering its component measures.
Note. DEPI ¼ deprivation index. Whereas overall scores are used by companies and organiza-
*p < .05. ***p < .001. tions that utilize the WGCTA as a tool for selection or career
advancement of employees (Watson & Glaser, 2002, 2018).
From an education perspective, it is important to know what
contributes to the overall WGCTA score in order to provide
F(1, 61) ¼ 0.634, p ¼ .429, Z2p ¼ .010; covariate age, F(1, 61) any recommendation to help further psychology students to
¼ 0.503, p ¼ .481, Z2p ¼ .008; or Evaluation of Arguments, improve their critical thinking. The findings of the hierarchical
F(1, 61) ¼ 1.651, p ¼ .204, Z2p ¼ .021; covariate age, F(1, 61) regression analysis suggest that the component measures that
¼ 1.265, p ¼ .265, Z2p ¼ .020. contribute least to the overall WGCTA are Evaluation of Argu-
The second aim of the Study 2 was to examine the data as a ments and Interpretation. This may suggest that there is value
means to provide possible recommendations regarding teach- in addressing the skills underlying these specific measures
ing to further enhance students’ performance in critical think- more explicitly, from a teaching and instruction point of view
ing. In particular, we focused on how much each WGCTA (see below), in order to advance student critical thinking.
subtest contributed to the overall WGCTA scores when stu-
dents were in their final year of study. To this effect, we carried
out a hierarchical regression analysis with overall score as the General Discussion
dependent variable.7 The aim of the present research was to investigate the extent to
When age, DEPI, and entry qualification scores were which taking a psychology degree is concomitant with stu-
included as independent variables in the first step, a nonsigni- dents’ increase in critical thinking skills where students are not
ficant model emerged, F(3, 51) ¼ 0.871, p ¼ .468, R2Adj ¼ .007 explicitly taught critical thinking but develop critical thinking
(see Table 5). skills through the learning experience traditionally character-
A significant model emerges when the subscale of Deduc- istic of psychology courses.
tion was included as independent variable, F(4, 50) ¼ 27.204, Findings of both the cross-sectional and longitudinal studies
p ¼ .0001, R2Adj ¼ .660, accounting for 66% of the variance, reported above were similar. With respect to the overall
with a further significant model emerging when Inference was WGCTA scores, students in their final year of their degree
Prat-Sala and van Duuren 7

obtained significantly higher WCGTA performance scores for students in order to develop good citizenship (Dam & Vol-
than students at the start of their degree. Overall, the findings man, 2004; Paul & Elder, 2019).
of our studies are consistent with, and extend to psychology, From the results of the hierarchical regression analysis
the claims made by Pascarella and Terenzini (2005) who sug- reported above, we can see that the WGCTA subtest of Evalua-
gest that attending university on the whole improves students’ tion of Arguments contributed a modest 2.9% toward the over-
critical thinking performance (see also Huber & Kuncel, 2016). all WGCTA score. Evaluation of Arguments requires the
Our results also extend previous findings that have included individual to analyze the evidence and arguments put forward
psychology students as participants and a subject-specific crit- in a text (or conversation). To this aim, it is important to be
ical thinking test (e.g., Haw, 2011; Lawson, 1999; Lawson objective and work logically through the arguments and infor-
et al., 2015) to the WGCTA critical thinking test and by asses- mation put forward. There is the need to distinguish between
sing participants increase in critical thinking skills for longer arguments appealing to logic rather than emotion and avoid
than one semester. At the same time, our findings contrast with privileging information which confirms a preferred perspec-
previous longitudinal research with students in nursing pro- tive. According to Lawson et al. (2015), the PCTE taps into
grams which found no increase in critical thinking skills the critical thinking skills of Evaluation of claims. The work of
(e.g., Bauwens & Gerhard, 1987; Behrens, 1996; Mines Lawson et al. shows that over the span of a 15-week semester,
et al., 1990). They also contrast with previous research that senior psychology majors scored significantly higher on the
investigated a performance increase from a specific interven- PCTE than junior psychology majors, senior biology majors,
tion (e.g., Renaud & Murray, 2008; Stark, 2012; Williams senior art majors, and introductory psychology students. Other
et al., 2004). interventions focusing on the increase of argument develop-
With respect to the WGCTA subtests, from both studies, we ment have also found positive results (e.g., Hasnunidah et al.,
found that students’ critical thinking performance increases for 2015; Kuhn & Udell, 2003). Although the long-term impact of
Recognition of Assumptions. Very few studies have looked at these interventions is unknown, teaching psychology under-
WGCTA subtest differences and the few that exist have looked graduates explicitly how to critically evaluate arguments might
at the relationship between WGCTA subtest and grade scores help increase their overall critical thinking skills.
(Gadzella et al., 2002; Steward & Al-Abdulla, 1989). For Similarly, from the results of the hierarchical regression
example, Gadzella et al. (2002) found that only the subscales analysis in Study 2, we can see that the WGCTA subtest of
of Interpretation and Evaluation of Arguments significantly Interpretation contributed 3.8% toward the overall WGCTA
predicted grade point average. score. According to Facione (1990), interpretation means to
understand the meaning or significance of information. Stu-
dents practice the skill of interpretation when they comprehend
Implications of the Results and express the meaning or significance of a wide variety of
Graduates need to demonstrate a range of skills and attributes experiences (Facione, 1990). Teaching students to effectively
to successfully enter the workplace, with better paid jobs fre- interpret text would help students toward developing interpre-
quently going to candidates with better critical thinking mea- tation critical thinking skills. Furthermore, the use of collabora-
sures. There are a very large number of occupations, including tive or peer-learning has been reported to enhance critical
occupations attractive to psychology graduates, where critical thinking (Gokhale, 1995), including interpretation.
thinking is included as a basic skill required for these occupa-
tions (e.g., Cottrell, 2017; O*Net Online, 2019; Liu et al.,
2014), and there is evidence of a significant high positive cor-
Limitations and Future Directions
relation between the WGCTA overall scores and job perfor- One limitation of the studies presented here is that the higher
mance (Pearson-TatentLens, 2016). The higher levels of performance levels of critical thinking at the end of the course
critical thinking skills that psychology students evidence at the could be due to students’ maturation (see, e.g., Huber & Kun-
end of their degree compared to the start of their course may cel, 2016; Pascarella & Terenzini, 2005) and not due to reading
highlight the relative value of taking a psychology degree in a degree in psychology. To statistically control for this, we used
terms of employability. age as a covariate for the analyses for both Studies 1 and 2 and
However, the need for a thorough command of critical included it in the first step of the hierarchical multiple regres-
thinking is not confined to the world of employment, rather it sion analysis. From the results, we concluded that age did not
relates to our ability to make sense of information flow gener- affect the significant difference between the overall critical
ally, where it is important that we are able to distinguish thinking scores of Year 1 and Year 3 students. In this context,
between sound or cogent arguments and the so-called “fake Pascarella and Terenzini (2005) have claimed that when studies
news” information and other forms of misinformation (Mus- control for maturation, university attendance still produces sig-
grove et al., 2018; Paul & Elder, 2019; Pennycook & Rand, nificant gains.
2019). For example, Pennycook and Rand (2019) show that Our studies did not include participants taking a degree
participants’ scores on analytical reasoning were positively other than psychology and hence our results cannot be general-
correlated with the ability to differentiate between “fake news” ized to students taking other disciplines. This raises the issue of
and real news. Furthermore, critical thinking is a valued skill the lack of a control group to compare psychology students’
8 Teaching of Psychology XX(X)

findings with. In other words, whether the findings in our stud- Inference: An inference is a conclusion that a person can draw
ies are related to students taking psychology rather than some from certain observed or supposed facts.
other relevant variable. However, this raises the question of
Example:
what constitutes a relevant control group in this context.
Including students from other disciplines could potentially Statement: “Two hundred school students in their early teens
reveal the relative efficacy, or otherwise, of the experience voluntarily attended a recent weekend student conference in
of a psychology curriculum in this setting. However, one Leeds. At this conference, the topics of race relations and
could argue that including students from different disciplines means of achieving lasting world peace were discussed, since
to compare the relative efficacy of different curricula in this these were problems that the students selected as being most
regard does not provide the best control measure either, in that vital in today’s world.”
a better control group might be similarly aged individuals not Proposed Inference: As a group, the students who attended this
pursuing a degree course. But then, as Huber and Kuncel conference showed a keener interest in broad social problems
(2016) argue, it is difficult to conduct such study comparing than do most other people in their early teens. (PT, because, as
individuals attending or not attending university while at the is common knowledge, most people in their early teens do not
same time controlling for many other potential confounds. show so much serious concern with broad social problems. It
Future research could include students enrolled in different cannot be considered definitely true from the facts given
disciplines to examine the relative efficacy of enhancing stu- because these facts do not tell how much concern other young
dents’ critical thinking in each discipline. teenagers may have. It is also possible that some of the students
The sample size utilized here was comparable to many volunteered to attend mainly because they wanted a weekend
studies that have examined students’ gains in critical thinking outing.) (Answer: TRUE; PROBABLY TRUE; INSUFFI-
utilizing either a cross-sectional or longitudinal design (e.g., CIENT DATA; PROBABLY FALSE; FALSE).
Jones & Morris, 2007; McLean & Miller, 2010; Mines et al.,
Recognition of Assumptions: An assumption is something pre-
1990; Roohr et al., 2017). Nevertheless, taking into consider-
supposed or taken for granted.
ation the large population of students in our higher education
institutions, the sample is relatively small. Furthermore, Example:
issues of sampling are particularly relevant for the longitudi-
Statement: “We need to save time in getting there so we’d
nal study as participants who took the test twice were by and
better go by plane.”
large a much more self-selected group, than the students who
Proposed assumption: Going by plane will take less time than
partook in the study only once at the beginning of their
going by some other means of transportation. (YES, it is
degree. Another limitation is the small number of males in
assumed in the statement that the greater speed of a plane
our sample, which could imply that current findings might be
over the speeds of other means of transportation will enable
restricted to female psychology students only. Finally, the
the group to reach its destination in less time.). (Answer: Yes
participants in our studies came from only one institution with
or No)
the same aims and ethos with respect to students learning and
development. Hence, further research with a large number of Deduction: In this test, each exercise consists of several state-
participants and more institutions is needed to confirm present ments (premises) followed by several suggested conclusions.
findings.
Example:
Statement: “Some holidays are rainy. All rainy days are boring.
Conclusion Therefore:”
From the findings, we conclude that the scores of students Proposed conclusion: Some holidays are not boring. (NO, the
taking a psychology degree were significantly higher in Year conclusion does not follow, even though you may know that
3 than Year 1, and hence, there was an enhancement of their some holidays are very pleasant.) (Answer: Yes or No)
critical thinking performance, even when critical thinking was Interpretation: The task is to judge whether or not each of the
not explicitly taught, as measured with WGCT, an industry- proposed conclusions or generalizations are warranted beyond
standard psychometric test. We suggest that to further increase a reasonable doubt from the information in the statement.
psychology, students’ critical thinking skills instructors might
focus on the development of skills related to the WGCT com- Example:
ponent measures of Evaluation of Arguments and Inference. Statement: “A study of vocabulary growth in children from
eight months to six years old shows that the size of spoken
vocabulary increases from 0 words at age eight months to
Appendix 2,562 words at age six years.”
Description and Examples of WGCTA Subtest Proposed Conclusion: None of the children in this study had
The examples provided here were retrieved from the following learned to talk by the age of six months. (YES, the conclusion
web page http://www.pearsonvue.com/phnro/wg_practice.pdf follows beyond a reasonable doubt since, according to the
Prat-Sala and van Duuren 9

statement, the size of the spoken vocabulary at eight months Higher national diploma is similar to associate’s degree in the
was 0 words.) (Answer: Yes or No). United States.
5. As the participants had a binary choice for answer to four of the
Evaluation of Arguments: In making decisions about impor-
scales (see Appendix), all raw scores were adjusted for a chance
tant questions, it is desirable to be able to distinguish between
(see Watson & Glaser, 2002; Wagner & Harvey, 2006).
arguments that are strong and arguments that are weak, as far as
6. Students on an accredited single honors degree in psychology in the
the question at issue is concerned.
UK take courses in psychology only. This differs from the U.S.
Example: degrees based on the liberal arts philosophy where students take
courses from a range of subjects to get a broad educational founda-
Statement: “Should all young people in the United Kingdom go
tion before specializing in their majors.
on to higher education?”
7. In the first step, we included age of participants, entry qualification
Proposed arguments: Yes; college provides an opportunity for
scores, and deprivation information (deprivation index scores) as
them to wear college scarves. (WEAK, this would be a silly
fixed factors. In the second step, we entered the five subscales of
reason for spending years in college.) (Answer: Strong Argu-
the Watson and Glaser Critical Thinking Appraisal (Year 3) using a
ment - Weak Argument).
stepwise method.
For all the subtest, except for Inference, the candidate is
required to make a binary choice for each item. Though these
subtests differ in the way they measure a respondent’s critical References
thinking they frequently require any subjective or personal-
American Psychological Association. (2013). APA guidelines for the
moral attitude towards the content of a particular item to be
undergraduate psychology major: Version 2.0. Retrieved
separated from its critical evaluation, in addition test takers
September 15, 2019, from http://www.apa.org/ed/precollege/
need to be able to isolate the truth or falsity of a statement from
about/psymajor-guidelines.pdf
whether it is logical or plausible.
Barnett, J. E., & Francis, A. L. (2012). Using higher order thinking
questions to foster critical thinking: A classroom study. Educa-
Declaration of Conflicting Interests
tional Psychology, 32(2), 201–211. https://doi.org/10.1080/01443
The authors declared no potential conflicts of interest with respect to 410.2011.638619
the research, authorship, and/or publication of this article. Bauwens, E. E., & Gerhard, G. G. (1987). The use of the Watson-
Glaser Critical Thinking Appraisal to predict success in a bacca-
Funding laureate nursing program. Journal of Nursing Education, 26(7),
The authors received no financial support for the research, authorship, 278–281.
and/or publication of this article. Behrens, P. J. (1996). The Watson-Glaser Critical Thinking Appraisal
and academic performance of diploma school students. Journal of
Notes Nursing Education, 35(1), 34–36. https://doi.org/10.3928/0148-
1. In the UK, an accredited psychology degree includes the curricu- 4834-196601-10
lum specified by the British Psychological Association, which Bensley, D. A. (2009). Thinking critically about critical thinking
states that students must undertake several courses on research approaches: Comment on Yancher, Slife, and Warne (2008).
methods, quantitative and qualitative analyses, as well as courses Review of General Psychology, 13(3), 275–277. https://doi.org/
on the core area of psychology, such as developmental psychology, 10.1037/a0015654
cognitive psychology, social psychology, historical and conceptual Bensley, D. A., Crowe, D. S., Bernhardt, P., Buckner, C., & Allman,
psychology, and individual differences. A. L. (2010). Teaching and assessing critical thinking skills for
2. In the UK, the majority of degrees, such as psychology, are 3 argument analysis in Psychology. Teaching of Psychology, 37(2),
years. Students in their first year (Year 1) are in their first year 91–96. https://doi.org/10.1080/00986281003626656
at university—the equivalent of U.S. freshman. Students in their The British Psychological Society. (2019). Standards for the accred-
third year (Year 3) are in their final year of their degree—the itation of undergraduate, conversion and integrated Masters pro-
equivalent of U.S. senior. grammes in psychology. Leicester. Retrieved September 12,
3. Higher Education Access Tracker (HEAT) is a nationally utilized 2019, from https://www.bps.org.uk/sites/bps.org.uk/files/Accred
widening participation profiler that analyzes student zip data anal- itation/Undergraduate%20Accreditation%20Handbook%202019
ysis using an algorithm which takes account of a wide range of .pdf
information including that from UK education funding bodies, Burke, B. L., Shears, S. R., Kraus, S., & Roberts-Cady, S. (2014).
parental income, employment, health, education and skills, barriers Critical analysis: A comparison of critical thinking changes in
to housing and services, living environment and housing. HEAT psychology and philosophy classes. Teaching of Psychology,
Scale ranges from 0 to 4 with 4 indicating highest deprivation 41(1), 28–36. https://doi.org/10.1177/0098628313514175
values and 0 the lowest. Cloete, M. (2018). The impact of an integrated assessment on the
4. BTEC diploma is awarded after completion of a program that is critical thinking skills of first-year university students. Accounting
more vocational than academic and is comparable to completion of Education, 27(5), 479–494. https://doi.org/10.1080/09639284.
a vocational senior high school program in the United States. 2018.1501717
10 Teaching of Psychology XX(X)

Cohen, J. (1988). Statistical power analysis for the behavioral Liu, O. L., Liu, H., Roohr, K. C., & McCaffrey, D. F. (2016). Inves-
sciences. Routledge. tigating college learning gain: Exploring a propensity score
Cottrell, S. (2017). Critical thinking skills: Effective analysis, argu- weighting approach. Journal of Educational Measurement, 43(3),
ment and reflection. Palgrave-Macmillan. 352–367. https://doi.org/10.1111/jedm.12112
Dam, G. t., & Volman, M. (2004). Critical thinking as a citizenship Liu, O. L., Mao, L., Frankel, L., & Xu, J. (2016). Assessing critical
competence: Teaching strategies. Learning and Instruction, 14, thinking in higher education: The HIEghtenTM approach and pre-
359–379. https://doi.org/10.1016/j.learninstruc.2004.01.005 liminary validity evidence. Assessment & Evaluation in Higher
Ennis, R. H. (1989). Critical thinking and subject specificity: Clarifi- Education, 41(5), 677–694. https://doi.org/10.1080/02602938.
cation and needed research. Educational Researcher, 18(3), 4–10. 2016.1168358
https://www.jstor.org/stable/1174885 McLean, C. P., & Miller, N. A. (2010). Changes in critical thinking
Facione, P. A. (1990). Critical thinking: A statement of expert con- skills following a course on science and pseudoscience: A quasi-
sensus for purposes of educational assessment and instruction. A experimental study. Teaching of Psychology, 37, 85–90. https://
Delphi Report. Educational Resources Information Center (ERIC). doi.org/10.1080/00986281003626714
Gadzella, B. M., Baloglu, M., & Stephens, R. (2002). Prediction of Mines, R. A., King, P. M., Hood, A. B., & Wood, P. K. (1990). Stages
GPA with educational psychology grades and critical thinking of intellectual development and associated critical thinking skills
scores. Education, 122(3), 618–623. in college students. Journal of College Student Development,
Gokhale, A. A. (1995). Collaborative learning enhances critical think- 31(6), 538–547.
ing. Journal of Technology Education, 7(1). Retrieved February Musgrove, A. T., Powers, R. J., Rebar, L. C., & Musgrove, G. J.
24, 2020, from https://scholar.lib.vt.edu/ejournals/JTE/v7n1/ (2018). Real or fake? Resources for teaching college students
gokhale.jte-v7n1.html how to identify fake news. College & Undergraduate Libraries,
Griggs, R. A., Jackson, S. L., Marek, P., & Christopher, A. N. (1998). 25(3), 243–260. https://doi.org/10.1080/10691316.2018.
Critical thinking in introductory psychology texts and supple- 1480444
ments. Teaching of Psychology, 25(4), 254–266. Naufel, K. Z., Appleby, D. C., Young, J., Van Kirk, J. F., Spencer, S.
Hasnunidah, N., Susilo, H., Irawati, M. H., & Sutomo, H. (2015). M., Rudmann, J., . . . Richmond, A. S. (2018). The skillful psychol-
Argument-driven inquiry with scaffolding as the development stra- ogy student: Prepared for success in the 21st century workplace.
tegies of argumentation and critical thinking skills of students in https://www.apa.org/careers/resources/guides/transferable-skills.
lampung, Indonesia. American Journal of Educational Research, pdf
3(9), 1185–1192. O*Net Online. (2019). The occupational information network.
Haw, J. (2011). Improving psychological critical thinking in Austra- Retrieved September 13, 2019, from https://www.onetonline.org/
lian university students. Australian Journal of Psychology, 63, skills/
150–153. https://doi.org/10.1111/j.1742-9536.2011.00018.x Ortiz, O. C. M. A. (2007). Does philosophy improve critical thinking
Hitchcock, D. (2018). The Stanford encyclopedia of philosophy (Fall skills? [Unpublished master’s thesis]. The University of Mel-
2018 edition). https://plato.stanford.edu/archives/fall2018/entries/ bourne. https://www.reasoninglab.com/wp-content/uploads/2017/
critical-thinking/ 05/Alvarez-Final_Version.pdf
Huber, C. R., & Kuncel, N. R. (2016). Does college teach critical Pascarella, E. T., & Terenzini, P. T. (2005). How college affects stu-
thinking? A meta-analysis. Review of Educational Research, dents: A third decade of research. Jossey-Bass.
82(2), 431–468. https://doi.org/10.3102/0034654315605917 Paul, R., & Elder, L. (2019). The thinker’s guide to conscientious
Jones, J. H., & Morris, L. V. (2007). Evaluation of critical thinking citizen on how to detect media bias and propaganda. Rowman &
skills in an associate degree nursing program. Teaching and Learn- Littlefield.
ing in Nursing, 2, 109–115. https://doi.org/10.1016/j.teln.2007.07. Pearson-TatentLens. (2016). Evaluating the admission process for a
006 London based business school, April 2016. https://www.talentlens.
Kuhn, D., & Udell, W. (2003). The development of argument skills. co.uk/whitepaper/evaluatingadmissionprocess/
Child Development, 74(5), 1245–1260. https://doi.org/10.1111/ Pennycook, G., & Rand, D. G. (2019). Lazy, not biased: Susceptibility
1467-8624.00605 to partisan fake news is better explained by lack of reasoning than
Lawson, T. J. (1999). Assessing psychological critical thinking as by motivated reasoning. Cognition, 188, 39–50. https://doi.org/10.
a learning outcome of a psychology majors. Teaching of Psy- 1016/j.cognition.2018.06.011
chology, 26(3), 207–209. https://doi.org/10.1207/S15328023 Renaud, R. D., & Murray, H. G. (2008). A comparison of subject-
TOP260311 specific and a general measure of critical thinking. Thinking
Lawson, T. J., Jordan-Fleming, M. K., & Bodle, J. H. (2015). Measur- Skills and Creativity, 3(2), 85–93. https://doi.org/10.1016/j.tsc.
ing psychological critical thinking: An update. Teaching of Psy- 2008.03.005
chology, 42(3), 248–253. https://doi.org/10.1177/00986283 Roohr, K., Olivera-Aguilar, M., Ling, G., & Rikoon, S. (2019). A
15587624 multi-level modeling approach to investigating students’ critical
Liu, O. L., Frankel, L., & Roohr, K. C. (2014). Assessing critical thinking at higher education institutions. Assessment & Evaluation
thinking in higher education: Current state and directions for in Higher Education, 44(2), 1–15. https://doi.org/10.1080/0260293
next-generation assessment. ETS Research Report Series, 1, 1–23. 8.2018.1556776
Prat-Sala and van Duuren 11

Roohr, K. C., Liu, H., & Liu, O. L. (2017). Investigating student Watson, G. B., & Glaser, E. M. (2002). Watson-Glaser critical think-
learning gains in college: A longitudinal study. Studies in Higher ing appraisal—UK edition manual. The Psychological Corporation
Education, 42(12), 2284–2300. https://doi.org/10.1080/03075079. Limited.
2016.1143925 Watson, G. B., & Glaser, E. M. (2018). Watson-Glaser™ critical
Solon, T. (2007). Generic critical thinking infusion and course content thinking appraisal: User’s guide and technical manual.
learning in introduction to psychology. Journal of Instructional Pearson.
Psychology, 34(2), 95–109. Wentworth, D. K., & Whitmarsh, L. (2017). Thinking like a psychol-
Stark, E. (2012). Enhancing and assessing critical thinking in a psy- ogist introductory psychology writing assignments: Encouraging
chological research methods course. Teaching of Psychology, critical thinking and resisting plagiarism. Teaching of Psychology,
39(2), 107–112. https://doi.org/10.1177/0098628312437725 44(4) 335–341. https://doi.org/10.1177/0098628317727909
Steward, R., & Al-Abdulla, Y. (1989). An examination of the rela- Williams, R., Oliver, R., & Stockdale, S. (2004). Psychological versus
tionships between critical thinking and academic success on a generic critical thinking as predictors and outcome measures in a
university campus [Research report]. ERIC Report No. large undergraduate human development course. The Journal of
ED318936. http://citeseerx.ist.psu.edu/viewdoc/download? General Education, 53(1), 37–58. https://doi.org/10.1353/jge.
doi¼10.1.1.843.7780&rep¼rep1&type¼pdf 2004.0022
Wagner, T. A., & Harvey, R. J. (2006). Development of a new critical Yanchar, S. C., Slife, B. D., & Warner, R. (2008). Critical thinking as
thinking test using item response theory. Psychological Assess- disciplinary practice. Review of General Psychology, 12(3),
ment, 18(1), 100–105. https://doi.org/10.1037/1040-3590.18.1.100 265–281. https://doi.org/10.1037/1089-2680.12.3.265

View publication stats