Sports Med 2008; 38 (4): 297-316

REVIEW ARTICLE 0112-1642/08/0004-0297/$48.00/0

© 2008 Adis Data Information BV. All rights reserved.

Validity, Reliability and Sensitivity of

Measures of Sporting Performance
Kevin Currell and Asker E. Jeukendrup
School of Sport and Exercise Sciences, University of Birmingham, Edgbaston, Birmingham, UK

Contents
Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297
1. Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298
2. Reliability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302
2.1 Time to Exhaustion versus Time Trials and Constant Duration Protocols . . . . . . . . . . . . . . . . . . . . . 302
2.2 Duration of Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308
2.3 Mode of Exercise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308
2.4 Soccer Running Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308
2.5 Tennis Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309
3. Sensitivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309
4. Factors to Control in a Performance Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 310
4.1 Familiarization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 310
4.2 Verbal Encouragement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312
4.3 Music . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312
4.4 Feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312
4.5 Measurements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313
5. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313

Abstract Performance testing is one of the most common and important measures used
in sports science and physiology. Performance tests allow for a controlled simula-
tion of sports and exercise performance for research or applied science purposes.
There are three factors that contribute to a good performance test: (i) validity;
(ii) reliability; and (iii) sensitivity. A valid protocol is one that resembles the
performance that is being simulated as closely as possible. When investigating
race-type events, the two most common protocols are time to exhaustion and time
trials. Time trials have greater validity than time to exhaustion because they
provide a good physiological simulation of actual performance and correlate with
actual performance. Sports such as soccer are more difficult to simulate. While
shuttle-running protocols such as the Loughborough Intermittent Shuttle Test may
simulate physiology of soccer using time to exhaustion or distance covered, it is
not a valid measure of soccer performance. There is a need to include measures of
skill in such protocols. Reliability is the variation of a protocol. Research has
shown that time-to-exhaustion protocols have a coefficient of variation (CV) of
>10%, whereas time trials are more reliable as they have been shown to have a CV
of <5%. A sensitive protocol is one that is able to detect small, but important,
changes in performance. The difference between finishing first and second in a
sporting event is <1%. Therefore, it is important to be able to detect small changes
with performance protocols. A quantitative value of sensitivity may be accom-
298 Currell & Jeukendrup

plished through the signal : noise ratio, where the signal is the percentage
improvement in performance and the noise is the CV.

The measurements of performance or fatigue are 2. Reliability: the protocol provides a similar result
two of the most important measures in sport science from day to day when no intervention is used.[7]
and physiology research. Much debate exists over 3. Sensitivity: the protocol is able to detect small,
the correct use of these protocols, with some investi- but important, changes in performance.
gators advocating the use of performance protocols The aim of this article is to review the concepts of
such as time trials[1] and others the use of fatigue validity, reliability and sensitivity in relation to the
protocols such as time to exhaustion[2] when investi- measurement of performance in endurance, racquet
gating the efficacy of a treatment. Which protocol is and team sports. In the sections on team and racquet
used depends on the research question. Recently, sports, soccer and tennis have been chosen as being
investigators have endeavoured to investigate more representative of the issues faced in measuring per-
complex sports, such as soccer, which require more formance as they are the most researched sports. The
complicated performance measures. The choice of article is aimed at those who wish to measure per-
which type of protocol to use is dependent upon the formance and understand the issues in choosing the
research question. This article will attempt to dis- correct performance test.
cuss the factors that are important when choosing an
appropriate performance test and then look at how 1. Validity
performance can be measured.
Performance in this context is the result of certain There are three types of validity that can be
physical activities, influenced by endogenous and applied to performance protocols: (i) logical validi-
exogenous variables, measured and related to cer- ty; (ii) criterion validity; and (iii) construct validity.
tain norms. The endogenous and exogenous vari- Logical validity or face validity assesses whether a
ables interact to form the construct of sports per- test measures what it intends to measure, but is very
formance.[3] These variables can be physical, techni- difficult to truly assess.[8] In contrast, criterion valid-
cal, mental and tactical. In sports such as soccer, ity allows for an objective measure of validity.
there are a large number of variables that have a There are two types of criterion validity: concurrent
complex interaction, each with a different level of and predictive.[9] Concurrent validity means that the
impact upon the outcome.[4] In contrast, a 100-m performance protocol is correlated with a criterion
running sprint has fewer variables and a less compli- measure.[9] For example, this could be correlating
cated interaction. laboratory cycling time trial performance with cy-
Performance protocols allow researchers to sim- cling time trial performance in competition. Predic-
ulate sporting performance or aspects thereof in a tive validity involves using a performance protocol
controlled scientific manner. This enables research- to subsequently predict performance.[9] An example
ers to manipulate certain variables to measure their of this type of validity is predicting performance
impact on sporting performance. In order to study from a test of maximal oxygen uptake (V̇O2max) and
the effects of pharmacological, nutritional or train- peak power output (Wmax), with Wmax explaining
ing interventions on performance, or to measure 94% of the variance in 20-km time-trial perform-
changes in performance over a season, or to detect ance and V̇O2max 82% of the variance.[10]
overreaching,[5] performance is often measured in a It can be argued that sports performance is a
controlled scientific environment. construct.[3] Construct validity refers to the degree in
There are three factors that need to be considered which a protocol measures a hypothetical construct,
when deciding which performance protocol should in this case performance. It can be measured by
be used: comparing two different groups of subjects with
1. Validity: the protocol resembles the performance different abilities.[9] For example, if one wished to
that is being simulated as closely as possible.[6] analyse the construct validity of a test of cycling

© 2008 Adis Data Information BV. All rights reserved. Sports Med 2008; 38 (4)
Validity, Reliability and Sensitivity of Measures of Sporting Performance 299

performance, a comparison of the performance of a Whereas a time trial allows for simulation of the
group of professional cyclists could be compared variable intensities seen during performance of
with a group of recreational cyclists. A test with events such as cycling,[13] a time-to-exhaustion test
good construct validity would easily discriminate is performed at a set intensity. Although it has been
between the two groups. shown that some of the physiological responses are
Traditionally, researchers have used a time-to- similar to both steady state and variable intensity
exhaustion protocol to measure endurance capacity. exercise,[14] there do appear to be differences in
This type of protocol is likely to have evolved from skeletal muscle carbohydrate metabolism as a con-
early animal studies where it is impossible to ask the sequence of variable intensity exercise.[15] During
animal to complete a set distance as quickly as 140 minutes of either variable intensity or steady-
possible. However, one can exercise an animal to state cycling at the same average power output (58%
exhaustion at a set intensity. A time-to-exhaustion Wmax), there was a 16% reduction in muscle glyco-
protocol involves the subject exercising at a constant gen use during variable intensity compared with
intensity, expressed either as a percentage of constant-load exercise, although this did not reach
V̇O2max or at a percentage of Wmax, until they can statistical significance. The total plasma glucose
no longer continue. This is usually indicated by a oxidized during the 146 minutes of variable intensi-
predetermined drop in cadence or intensity. It is, ty exercise was significantly greater than during
however, more appropriate to term this protocol a steady-state exercise (99 vs 84 g/140 min).[15] While
‘test of exercise capacity’ and not a ‘test of perform- it has been shown that a time trial provides a good
ance’.[11] A more recent development has been the physiological simulation of performance,[12] the dif-
use of time trials to measure endurance perform- ferences in carbohydrate metabolism between varia-
ance. These involve completing a set distance or ble and constant intensity exercise[15] suggests that a
amount of work as quickly as possible. time trial is more valid than a time-to-exhaustion
For a performance protocol to be a logically valid test, as a time trial simulates the variable intensities
measure of performance, it must appear to measure that can be seen during performance.
the performance in question. Therefore, a time-trial Laursen et al.[16] investigated the relationship of
protocol is a more logically valid simulation of race exercise test variables to Ironman triathlon cycling
events such as cycling or running as it is an actual performance. It was found that when exercising to
event, whereas there is no sporting event that re- exhaustion at the power output at the ventilatory
quires one to exercise at a set intensity until exhaus- threshold, there was no significant correlation with
tion, which is required in a time-to-exhaustion test. cycling performance during an Ironman triathlon.
Further evidence for greater logical validity of Time trials have been shown to correlate well with
time trials compared with time to exhaustion can be actual performance in both cycling[17] and run-
found when examining the physiological responses ning.[18] Palmer et al.[17] found a strong relationship
of the two protocols. While sporting performance is between time to completion for a 40-km cycling
more than just a physiological construct, many of time trial performed on a Kingcycle™ 1 ergometer
the interventions that are researched tend to be phys- compared with time to completion of two outdoor
iological in nature, such as nutritional or training 40-km cycling time trials. A 10-km running time
interventions. Therefore, in order to be a valid simu- trial after a 90-minute preload at 65% V̇O2max was
lation of performance, the protocol should provide found to have a high correlation (r = 0.95) with time
similar physiological responses to actual perform- to complete a 10-km road race. While more research
ance. Foster et al.[12] showed that there were no is needed to establish these relationships, especially
physiological differences in a 5-km time trial during for time to exhaustion, the absence of a relationship
a laboratory trial and actual performance. Subjects between time to exhaustion and performance[16] and
also perceived the laboratory trial to be similar to the presence of a relationship between time trial and
what would occur during actual performance. performance[17,18] suggests time trials are a more

1 The use of trade names is for product identification purposes only and does not imply endorsement.

© 2008 Adis Data Information BV. All rights reserved. Sports Med 2008; 38 (4)
300 Currell & Jeukendrup

valid measure of cycling or running performance cruising, jogging. Motor skill proficiency in specific
than time to exhaustion. skills such as tackling and shooting can also be
Use of a time-to-exhaustion protocol may be quantified.[21,22] Similarly, match analysis can be
useful when investigating the mechanisms by which used in other multiple sprint sports. An advantage of
an intervention affects performance, as the constant using such a technique is that match play itself is
intensity provides a controlled environment in being analysed, so the method is valid.
which to take physiological measurements. The evi- Di Salvo et al.[23] compared the velocity recorded
dence does seem to suggest that time trials are a with a video analysis system over distances ranging
more valid representation of performance in race from 15 to 60 m, some of which included direction
events than time to exhaustion. They have a greater changes with the velocity recorded using electronic
logical validity, have been shown to be a good timing gates. Throughout all distances, the veloci-
simulation of the physiological responses seen ties recorded using the video analysis system corre-
during performance and correlate well with actual lated well with the timing gates, suggesting that
performance. Therefore, when wishing to investi- video analysis systems are a valid measure of physi-
gate whether an intervention affects performance, a cal performance during soccer performance.
time trial would be a more appropriate protocol to More controlled simulations of match play have
use. been developed for use by researchers, especially in
There are two main types of time trial that could soccer and tennis. Drust et al.[24] devised a treadmill-
be used. The target could be set as distance or work. based soccer-specific protocol. The protocol con-
An advantage of using a distance target is that it is a sisted of blocks of treadmill running of speeds vary-
more valid representation of actual performance. On ing between standing still through to 21 km/hour,
most ergometers, this involves the measurement of with the protocol being based upon previous match-
power and converting this into a speed. However, analysis data.[25] The protocol lasted for a total of 46
the power-speed relationship is dependent on many minutes 11 seconds and was designed to allow simu-
factors including the environment, aerodynamics, lation of one-half of a soccer match. It was found
body size and topography,[19] and it can be seen that that the physiological response to the protocol was
indoor time trials are quicker than those outdoors. similar to that found in match play. Other research-
The use of a work target, while not as valid, allows ers have also developed treadmill-based protocols
for a more controlled performance test. that have been shown to simulate the physiological
Sports with simple techniques, such as running responses to soccer match play.[26,27] However, the
and cycling, are easily replicated in a laboratory protocol did not involve any eccentric muscle ac-
situation. However, rowing is an example of sport tions, such as turning direction that occurs during
where technique is incredibly important to perform- match play. It has been shown that there are >900
ance. Elliott et al.[20] showed that rowing on a different discrete movements during a football
Rowperfect™ ergometer required a similar tech- match, which include jogging, cruising, sprinting,
nique to on-water rowing. Therefore, when measur- walking, backwards movement, sideways move-
ing rowing performance in a laboratory, the similari- ment, jumping, skill (e.g. kicking or passing a ball)
ty to on-water rowing must be considered. and standing still.[25]
Some sports, such as soccer and tennis, are far The Loughborough Intermittent Shuttle Test
more complex and, therefore, the measurement of (LIST) is a protocol that was devised using match
performance in such sports becomes much more analysis data and involves intermittent shuttle exer-
difficult. Two main methods have been used: cise with activities such as walking, cruising and
(i) match analysis techniques where performance is sprinting included as well as turning at the end of
measured through video analysis of actual match each 20-m shuttle. The protocol is split into two
play; and (ii) an artificial simulation of the sport in parts: part A involves intermittent activity for five
question. In a sport such as soccer, video analysis blocks of 15 minutes with 3 minutes of rest in
can be used to quantify such variables as distance between. Following this, the subjects were required
covered, type of activities undertaken, e.g. sprinting, to run the 20-m shuttles alternating between speeds

© 2008 Adis Data Information BV. All rights reserved. Sports Med 2008; 38 (4)
Validity, Reliability and Sensitivity of Measures of Sporting Performance 301

equivalent to 55% and 95% V̇O2max until exhaus- VHIR and total sprinting distance. The protocols
tion occurred.[28] The subjects covered approximate- were also able to distinguish between ability levels
ly 12.4 km, and performed between 55 and 60 suggesting they have good construct validity.
turns,[28] which is similar to that seen during match The Yo-Yo intermittent recovery test has also
analysis.[25] Responses in blood lactate, heart rate been shown to be highly correlated with high-inten-
and fluid loss were similar to those during match sity running during a soccer match.[31] This would
play,[28] demonstrating that the protocol is a valid suggest that protocols such as the Yo-Yo intermit-
representation of the physiological responses seen tent recovery test and those of Rampinini et al.[30]
during a soccer match. While the protocol may be a are valid measures of soccer performance. They
valid physiological simulation of soccer match play, may be of particular use when assessing the efficacy
the performance measure is a run to exhaustion, of long-term interventions such as training tech-
which does not represent soccer match play. niques. However, they may not be valid for mea-
Other shuttle-running protocols similar to the surements of acute interventions such as carbohy-
LIST have been shown to correlate poorly with drate ingestion.
various parameters of soccer performance. Bangsbo The protocols of Nicholas et al.[28] and Bangsbo
and Lindquist[29] compared various laboratory and and Lindquist[29] use parameters such as time to
field-based protocols of soccer performance to exhaustion or distance covered as their performance
match play. Two protocols were analysed, the first measure. Few studies have included tests of skill and
was a protocol that consisted of field running of a physiological simulation of soccer match play. Abt
varying running intensities for 46 minutes, followed et al.[32] used a 60-minute intermittent treadmill run-
by a rest period of 14 minutes. The second half ning protocol, with a Zalenka Functional Perform-
consisted of two parts both performed on a tread- ance Test before and after the running trials. The
mill, part A involved running at speeds varying from Zalenka Functional Performance Test is one that
0 to 25 km/hour for 35 minutes followed by a involves a series of shooting and dribbling tasks.
protocol where the subjects alternated between 18 While it may represent skill performance in soccer,
and 8 km/hour until exhaustion occurred. The dis- the study of Abt et al.[32] is representative of many
tance covered during the protocol did not correlate studies that have either not been of representative
with that during match play; however, it did corre- duration of soccer match play,[33,34] or have only
late with high-intensity distance covered during included skill performance tests after simulated
match play. match play.[35,36] The time-course of the decrease of
The second protocol of Bangsbo and Lindquist[29] skill performance has not been investigated. During
was a field-based protocol consisting of forward soccer match play, while these are important indices
running, backwards running and sidestepping, with of performance, the most important measure is how
the speed of the protocol alternating between high- many goals are scored, this would be very difficult
and low-intensity exercise. The duration of the test to measure in a simulation of soccer performance.
was 16.5 minutes. Distance covered during the field The Leuven Tennis Performance Test (LTPT)[37]
test was not correlated with distance or high-intensi- included measures of skill performance into the
ty distance during match play. Therefore, it can be physiological simulation of tennis match play, there-
argued that these protocols do not have a high validi- fore increasing the validity of the protocol. The
ty. LTPT involved 350 strokes divided into five games
Rampinini et al.[30] found that physical perform- of ten rallies. Each rally starts with a first and second
ance in an incremental running test to exhaustion serve followed by five balls projected at the sub-
and a repeated sprint ability test were related to jects, with these balls to be returned to the areas of
match specific physical performance. Peak velocity the courts indicated by lighted signals placed on top
at exhaustion in the incremental test was related to of the projection machine. The projector varied the
total distance covered (TD), high-intensity running direction and the time interval between projections
(HIR) and very high-intensity running (VHIR). Re- and thus created neutral, defensive and offensive
peated sprint ability was highly correlated with situations. Performance was measured via a number

© 2008 Adis Data Information BV. All rights reserved. Sports Med 2008; 38 (4)
302 Currell & Jeukendrup

of variables, digital recording techniques were em- son and Nevill[7] and Hopkins[6]). In the case of this
ployed to record ball velocity, distance to the side- review, the measure of reliability used, where poss-
line, distance to the baseline, a velocity/precision ible, will be the coefficient of variation (CV). This
(VP) index (see equation 1) and velocity/precision/ expresses the standard deviation (SD) of the mea-
error (VPE) index (see equation 2). sure as a percentage of the mean, making it easier to
compare the amount of variation between different
VPindex = 100 * measuredballvelocity2 / baselineballvelocity2
protocols. Other measures that are often used are
log10 (distance to the sideline + 10)
Pearson’s product moment correlation (r) where a
high significant correlation may lead to the conclu-
(Eq. 1)
VPE index = VPindex * % non-error strokes sion that a protocol is reliable. The intraclass corre-
lation coefficient (ICC) is another form of correla-
(Eq. 2) tion that can be used to assess reliability with a value
The protocol was then followed by a 70-m shut- of 0.7–0.8 being questionable, and >0.9 being high
tle-run until exhaustion.[37] The LTPT provides a reliability.[40] The 95% limits of agreement,[41]
good example of how skill performance can be where the variation is expressed in absolute terms in
included into a physiological simulation to make the the form of an interval can be used. Limits of
protocol more valid. agreement can be thought of as an error or tolerance
interval,[7] they allow subjective comparison of the
At present, more work needs to be conducted to
limits of agreement and the mean value of the mea-
produce valid performance protocols for sports such
sure (see table I).
as soccer and tennis. It is important that the skill
element of these protocols is measured. One compo- 2.1 Time to Exhaustion versus Time Trials and
nent that has not been investigated has been percep- Constant Duration Protocols
tion. Research has shown that skilled players can
more successfully identify patterns of play, are able The majority of studies investigating the reliabili-
to anticipate events and are able to focus their vision ty of performance protocols have only investigated
on the most important information available to one type of protocol. Jeukendrup et al.[1] compared
them.[38] Therefore, future research should attempt the reliability of different performance protocols.
to include measures of perception into the perform- They compared a time to exhaustion at 75% Wmax, a
ance protocol. time trial where subjects completed the work equi-
It may be more relevant to call all of the protocols valent to 75% Wmax for 1 hour as fast as possible
indicators of improved performance because in and a constant duration protocol where subjects
sports such as soccer or tennis the aim is to score exercised at 70% Wmax for 45 minutes followed by a
more goals or win more games and sets than the 15-minute period where they had to perform as
opposition. While it may be possible to measure much work as possible. Ten subjects performed six
skill performance or physiological performance, repeats of each protocol. The time-to-exhaustion
these may not necessarily lead to the athlete winning protocol was shown to have the highest variation
the competition. with a CV of 26.6%, compared with CVs of 3.4%
and 3.5% for the time trial and constant duration
2. Reliability protocols, respectively.
The study of Jeukendrup et al.[1] shows that a
Reliability is an important measure as it gives an time to exhaustion has poor reliability compared
indication of the biological and technical variation with time trial and constant duration protocols. As
of the protocol.[39] Measures of reliability can be can be seen in table II, time-to-exhaustion tests,
used to calculate the necessary sample size for a which are completed at an intensity less than
given effect size, therefore reducing the risk of a V̇O2max, typically have a CV of >10%, whereas
type II error occurring.[7] time trials and constant duration protocols typically
There are various ways in which reliability can be have CVs of <5% (see table III and table IV).
expressed (for a more in-depth discussion see Atkin- Therefore, other research that has been conducted

© 2008 Adis Data Information BV. All rights reserved. Sports Med 2008; 38 (4)
Validity, Reliability and Sensitivity of Measures of Sporting Performance 303

Table I. Statistics used in reliability analysis

Statistic Formulae Definition Advantages Disadvantages
Pearsons r ΣZxZy Extent to which two variables Get a significance value Cannot detect changes in
r= are related the mean
N Influenced by inter-
subject variation
ICC F-1 Measures the relative Use for more than one retest Can be sensitive to
ICC =
F + (k - 1) homogeneity within groups in systematic bias
ratio to the total variation Affected by sample
heterogeneity
CV x Expresses error as a Easy to compare between Only accounts for 68% of
CV = × 100
SD percentage of the mean methods the variability
Dimension less
Provides magnitude of what the
day to day differences are
LOA LOA = xdiff ± (1.96 × SDdiff) Reference interval for the test Assume population of test Affected when the
retest differences expected for retest differences measurement error
95% of the population becomes larger as the
magnitude of the test
score increases
(heteroscedasticity)
CV = coefficient of variation; F = F ratio from ANOVA analysis; ICC = intraclass correlation coefficient; k = (observations – tests)/(subjects –
1); LOA = limits of agreement; N = number of pairs of scores; SD = standard deviation; Z = Z scores for each subject on the X and Y
variables; x = mean.

further supports the study of Jeukendrup et al.[1] in power will be low, or virtually zero. Similarly,
the conclusion that time-to-exhaustion tests are less Hinckson and Hopkins[65] used both the critical
reliable than time trials and constant duration proto- power concept and log-log modelling to determine
cols. performance times from an individual’s time to ex-
haustion and found the CV to be as low as 1.0% and
The high variability in time to exhaustion may be
1.3%, respectively, for the two models. This in-
due to the relationship between exercise duration
volved the subjects running to exhaustion in ~2, 4
and power output. It has been suggested that the
and 6 minutes twice, 5 days apart. Using the critical
large variations observed are an artefact of this
power equation (equation 3), the constants of a and
relationship as small changes in a subject’s power
m were estimated using the runs to exhaustion and
output led to large changes in time to exhaustion.[2] were subsequently used to predict performance over
In order to account for the exercise duration and 800, 1500 and 3000 m. However, the critical power
power output relationship, Hopkins et al.[2] convert- equation used to convert the time to exhaustion into
ed CVs for time into CVs for power by differentiat- performance times involves determining two con-
ing the relationship between power and perform- stants, one for aerobic capacity and one for anaerob-
ance. They then calculated a ratio of CV for all tests ic capacity:
compared with a time trial. It was found that a time- D = a + mT
to-exhaustion test had a CV of 0.6 compared with
(Eq. 3)
time trials and constant duration tests, which had a
CV of 1.2. This would suggest that time-to-exhaus- where D = distance, a = anaerobic capacity, m =
tion tests have a lower variation than time trials. maximum aerobic power and T = time.
However, this CV was with regard to mean power Hinckson and Hopkins[65] themselves suggest
over the duration of the trials. In a time-to-exhaus- that the value for anaerobic capacity is unreliable
tion test, a subject rides at a set intensity until (CV = 14%). The log-log model took the form
volitional exhaustion, thus the mean power will be (equation 4):
log(T) = log(D)/(1 + k) − c/1 + k
set before the test is begun, and power output does
not vary. Therefore, the actual variation in mean (Eq. 4)

© 2008 Adis Data Information BV. All rights reserved. Sports Med 2008; 38 (4)
© 2008 Adis Data Information BV. All rights reserved.

304
Table II. Reliability of time-to-exhaustion (TTE) protocols

Study Subjects Training V̇O2max Mode Performance Trials Fam Inter- Mean CV (%) Feedback Enc PM Ergometer
(n, sex, status (mL/kg/ measure trial time to
age) min) time completion
(sec)
Lindsay et 12 M, 25 y NR 65.7 Cycling TTE 150% 3 NR 59 1.7 NR NR NR NR
al.[42] Wmax

Coggan 9, sex NR, Endurance 57.8 Cycling TTE 125% 4 NR 72 h 98 5.3 NR Verbal NR Electrically
and 26 y trained V̇O2max braked
Costill[43]

Graham 4 M, 21 y Trained 62.5 Cycling TTE 120% 4 NR 3–4 d 145 10 NR NR Expired NR

and V̇O2max gas
McLellan[44]

Billat et 8 M, 29 y Subelite 69.0 Running TTE at V̇O2max 3 NR 1 wk 403 17 NR Verbal Expired Treadmill
al.[45] gas

Laursen et 8M, 31 y Endurance 70.4 Running TTE at mean 2 Yes 2–5 d 371 13.2 No No No Treadmill
al.[46] trained 1500 m TT
time

McLellan 15 M, 27 y NR 47.0 Cycling TTE 80% 5 NR 72 h ~1050 17.3 No Verbal and Blood, Electrically
et al.[47] V̇O2max music rectal braked
present probe,
expired
gas, HR

Laursen et 8 M, 31 y Endurance 70.4 Running TTE at mean 2 Yes 2–5 d 1086 15.1 No No No Treadmill
al.[46] trained 5 km TT time

Jeukendrup 10 M, 25 y Well 72 Cycling TTE 75% Wmax 5 Yes 3705 26.6 % Yes No EMB
et al.[1] trained completed
competitive

Krebs and 10 M, age NR Cycling TTE 80% 2 NR 1 wk 5.2–55.9 NR NR NR

Powers[48] NR V̇O2max

Maughan 6 M, 29 y Healthy 53 Cycling TTE 70% 2 Yes 7d 4200 5.6 NR NR Expired NR

et al.[49] V̇O2max gas

Currell & Jeukendrup

Gleser 8 M, 26 y Untrained 40.5 Cycling TTE 75% 3 NR 1 wk 7080 13 NR Rode side Expired Electrically
Sports Med 2008; 38 (4)

and V̇O2max by side, gas, temp, braked

Vogel[50] competed ECG
in teams

CV = coefficient of variation; EMB = electromagnetically braked; Enc = encouragement given; Fam = familiarization; HR = heart rate; M = males; n = no. of subjects; NR = not
reported; PM = physiological measurements performed during the trial; temp = temperature; TT = time trial; V̇O2max = maximal oxygen uptake; Wmax = peak power output.
© 2008 Adis Data Information BV. All rights reserved.

Validity, Reliability and Sensitivity of Measures of Sporting Performance

Table III. Reliability of time trial (TT) protocols

Study Subjects Training status V̇O2max Mode Performance Trials Fam Inter- Mean time CV (%) Feedback Enc PM Ergometer
(n, sex, age) (mL/kg/ measure trial to
min) time completion
(sec)
Hickey et 8 M, 22 y Well trained 62.2 Cycling TT work output 4 NR 72 h 33 2.43 Work NR V̇O2 Isokinetic
al.[51] 14 kJ completed

Hodges et 8 M, 20 y Well trained 68.4 Running 1500 m TT 2 Yes 1 wk 272 0.82 No No No Outdoor
al.[52] outdoors track

Laursen et 8 M, 31 y Endurance 70.4 Running 1500 m TT 2 Yes 2–5 d 316 3.3 No No No Treadmill
al.[46] trained

Jensen and 7 M, 23 y Trained Cycling 5 km TT 2 NR 1 wk 383 2.3 NR NR V̇O2, EMB

Johansen[53] blood

Hickey et 8 M, 22 y Well trained 62.2 Cycling TT work output 4 NR 72 h 721 0.95 Work NR V̇O2 Isokinetic
al.[51] 200 kJ completed

Laursen et 8 M, 31 y Endurance 70.4 Running 5 km TT 2 Yes 2–5 d 1184 2.0 No No No Treadmill

al.[46] trained

Palmer et 10 M, 24 y Highly trained Cycling 20 km TT 3 Yes 72 h 1620 1.11 Distance NR No Kingcycle™

al.[17]

Nicholson 30 M, 30 y 55.1 Running 10 km TT 2 1 wk 3.7 Treadmill

and indoor track
Sleivert[54]

Currell et 7 M, 27 y Well trained 66.0 Cycling TT in a 3 Yes 1 wk 2546 3.8 No No No EMB

al.[55] glycogen-
depleted state

Russell et 8 M and F, Endurance 55 Running 90 min preload 2 Yes 3–5 wk 2740 1 Distance, Movie/ No Treadmill
al.[18] 35 y trained 65% V̇O2max + time music,
10 km TT prizes
awarded
Sports Med 2008; 38 (4)

Tan and 8 M, 22 y Competitive 64.8 Cycling 36 km outdoor 3 No 1 wk 2.9 No No No Outdoor

Aziz[56] TT

Continued next page

305
© 2008 Adis Data Information BV. All rights reserved.

306
Table III. Contd
Study Subjects Training status V̇O2max Mode Performance Trials Fam Inter- Mean time CV (%) Feedback Enc PM Ergometer
(n, sex, age) (mL/kg/ measure trial to
min) time completion
(sec)
Smith et 8 M, 31 y Non-elite 70.4 Cycling 40 km TT 3 NR 3–10 d 3261 0.7 HR, time, NR V̇O2 Kingcyle™
al.[19] endurance indoors distance and
trained blood
samples

Palmer et 10 M, 24 y Highly trained Cycling 40 km TT 3 Yes 72 h 3330 0.97 Distance NR No Kingcycle™

al.[17]

Lindsay et 12 M, 25 y Competitive 65.7 Cycling 40 km TT 3 No 3424 0.89 Distance NR Kingcycle™

al.[42]

Laursen et 43 M, 25 y Trained 64.8 Cycling 40 km TT 3 NR 1 wk 3436 0.9 No NR No Cyclosimula

al.[57]

Smith et 8 M, 31 y Non-elite 70.4 Cycling 40 km TT 3 NR 3–10 d 3449 1.1 HR and NR No Own bike
al.[19] endurance outdoors distance + SRM™
trained

Jeukendrup 10 M, 25 y Well trained 72.9 Cycling 1 h TT 5 Yes 3731 3.4 Cadence Yes No EMB
et al.[1] competitive

Jensen and 7 M, 23 y Trained Cycling 50 km TT 2 NR 1 wk 4496 4.2 Distance NR NR Stationary

Johansen[53] magnetic
bike

Widrick et 8 M, 26 y Endurance 58.4 Cycling Glycogen NR 7030 3.5–5.7 Distance Verbal V̇O2 Isokinetic
al.[58] trained depleted + 70 and
km TT financial

Hickey et 8 M, 22 y Well trained 62.2 Cycling TT work output 4 Yes 72 h 6307 1.01 Work NR V̇O2 Isokinetic
al.[51] 1600 kJ completed

Schabort et 8 M, 26 y Competitive 64.8 Cycling 100 km TT + 4 3 Yes 5–7 d 9094 1.7 HR and NR No Kingcycle™
al.[59] endurance × 1 km and 4 × distance

Currell & Jeukendrup

Sports Med 2008; 38 (4)

trained 4 km sprints

Madsen et Cycling 100 km TT 3.5

al.[60]

CV = coefficient of variation; F = females; EMB = electromagnetically braked; Enc = encouragement given; Fam = familiarization; HR = heart rate; M = males; n = no. of subjects;
NR = not reported; PM = physiological measurements performed during the trial; V̇O2 = oxygen uptake; V̇O2max = maximal oxygen uptake.
© 2008 Adis Data Information BV. All rights reserved.

Validity, Reliability and Sensitivity of Measures of Sporting Performance

Table IV. Reliability of constant duration tests of performance

Study Subjects Training status V̇O2max Mode Performance Trials Fam Inter- CV (%) Feedback Enc PM Ergometer
(n, sex, age) (mL/kg/ measure trial
min) time

Doherty et 10 M, 32 y University Cycling 2 min at Wmax 2 Yes 3.6 No NR No Kingcycle™

[61]
al. sportsmen plus 1 min all-

out

Distance 3.0

Paton and 11 M, 28 y Well trained Cycling 5 min all-out 3 Yes 3–8 d 2.2 Time NR No Kingcycle™

Hopkins[62]

11 M, 28 y Well trained Cycling 5 min all-out 3 Yes 3–8 d 1.5 Time NR No Powertap™

11 M, 28 y Well trained Cycling 5 min all-out 3 Yes 3–8 d 1.6 Time NR No SRM™

Jeukendrup 10 M, 25 y Well trained 72.9 Cycling Preload 45 min 5 Yes 3.5 Time Yes No EMB

et al.[1] competitive 70% Wmax 15

min as far as

possible

Bishop[63] 20 F, 28 y Cyclists and 47.4 Cycling Highest PO for 2 NR 1 wk 2.7 Cadence, power NR HR and Wind braked

triathletes 1h output, HR and RPE cycle ergometer

time

Schabort et 8 M, 27 y Endurance 66 Running Distance in 60 3 NR 7–10 d 2.7 Time and speed NR No Treadmill

al.[64] trained min

Sports Med 2008; 38 (4)

CV = coefficient of variation; EMB = electromagnetically braked; Enc = encouragement given; Fam = familiarization; HR = heart rate; M = males; n = no. of subjects; NR = not

reported; PM = physiological measurements performed during the trial; PO = power output; RPE = rating of perceived exertion; V̇O2max = maximal oxygen uptake; Wmax = peak

power output.

307
308 Currell & Jeukendrup

where D = distance, T = time and k and c are slightly higher CV over 500 m (2.8%) compared
constants, the constant in this model were shown to with 2000 m (1.3%).[67]
have a CV between 11–16%. There are a number of reasons as to why longer
It is difficult to see how calculating a perform- duration protocols have a higher variation than those
ance time based on a number of assumptions and of shorter duration. With longer duration protocols
repeat measurements is superior to the measurement the reliability of power is determined by the reliabil-
of performance itself.[66] The critical power concept ity of the glycolytic and aerobic systems.[2] These, in
is only applicable to high-intensity exercise, which turn, can be greatly influenced by changes in train-
would exclude its use in events that are of greater ing and nutrition. It can be seen that if subjects are
duration than ~10 minutes. undergoing a period of hard training that perform-
There have been reports of CVs of <10% in the ance can be decreased.[5] Similarly, nutrition has
literature for time to exhaustion. Maughan et al.[49] been shown to influence endurance performance.[68]
showing a CV of 5.6% using an intensity eliciting an Indeed, it has been suggested that muscle glycogen
oxygen uptake of 70% V̇O2max. This smaller varia- concentrations play a role in determining pacing
tion may have been due to the subjects receiving strategy during a time trial.[69] During longer dura-
information with regards to their performance, thus tion protocols, motivation may also play a larger
allowing them to know when they had reached the role.[2]
same point in time on each trial.
2.3 Mode of Exercise

2.2 Duration of Protocol The mode of exercise does not appear to influ-
ence the reliability of performance protocols. Time
While time-to-exhaustion protocols tend to have trials in cycling,[19,51,56,58] running[52,54] and row-
a CV >10%, when they are completed at very high ing[70] have all been shown to have CVs of <5%.
intensities the CV is reduced. Billat et al.[45] investi- Similar to time trials, constant duration protocols
gated the reliability of a time-to-exhaustion protocol have been shown to be equally as reliable, indepen-
at an intensity equivalent to 100% V̇O2max and dent of mode of exercise. Schabort et al.[64] showed
reported a CV of 17%. Protocols using an intensity that a 1-hour running protocol was a reliable mea-
of 125% V̇O2max appear to have a CV of sure of performance with a CV of 2.7%. A similar
5–10%.[43,44] When the exercise intensity was set at protocol to assess cycling performance was found to
150% Wmax, the CV for time to exhaustion was have a CV of 2.7%.[63]
1.7%.[42] Hopkins et al.[2] concluded that protocols
of approximately 60 seconds produced less variation 2.4 Soccer Running Performance
than protocols of longer or shorter duration. It has been suggested that the workrate in a
In contrast to the reduced CV for time to exhaus- soccer match can be highly influenced by match
tion with increased intensity, time trials have similar tactics of both teams and to psychological factors.[71]
CV, independent of intensity. Laursen et al.[57] Indeed, it is apparent that the activity profile of a
analysed the reliability of a 40-km time trial, using team is highly correlated with that of the opposi-
43 well trained cyclists, and found a CV of 0.9% for tion.[72] The distance covered in the second half of a
time to completion. Studies with smaller sample soccer match is also dependent upon how much
sizes have still produced CVs of around 0.9% for a work is completed in the first half of a match.[72]
40-km time trial.[17,42] There also seems to be differences in the activity
Shorter distances and, therefore, higher intensi- profile between positions.[73] These uncontrollable
ties have been shown to be reliable with CVs for factors may make it difficult to find differences in
20-km cycling being 1.11%[17] and for 5-km cycling performance due to an intervention. Indeed, match
2.3%.[53] Similarly, longer duration time trials such to match variation of total distance covered and
as 100-km have been shown to have a low CV.[59,60] high-intensity running has been shown to be 15%
Rowing time trials have been shown to have a and 47%, respectively.[74] This would make it very

© 2008 Adis Data Information BV. All rights reserved. Sports Med 2008; 38 (4)
Validity, Reliability and Sensitivity of Measures of Sporting Performance 309

difficult to detect any worthwhile changes due to an ty.[40] Percell[78] produced a protocol to measure
intervention. tennis groundstroke performance. The protocol in-
Little research has been conducted investigating volved hitting ten forehand followed by ten back-
the reliability of soccer-specific running protocols. hand strokes returning the ball from a projector. The
During the LIST, the time it took for the subjects to subjects were required to place the ball as close to
reach exhaustion was ~6 minutes, with the 95% the centre of the baseline as possible, with various
limits of agreements being –3.19–2.16 minutes. The points being given the further the ball from the
95% limits of agreement for 20m sprint time record- baseline. The protocol was found to have r = 0.84,
ed throughout the protocol were –0.14–0.12 seconds which was found to be significant; however, as
with the mean time being ~2.42 seconds.[28] The discussed earlier in this section, this may not be the
authors concluded that this was a reliable measure of best measure of reliability.[7]
soccer performance. The LTPT investigated tennis performance with
Krustrup et al.[75] showed that the Yo-Yo inter- the performance measures producing varying levels
mittent recovery test had a CV of 9.8%. The proto- of reliability. Ball velocity appeared to be the most
col involves a run to exhaustion, which can explain reliable measure as it produced an ICC of 0.70–0.91
the slightly high CV. A similar protocol assessing for the different shots and situations, providing reli-
repeated sprint ability in soccer players has been ability from moderate to high.[40] Similarly, the VPE
shown to have a CV of 0.8% for mean time to index produced an ICC of 0.65–0.90. The measures
completion, and 1.3% for best sprint time. of percentage error, distance to sideline, distance to
Whereas the LIST attempts to simulate the physi- baseline and VP index tended to produce a lower
ological responses to soccer, the Loughborough ICC ranging from 0.15 to 0.81, indicating poor to
Soccer Passing Test (LSPT) and the Loughborough moderate reliability.[37] While including tests of skill
Soccer Dribbling Test (LSDT) are sport-specific into a physiological simulation may increase the
tests of skill. The LSPT involved passing 16 foot- validity, the skill tests need to be reliable, which is
balls into four marked areas, with the time to com- not the case in the LTPT.[37] More investigations
plete the task recorded. The LSDT involved drib- need to be conducted looking into the incorporation
bling a ball between a line of six cones 3 m apart of skill tests into physiological simulations of sports
with the sum of ten trials being the performance such as soccer and tennis, with a particular emphasis
measure. The LSPT produced 95% limits of agree- on the reliability and validity of such protocols.
ment of 0.08–6.43 seconds with a mean score of
148.26 seconds, the 95% limits of agreement for the 3. Sensitivity
LSDT were 0.03–6.14 seconds with a mean of 55.10 When choosing a protocol, it is important that the
seconds. It was concluded that these skill measures smallest worthwhile effect can be detected. Analysis
are reliable indicators of soccer performance.[76] of the finishing times for the 2004 Athens Olympics
While it has been shown that physiological[28] cycling time trial shows that the difference between
and skill[76] simulations of soccer are reliable, more first and second was 0.52%, showing that small
research needs to be conducted in this area. Re- improvements in performance can lead to changes in
search needs to be conducted into skill performance the outcome of a race. Hopkins et al.[79] used mathe-
throughout a physiological simulation, and to test matical models of finishing times of the 1997 Inter-
how reliable this is. national Amateur Athletics Federation series to look
at the smallest worthwhile increase in performance.
2.5 Tennis Performance It was concluded that, in order to increase the chance
of winning by 10%, the intervention must increase
The Avery-Richardson Tennis Serve Test in- performance by 0.3 times the athletes’ CV between
volves 20 serves into designated areas with values events. For this to increase to 20%, the smallest
for accuracy and speed being measured. The relia- worthwhile enhancement is 0.7 × CV.
bility of the measure produced ICC between 0.64 In order to determine the smallest worthwhile
and 0.81,[77] suggesting poor to moderate reliabili- effect for an improvement in performance to be

© 2008 Adis Data Information BV. All rights reserved. Sports Med 2008; 38 (4)
310 Currell & Jeukendrup

seen, the variation in sporting performance needs to tween the groups; however, cyclists had a signifi-
be determined. Paton and Hopkins[80] calculated that cantly faster 40-km time trial time. The reason for
for cycling events, such as the 1-km sprint, the these results may be due to the time to exhaustion
variation in performance is ~0.5% in elite male lasting only around 4 minutes compared with the 1
cyclists. As the distance of the event increases, so hour needed to complete a 40-km time trial. This
too does the variation in performance, and with time may suggest that time trials are more sensitive than
trials of >1 hour the variation becomes greater than time-to-exhaustion protocols; however, further
~1%. It appears that in running events, the within- work needs to be completed looking at the effect of
subject CV ranges from 1.5% to 9.2%, depending on the same intervention on different protocols of a
sex, distance run and experience.[81] similar duration.
Jeukendrup and Martin[82] used a mathematical Validity, reliability and sensitivity are not three
model to look at the improvements that different separate factors; they all interact with each other. A
interventions can have upon cycling time trial per- protocol can be reliable, but not valid, whereas a
formance. It was suggested that even the largest valid protocol must be reliable.[7] The relationship
training effects may improve time trial performance between reliability and validity is such that a relia-
in elite cyclists by 2%, with nutritional interventions bility correlation coefficient is the square of the
producing smaller effects. Therefore, it is important validity correlation coefficient.[9]
to be able to detect these small improvements that It is important to understand that the magnitude
interventions can have. of any effect seen in a test of performance in a
It may be useful to have a quantitative measure of laboratory may not ‘transfer’ directly to an actual
sensitivity. One way to express this would be performance situation. For example, Jeukendrup et
through a signal to noise ratio or a sensitivity index. al.[84] showed an increase in time to complete a
This could be achieved by the means of calculating laboratory-based 1-hour time trial of 2.3% due to
the effect size by the ratio of the change in perform- carbohydrate feeding; however, would the same cy-
ance and the within-subject CV. In order to calculate clists have had the same improvement in a real
this, it is important to take an intervention that is competition? The relationship between the size of
known to improve performance. In this review, car- the effect seen in laboratory situations and the size
bohydrate feeding was used as an intervention that is of the effect in the real world is as yet unknown, but
known to improve performance.[83] From the data is an important relationship to understand.
presented in several papers, the sensitivity index
was calculated and presented in table V. The higher 4. Factors to Control in a
the number, the greater the sensitivity of the proto- Performance Test
col used; however, there does tend to be a publica-
tion bias towards positive results so these may be 4.1 Familiarization
‘maximal’ values.
Another possible method to investigate the sensi- There are a number of factors that need to be
tivity of a protocol would be to compare the re- considered in a performance protocol. First, subjects
sponses of different protocols to different interven- should be familiarized with the protocol being used.
tions, few studies have investigated this. Laursen When the trials have been included in a reliability
et al.[90] looked at the effect of a 4-week interval analysis the CV between the first two trials is
training programme on cycling performance. It was 1.3-fold greater than between following trials.[2]
found that there was a significant improvement in Jeukendrup et al.[1] did not show any learning effect
40-km time trial time as a consequence of the train- over six trials for a time trial lasting approximately
ing; however, time to exhaustion at Wmax did not 40 minutes. This may have been because subjects
show any differences pre- and post-training. Subse- were trained cyclists who were familiar with the
quently, Laursen et al.[16] compared the cycling per- laboratory testing procedures and regularly compet-
formance of cyclists and triathletes. No differences ed in cycling time trials. However, a study by Laur-
could be found in time to exhaustion at Wmax be- sen et al.[57] showed a learning effect for 40-km time

© 2008 Adis Data Information BV. All rights reserved. Sports Med 2008; 38 (4)
© 2008 Adis Data Information BV. All rights reserved.

Validity, Reliability and Sensitivity of Measures of Sporting Performance

Table V. Sensitivity of time trials (TT) and time-to-exhaustion (TTE) protocols

Study n V̇O2max Performance Feeding schedule Fasted Effect Sensitivity Encouragement Feedback Physiological
(mL/kg/ protocol compared with index Measures
min) placebo (min)
Time trials

Jeukendrup et 19 72.9 TT with work done Isostar fed at 70 g/h 1 h 58 (CHO) vs 4.61 No Work
al.[84] being equal to 75% 60 (P) performed and
Wmax for 1 h % work
performed

Davis et al.[85] 19 63.8 2 × TT. 1st to 10800 Glucose at 18 g/h 10 h 159 vs 163 0.58 Money, in No
revs, 2nd to 2700 groups
revs

Time to exhaustion

Carter et al.[86] 7 59.5 TTE at 60% V̇O2max Maltodextrin at 70 g/ Y 145 (CHO) vs 2.96 NR NR Blood, expired
h 123 (P) gas, RPE,
rectal temp

TTE at 73% V̇O2max Maltodextrin at 90 g/ Y 60.6 (CHO) vs 2.45

h 50.8 (P)

Chryssanthopoulos 9 67.0 TTE at 70% V̇O2max Glucose 75 g 30 Y 133 (CHO) vs 0.67 Verbal NR Blood, expired
et al.[87] running min prior to exercise 121 (P) gas, RPE

Wilber and 10 64.9 TTE at 80% V̇O2max Glucose and Y 115 (CHO) vs 1.44 NR No Blood, expired
Moffatt[88] sucrose, 17.5 g 5 92 (P) gas
min prior to exercise
and 32 g/h
thereafter

Coggan and 6 65.0 TTE at 70% V̇O2max 210g Glucose N 205 (CHO) vs 3.6 Verbal No RPE, expired
Coyle[89] polymer and 169 (P) gas, blood
Sports Med 2008; 38 (4)

sucrose solution at
135 min

CHO = carbohydrate group; n = no. of subjects; NR = not reported; P = placebo group; revs = revolutions; RPE = rating of perceived exertion; temp = temperature; V̇O2max =
maximal oxygen uptake; Wmax = peak power output; Y = yes.

311
312 Currell & Jeukendrup

trials. It was seen that when the first trial was 3. There is a response to the rhythmical component
compared with the third trial the CV was 2.9%, of music.[95]
compared with 0.9% when the second trial was During exercise for 15 minutes at 70% V̇O2max,
compared to the third trial. Therefore, it can be seen listening to music caused a significant decrease in
that subjects should be familiarized with the per- heart rate, systolic blood pressure, plasma lactate
formance protocol by at least one trial before mea- concentrations and RPE when compared with a no-
surement commences. music trial.[96] While Boutcher and Trenske[97] did
not find any physiological effect of music during
4.2 Verbal Encouragement exercise, they did find a reduction in RPE. As music
appears to have a performance[93] and physiological
In many investigations, some form of encourage- effect,[96] then music may prove to be a confounding
ment was given during the performance trials. Fre- factor when measuring performance.
quently used forms of encouragement are verbal and
music. Investigations have shown that during re- 4.4 Feedback
peated maximal contractions the force produced is Feedback given to the subjects may also influ-
greater when verbal encouragement is given com- ence performance. Giving feedback may alter the
pared with when none is given.[91] The use of verbal athletes perception of the event. Nikolopoulos et
encouragement could introduce a source of variation al.[98] investigated the effect of changing the per-
as the encouragement may differ between each trial ceived distance while keeping the actual distance
and in the subjects reaction to the encouragement. constant upon pacing strategy. The subjects per-
Hulleman et al.[92] showed that giving extrinsic mo- formed seven trials, in the first five they were told
tivation in the form of a financial reward did not that the distance they were cycling was 40 km,
significantly affect 1500-m cycling performance. whereas in fact they performed three 40-km rides,
one of which was a familiarization ride, one an
4.3 Music experimental ride, and one a ride with feedback, one
34-km ride and one 46-km ride. The final two rides
Music is used in studies as a motivational tool.[47] were 34- and 46-km rides where the subjects knew
However, because of personal preferences, this can- what the distance was. Power output was similar for
not be controlled for across different individuals. the 34-, 40- and 46-km rides when the subjects were
Atkinson et al.[93] studied the effect of high-tempo told all three rides were 40 km. In the known 46-km
music on cycling time-trial performance and found a condition, the power output was 13 W lower than in
significant decrease (2%) in the time taken to com- the unknown condition. These results suggest that it
plete a 10-km cycling time trial due to the music. is perceived as opposed to actual distance that deter-
This performance improvement was explained by an mines power output during cycling. It has been
increase in speed over the first 3 km, higher heart suggested that changes in power output during a
rates and ratings of perceived exertion (RPE) were cycle time trial are based on perceived effort;[99]
seen during the music trial, due to the higher exer- therefore, any feedback that is given may affect the
cise intensity. However, music has been found to perceived effort of the exercise and, subsequently,
have no effect upon Wingate anaerobic test perform- performance.
ance, suggesting that during shorter-duration events The sources of information that are available
music does not affect performance.[94] have been shown to affect RPE. When subjects
There are three hypotheses as to why music may exercised for 15 minutes at both 50% and 80%
affect performance during exercise: V̇O2max with no visual or audible information, RPE
1. Listening to music diverts the attention of the was significantly greater than when audible or visual
performer from the feelings of fatigue during exer- information alone was given, music also was shown
cise. to reduce RPE.[100] Therefore, it is important that
2. Music acts as a stimulant or sedative during researchers are very careful with giving subjects
exercise. clues about their performance during a performance

© 2008 Adis Data Information BV. All rights reserved. Sports Med 2008; 38 (4)
Validity, Reliability and Sensitivity of Measures of Sporting Performance 313

trial. Similarly, no feedback on performance should Based on the available literature, it is suggested
be given until all trials are completed. that in order to best control all factors during a
performance trial, the following conditions should
4.5 Measurements be followed:
• no feedback of performance either during or after
It has been suggested that taking physiological the trial until all trials are completed;
measures such as expired gas or blood during a • no distractions during performance trials, i.e. no
performance protocol will interfere with the per- music, videos or conversations;
formance of the subject, with the possible mecha- • no encouragement;
nism for this being linked to concentration.[1] Often • no physiological measurements unless absolutely
it is important to take these physiological measures necessary;
to answer more mechanistic questions, but it may • do not give performance clues by taking mea-
not be possible to answer both mechanistic and sures or giving drinks at set timepoints, try to
applied questions using the same protocol. How- give them at a set percentage of completion;
ever, a steady-state preload can be added to a proto- • temperature and humidity should be kept con-
col to measure physiological variables without af- stant;
fecting performance, such as that used by Jentjens • equipment such as shoes and clothing should be
and Jeukendrup[101] who used a 1-hour pre-load at kept the same in each trial.
65% Wmax followed by a time trial of ≈40 minutes While validity and reliability are essential com-
at 80% Wmax on a cycle ergometer. ponents of performance tests and have been
researched to a certain extent, the concepts of sensi-
5. Conclusion tivity and transfer have to the best of our knowledge
not been analysed. When designing new perform-
There have been many methods that have been ance protocols, researchers should attempt to inves-
used to attempt to measure performance. Tradition- tigate the sensitivity of the protocol and to see how
ally, researchers have attempted to measure endur- much of the effect seen by a particular intervention
ance performance, with a debate about whether a ‘transfers’ to actual performance. While reliability,
time-to-exhaustion or time-trial protocol is the most validity, sensitivity and transfer are all interrelated
appropriate. Time trials appear to have lower varia- when information on each component is available,
tion (CV <5%) than time-to-exhaustion (CV >25%) researchers will be better able to make decisions
protocols[1] and to be a more valid representation of about which performance protocol to use.
what occurs during performance; therefore, this cur-
rently seems the preferred measure of performance Acknowledgements
in many studies. However, more work needs to be
conducted into the validity of these protocols. No sources of funding were used to assist in the prepara-
tion of this review. The authors have no conflicts of interest
In recent years, there has been an increase in the that are directly relevant to the content of this review.
occurrence of research where simulation of more
complex sports such as soccer and tennis has been
used. Protocols have tended to either provide relia- References
1. Jeukendrup A, Saris WHM, Brouns F, et al. A new validated
ble and valid simulations of only the physiological endurance performance test. Med Sci Sports Exerc 1996; 28
responses such as heart rate and blood lactate[28] or (2): 266-70
2. Hopkins WG, Schabort EJ, Hawley JA. Reliability of power in
of skill performance;[35] however, protocols simulat- physical performance tests. Sports Med 2001; 31 (3): 211-34
ing both physical and skill components of perform- 3. Atkinson G. Sport performance: variable or construct? J Sports
ance are scarce.[37] Future research should focus on Sci 2002; 20 (4): 291-2
4. Reilly T, Gilbourne D. Science and football: a review of applied
including skill aspects into protocols and it is impor- research in the football codes. J Sports Sci 2003; 21 (9):
tant that any new protocols that are developed are 693-705
5. Halson SL, Bridge MW, Meeusen R, et al. Time course of
assessed for reliability and validity prior to use in performance changes and fatigue markers during intensified
intervention studies. training in trained cyclists. J Appl Physiol 2002; 93 (3): 947-56

© 2008 Adis Data Information BV. All rights reserved. Sports Med 2008; 38 (4)
314 Currell & Jeukendrup

6. Hopkins WG. Measures of reliability in sports medicine and 28. Nicholas CW, Nuttall FE, Williams C. The Loughborough
science. Sports Med 2000; 30 (1): 1-15 Intermittent Shuttle Test: a field test that simulates the activity
7. Atkinson G, Nevill AM. Statistical methods for assessing mea- pattern of soccer. J Sports Sci 2000; 18 (2): 97-104
surement error (reliability) in variables relevant to sports medi- 29. Bangsbo J, Lindquist F. Comparison of various exercise tests
cine. Sports Med 1998; 26 (4): 217-38 with endurance performance during soccer in professional
8. Thomas JR, Nelson JK. Research methods in physical activity. players. Int J Sports Med 1992; 13 (2): 125-32
Champaign (IL): Human Kinetics, 1990 30. Rampinini E, Bishop D, Marcora S, et al. Validity of simple
9. Thomas JR, Nelson JK. Research methods in physical activity. field tests as indicators of match related physical performance
Champaign (IL): Human Kinetics, 2001 in top level professional soccer players. Int J Sport Med 2007;
10. Hawley JA, Noakes TD. Peak power output predicts maximal 28 (3): 228-35
oxygen uptake and performance time in trained cyclists. Eur J 31. Krustrup P, Mohr M, Amstrup T, et al. The yo-yo intermittent
Appl Physiol 1992; 65 (1): 79-83 recovery test: physiological response, reliability, and validity.
11. Williams C, Nute MG, Broadbank L, et al. Influence of fluid Med Sci Sports Exerc 2003; 35 (4): 697-705
intake on endurance running performance: a comparison be- 32. Abt G, Zhou S, Weatherby R. The effect of a high-carbohydrate
tween water, glucose and fructose solutions. Eur J Appl Physi- diet on the skill performance of midfield soccer players after
ol Occup Physiol 1990; 60 (2): 112-9 intermittent treadmill exercise. J Sci Med Sport 1998; 1 (4):
12. Foster C, Green MA, Snyder AC, et al. Physiological responses 203-12
during simulated competition. Med Sci Sports Exerc 1993; 25 33. Cox G, Mujika I, Tumilty D, et al. Acute creatine supplementa-
(7): 877-82 tion and performance during a field test simulating match play
13. Palmer GS, Hawley JA, Dennis SC, et al. Heart rate responses in elite female soccer players. Int J Sport Nutr Exerc Metab
during a 4-d cycle stage race. Med Sci Sports Exerc 1994; 26 2002; 12 (1): 33-46
(10): 1278-83 34. Welsh RS, Davis JM, Burke JR, et al. Carbohydrates and
14. Liedl MA, Swain DP, Branch JD. Physiological effects of physical/mental performance during intermittent exercise to
constant versus variable power during endurance cycling. Med fatigue. Med Sci Sports Exerc 2002; 34 (4): 723-31
Sci Sports Exerc 1999; 31 (10): 1472-7 35. McGregor SJ, Nicholas CW, Lakomy HK, et al. The influence
15. Palmer GS, Borghouts LB, Noakes TD, et al. Metabolic and of intermittent high-intensity shuttle running and fluid inges-
performance responses to constant-load vs variable-intensity tion on the performance of a soccer skill. J Sports Sci 1999; 17
exercise in trained cyclists. J Appl Physiol 1999; 87 (3): (11): 895-903
1186-96 36. Ostojic S, Mazic S. Effects of a carbohydrate-electrolyte drink
16. Laursen PB, Rhodes EC, Langill RH, et al. Relationship of on specific soccer tests and performance. J Sports Sci Med
exercise test variables to cycling performance in an Ironman 2002; 1: 47-53
triathlon. Eur J Appl Physiol 2002; 87 (4-5): 433-40 37. Vergauwen L, Spaepen AJ, Lefevre J, et al. Evaluation of stroke
17. Palmer GS, Dennis SC, Noakes TD, et al. Assessment of the performance in tennis. Med Sci Sports Exerc 1998; 30 (8):
reproducibility of performance testing on an air-braked cycle 1281-8
ergometer. Int J Sports Med 1996; 17 (4): 293-8 38. Williams AM. Perceptual skill in soccer: implications for talent
18. Russell RD, Redmann SM, Ravussin E, et al. Reproducibility of identification and development. J Sports Sci 2000; 18 (9):
endurance performance on a treadmill using a preloaded time 737-50
trial. Med Sci Sports Exerc 2004; 36 (4): 717-24 39. Bagger M, Petersen PH, Pedersen PK. Biological variation in
19. Smith MF, Davison RC, Balmer J, et al. Reliability of mean variables associated with exercise training. Int J Sports Med
power recorded during indoor and outdoor self-paced 40 km 2003; 24 (6): 433-40
cycling time-trials. Int J Sports Med 2001; 22 (4): 270-4 40. Vincent W. Statistics in kinesiology. 3rd ed. Champaign (IL):
20. Elliott B, Lyttle A, Birkett O. The rowperfect ergometer; a Human Kinetics, 2005
training aid for on water single scull rowing. Sports Biomech 41. Bland JM, Altman DG. Statistical methods for assessing agree-
2002; 1 (2): 123-34 ment between two methods of clinical measurement. Lancet
21. Helgerud J, Engen LC, Wisloff U, et al. Aerobic endurance 1986; I (8476): 307-10
training improves soccer performance. Med Sci Sports Exerc 42. Lindsay FH, Hawley JA, Myburgh KH, et al. Improved athletic
2001; 33 (11): 1925-31 performance in highly trained cyclists after interval training.
22. Zeederberg C, Leach L, Lambert EV, et al. The effect of Med Sci Sports Exerc 1996; 28 (11): 1427-34
carbohydrate ingestion on the motor skill proficiency of soccer 43. Coggan AR, Costill DL. Biological and technological variabil-
players. Int J Sport Nutr 1996; 6 (4): 348-55 ity of three anaerobic ergometer tests. Int J Sports Med 1984; 5
23. Di Salvo V, Collins A, McNeill B, et al. Validation of prozone: a (3): 142-5
new video based performance analysis system. Int J Perf 44. Graham K, McLellan T. Variability of time to exhaustion and
Analysis Sport 2006; 6 (1): 108-19 oxygen deficit in supramaximal exercise. Aust J Sci Med Sport
24. Drust B, Reilly T, Cable NT. Physiological responses to labora- 1989; 21 (4): 11-4
tory-based soccer-specific intermittent and continuous exer- 45. Billat V, Renoux JC, Pinoteau J, et al. Reproducibility of
cise. J Sports Sci 2000; 18 (11): 885-92 running time to exhaustion at V̇O2max in subelite runners. Med
25. Reilly T, Thomas V. A motion analysis of work-rate in different Sci Sports Exerc 1994; 26 (2): 254-7
positional roles in professional football match play. J Hum 46. Laursen PB, Francis GT, Abbiss CR, et al. Reliability of time to
Move Stud 1976; 2: 87-97 exhaustion versus time trial running tests in runners. Med Sci
26. Thatcher R, Betterham A. Development and validation of a Sports Exerc 2007; 39 (8): 1374-9
sport specific exercise protocol for elite youth players. J Sports 47. McLellan TM, Cheung SS, Jacobs I. Variability of time to
Med Phys Fitness 2004; 44 (1): 15-22 exhaustion during submaximal exercise. Can J Appl Physiol
27. MP G, McNaughton LR, Lovell R. Physiological and mechani- 1995; 20 (1): 39-51
cal response to soccer specific intermittent activity and steady 48. Krebs P, Powers S. Reliability of laboratory endurance tests
state activity. Res Sports Med 2006; 14 (1): 29-52 [abstract]. Med Sci Sports Exerc 1989; 21: S10

© 2008 Adis Data Information BV. All rights reserved. Sports Med 2008; 38 (4)
Validity, Reliability and Sensitivity of Measures of Sporting Performance 315

49. Maughan RJ, Fenn CE, Leiper JB. Effects of fluid, electrolyte 70. Schabort EJ, Hawley JA, Hopkins WG, et al. High reliability of
and substrate ingestion on endurance capacity. Eur J Appl performance of well-trained rowers on a rowing ergometer. J
Physiol Occup Physiol 1989; 58 (5): 481-6 Sports Sci 1999; 17 (8): 627-32
50. Gleser MA, Vogel JA. Endurance exercise: effect of work-rest 71. Mohr M, Krustrup P, Bangsbo J. Match performance of high
schedules and repeated testing. J Appl Physiol 1971; 31 (5): standard soccer players with special reference to developement
735-9 of fatigue. J Sport Sci 2003; 21 (7): 519-28
51. Hickey MS, Costill DL, McConell GK, et al. Day to day 72. Rampinini E, Coutts A, Castagna C, et al. Variation in top level
variation in time trial cycling performance. Int J Sports Med soccer match performance. Int J Sport Med 2007; 28 (12):
1992; 13 (6): 467-70 1018-24
52. Hodges K, Hancock S, Currell K, et al. Pseudoephedrine en- 73. Di Salvo V, Baron R, Tschan H, et al. Performance characteris-
hances performance in 1500-m runners. Med Sci Sports Exerc tics according to playing position in elite soccer. Int J Sport
2006; 38 (2): 329-33 Med 2007; 28 (3): 222-7
53. Jensen K, Johansen L. Reproducibility and validity of physio- 74. Weston M, Castagna C, Impellizzeri FM, et al. Analysis of
logical parameters measured in cyclists riding on racing bikes physical match performance in English Premier League soccer
placed on a stationary magnetic brake. Scand J Med Sci Sports referees with particular reference to first half and player work
1998; 8 (1): 1-6 rates. J Sci Med Sport 2007; 10 (6): 390-7
54. Nicholson RM, Sleivert GG. Indices of lactate threshold and 75. Krustrup P, Mohr M, Nybo L, et al. The Yo-Yo IR2 test:
their relationship with 10-km running velocity. Med Sci Sports physiological response, reliability, and application to elite
Exerc 2001; 33 (2): 339-42 soccer. Med Sci Sports Exerc 2006; 38 (9): 1666-73
55. Currell K, Jentjens R, Jeukendrup A. Reliability of a cycling 76. McGregor SJ, Hulse M, Strudwick A, et al. The reliability and
time trial in a glycogen depleted state. Eur J Appl Physiol validity of two tests of soccer skill [abstract]. J Sport Sci 1999;
2006; 98 (6): 583-9 17: 815
56. Tan F, Aziz A. Reproducibility of outdoor flat and uphill 77. Avery C, Richardson P, Jackson W. A practical tennis serve test:
cycling time trials and their performance correlates with peak measurement of skill under simulated game conditions. Res Q
power output in moderately trained cyclists. J Sports Sci Med Exerc Sport 1998; 50: 554-64
2005; 4: 278-84
78. Percell K. A tennis forehand-backhand drive skill test which
57. Laursen PB, Shing CM, Jenkins DG. Reproducibility of a labor- measures ball control and stroke firmness. Res Q Exerc Sport
atory-based 40-km cycle time-trial on a stationary wind-trainer 1981; 52: 238-45
in highly trained cyclists. Int J Sports Med 2003; 24 (7): 481-5
79. Hopkins WG, Hawley JA, Burke LM. Design and analysis of
58. Widrick JJ, Costill DL, Fink WJ, et al. Carbohydrate feedings research on sport performance enhancement. Med Sci Sports
and exercise performance: effect of initial muscle glycogen Exerc 1999; 31 (3): 472-85
concentration. J Appl Physiol 1993; 74 (6): 2998-3005
80. Paton CD, Hopkins WG. Tests of cycling performance. Sports
59. Schabort EJ, Hawley JA, Hopkins WG, et al. A new reliable
Med 2001; 31 (7): 489-96
laboratory test of endurance performance for road cyclists.
Med Sci Sports Exerc 1998; 30 (12): 1744-50 81. Hopkins WG, Hewson DJ. Variability of competitive perform-
ance of distance runners. Med Sci Sports Exerc 2001; 33 (9):
60. Madsen K, MacLean DA, Kiens B, et al. Effects of glucose,
1588-92
glucose plus branched-chain amino acids, or placebo on bike
performance over 100 km. J Appl Physiol 1996; 81 (6): 82. Jeukendrup AE, Martin J. Improving cycling performance: how
2644-50 should we spend our time and money. Sports Med 2001; 31
61. Doherty M, Balmer J, Davison RC, et al. Reliability of a (7): 559-69
combined 3-min constant load and performance cycling test. 83. Jeukendrup AE. Carbohydrate intake during exercise and per-
Int J Sports Med 2003; 24 (5): 366-71 formance. Nutrition 2004; 20 (7-8): 669-77
62. Paton CD, Hopkins WG. Ergometer error and biological varia- 84. Jeukendrup AE, Brouns F, Wagenmakers AJM, et al. Carbohy-
tion in power output in a performance test with three cycle drate feedings improve 1 h trial cycling performance. Int J
ergometers. Int J Sports Med 2006; 27 (6): 444-7 Sports Med 1997; 18: 125-9
63. Bishop D. Reliability of a 1-h endurance performance test in 85. Davis JM, Lamb DR, Pate RR, et al. Carbohydrate-electrolyte
trained female cyclists. Med Sci Sports Exerc 1997; 29 (4): drinks: effects on endurance cycling in the heat. Am J Clin
554-9 Nutr 1988; 48 (4): 1023-30
64. Schabort EJ, Hopkins WG, Hawley JA. Reproducibility of self- 86. Carter J, Jeukendrup AE, Mundel T, et al. Carbohydrate supple-
paced treadmill performance of trained endurance runners. Int mentation improves moderate and high-intensity exercise in
J Sports Med 1998; 19 (1): 48-51 the heat. Pflugers Arch 2003; 446 (2): 211-9
65. Hinckson EA, Hopkins WG. Reliability of time to exhaustion 87. Chryssanthopoulos C, Hennessy LC, Williams C. The influence
analyzed with critical-power and log-log modeling. Med Sci of pre-exercise glucose ingestion on endurance running capa-
Sports Exerc 2005; 37 (4): 696-701 city. Br J Sports Med 1994; 28 (2): 105-9
66. Jeukendrup AE, Currell K. Should time trial performance be 88. Wilber RL, Moffatt RJ. Influence of carbohydrate ingestion on
predicted from three serial time-to-exhaustion tests? Med Sci blood glucose and performance in runners. Int J Sport Nutr
Sports Exerc 2005; 37 (10): 1820; author reply 1821 1992; 2 (4): 317-27
67. Soper G, Hume PA. Reliability of power output during rowing 89. Coggan AR, Coyle EF. Metabolism and performance following
changes with ergometer type and race distance. Sports Bi- carbohydrate ingestion late in exercise. Med Sci Sports Exerc
omech 2004; 3 (2): 237-48 1989; 21 (1): 59-65
68. American College of Sports Medicine, American Dietetic Asso- 90. Laursen PB, Shing CM, Peake JM, et al. Interval training
ciation, Dietitians of Canada. Joint position statement: nutri- program optimization in highly trained endurance cyclists.
tion and athletic performance. Med Sci Sports Exerc 2000; 32 Med Sci Sports Exerc 2002; 34 (11): 1801-7
(12): 2130-45 91. St Clair Gibson A, Noakes TD. Evidence for complex system
69. Rauch HG, St Clair Gibson A, Lambert EV, et al. A signalling integration and dynamic neural regulation of skeletal muscle
role for muscle glycogen in the regulation of pace during recruitment during exercise in humans. Br J Sports Med 2004;
prolonged exercise. Br J Sports Med 2005; 39 (1): 34-8 38 (6): 797-806

© 2008 Adis Data Information BV. All rights reserved. Sports Med 2008; 38 (4)
316 Currell & Jeukendrup

92. Hulleman M, De Koning JJ, Hettinga F, et al. The effect of 99. Paterson S, Marino FE. Effect of deception of distance on
extrinsic motivation on cycle time trial performance. Med Sci prolonged cycling performance. Percept Motor Skills 2004;
Sports Exerc 2007; 39 (4): 709-15
93. Atkinson G, Wilson D, Eubank M. Effects of music on work- 98: 1017-26
rate distribution during a cycling time trial. Int J Sports Med 100. Nethery VM. Competition between internal and external
2004; 25 (8): 611-5 sources of information during exercise: influence on RPE and
94. Pujol TJ, Langenfeld ME. Influence of music on Wingate Ana-
erobic Test performance. Percept Mot Skills 1999; 88 (1): the impact of the exercise load. J Sports Med Phys Fitness
292-6 2002; 42 (2): 172-8
95. Karageorghis CI, Terry PC. The psychological effects of music 101. Jentjens RL, Jeukendrup AE. Effects of pre-exercise ingestion
in sport and exercise: a review. J Sports Behav 1997; 20 (1):
54-68 of trehalose, galactose and glucose on subsequent metabolism
96. Szmedra L, Bacharach DW. Effect of music on perceived exer- and cycling performance. Eur J Appl Physiol 2003; 88 (4-5):
tion, plasma lactate, norepinephrine and cardiovascular hemo- 459-65
dynamics during treadmill running. Int J Sports Med 1998; 19
(1): 32-7
97. Boutcher S, Trenske M. The effects of sensory deprivation and
music on perceived exertion and affect during exercise. J Sport Correspondence: Prof. Asker E. Jeukendrup, School of Sport
Exerc Psychol 1990; 12: 167-76 and Exercise Sciences, University of Birmingham, Birming-
98. Nikolopoulos V, Arkinstall MJ, Hawley JA. Pacing strategy in
simulated cycle time-trials is based on perceived rather than
ham, B15 2TT, UK.
actual distance. J Sci Med Sport 2001; 4 (2): 212-9 E-mail: A.E.Jeukendrup@bham.ac.uk

© 2008 Adis Data Information BV. All rights reserved. Sports Med 2008; 38 (4)