Annals of Physical and Rehabilitation Medicine 58 (2015) 14–22

Available online at

ScienceDirect
www.sciencedirect.com

Original article

Thought-based row-column scanning communication board

for individuals with cerebral palsy
Reinhold Scherer a,b,c,*, Martin Billinger a,b, Johanna Wagner a,b, Andreas Schwarz a,b,
Dirk Tassilo Hettich d, Elaina Bolinger d, Mariano Lloria Garcia e, Juan Navarro e,
Gernot Müller-Putz a,b
a
Institute for Knowledge Discovery, Graz University of Technology, Graz, Austria
b
BioTechMed, Graz, Austria
c
Rehabilitation center Judendorf-Straßengel, Judendorf-Straßengel, Austria
d
Eberhard Karls University of Tübingen, Tübingen, Germany
e
Avapace, Asociación Valenciana de Ayuda a la Parálisis Cerebral, Valencia, Spain

A R T I C L E I N F O A B S T R A C T

Article history: Impairment of an individual’s ability to communicate is a major hurdle for active participation in
Received 10 July 2014 education and social life. A lot of individuals with cerebral palsy (CP) have normal intelligence, however,
Accepted 12 November 2014 due to their inability to communicate, they fall behind. Non-invasive electroencephalogram (EEG) based
brain-computer interfaces (BCIs) have been proposed as potential assistive devices for individuals with
Keywords: CP. BCIs translate brain signals directly into action. Motor activity is no longer required. However,
Brain translation of EEG signals may be unreliable and requires months of training. Moreover, individuals with
Computer interface
CP may exhibit high levels of spontaneous and uncontrolled movement, which has a large impact on EEG
Electroencephalogram
Cerebral palsy
signal quality and results in incorrect translations. We introduce a novel thought-based row-column
Sensory motor rhythm scanning communication board that was developed following user-centered design principles. Key
Human-computer interaction features include an automatic online artifact reduction method and an evidence accumulation procedure
Communication board for decision making. The latter allows robust decision making with unreliable BCI input. Fourteen users
Assistive technology with CP participated in a supporting online study and helped to evaluate the performance of the
developed system. Users were asked to select target items with the row-column scanning
communication board. The results suggest that seven among eleven remaining users performed better
than chance and were consequently able to communicate by using the developed system. Three users
were excluded because of insufficient EEG signal quality. These results are very encouraging and
represent a good foundation for the development of real-world BCI-based communication devices for
users with CP.
ß 2015 Elsevier Masson SAS. All rights reserved.

1. Introduction challenged as a result of their inability to communicate [3]. Com-

munication solutions are available; however, they strongly depend
Cerebral palsy (CP) is a non-progressive condition caused by on motor activity and on the assistance of others. Brain-Computer
damage to the brain during early developmental stages [1,2]. Indi- Interfaces (BCIs) represent a possible alternative communication
viduals with CP may have a range of problems related to motor channel [4].
control, speech, comprehension, or other cognitive impairments. BCIs translate brain activity directly into action [5–11]. Non-
Around one quarter of individuals with CP have normal intelli- invasive BCIs are typically based on electroencephalographic (EEG)
gence, but nevertheless are often classified as cognitively signals. To send messages, BCI users either focus on sensory stimuli
(visual [12], auditory [13] or somatosensory [14]) and generate
evoked potentials (EPs) or perform, independently of any stimulus,
* Corresponding author at: Institute for Knowledge Discovery, Graz University
specific mental imagery and induce transient changes in sponta-
of Technology, boulevard Doyen-Gaston-Giraud, 34395 Graz, France.
Tel.: +33 6 65 84 90 73. neous EEG rhythms (Event-Related Desynchronization [15]). Such
E-mail address: reinhold.scherer@tugraz.at (R. Scherer). mental imagery includes, amongst others, motor imagery (the

http://dx.doi.org/10.1016/j.rehab.2014.11.005
1877-0657/ß 2015 Elsevier Masson SAS. All rights reserved.
 R. Scherer et al. / Annals of Physical and Rehabilitation Medicine 58 (2015) 14–22 15

kinesthetic imagination of movement [16]), mental arithmetic, and only notified on whether imagery (equals switch activated) was
the mental generation of words [17–20]. detected while the current row (item) was highlighted or not.
Within the framework of the FP7 Framework EU Research Fourthly, spelling words by selecting individual letters was very
Project ABC, we recently started developing BCI technology for difficult for CP users. To facilitate selection, we used symbols and
individuals with CP. The aim of the project is to develop assistive images representing the action to be performed. Fifthly, BCI
technology that improves independent interaction, enhances non- training paradigms and BCI-based control are usually different, in
verbal communication and allows expression and management of that training does not provide feedback on detected imagery
emotions for users with CP. Several challenges have to be faced: activity. This distinction was not transparent for users. Conven-
Firstly, EEG sensor placement can be difficult due to body posture tional training paradigms were moreover too abstract and boring
or head and neck support systems and will benefit from novel for the user. Consequently, training and control paradigms were
materials and sensor processes that are user-friendlier. Secondly, combined. Reducing graphical user interface (GUI) complexity
involuntary movements and spasms generate bioelectrical activity makes instructions for the user simpler and less ambiguous.
that lead to artifacts, which can produce misleading EEG signals or Based on these specifications, and with the aim to overcome
destroy them altogether [21]. Ensuring high signal quality is some of the challenges mentioned initially, we developed the
essential [22–24]. Thirdly, time-consuming BCI calibration pro- current BCI system (Fig. 1).
cesses need to be optimized. While EP-based BCIs typically achieve
higher detection rates and require substantially less training time 2.2. System architecture
than imagery-based BCIs, which type will be most useful depends
on residual motor and cognitive capabilities of the user. Major The BCI has a distributed, modular architecture and was
issues that impact the detection performance are the non- implemented by using open standards when possible. In the
stationarity and inherent variability of EEG signals. Novel methods current implementation a Windows operating system (OS) based
and user-group related protocols have to be developed that allow laptop computer was used for EEG acquisition and signal
predicting robust control signals from EEG data. Fourthly, BCI processing. Feedback and the application were presented on an
training paradigms and instructions have to be adapted to the CP Android OS tablet computer (Fig. 1). BCI modules were imple-
user’s individual capabilities and skills. Depending on situation mented following the TOBI interface specifications [27,28]. Com-
and availability, individuals with CP are able to attend school or munication between the BCI and the Android application was
special training programs. Therefore, each user is different and based on a specially developed ABC protocol. This allows users to
information must be presented in a user-specific manner. interact with the application by means of other input signals and
To ensure usability and functionality of our developments, we modalities developed within the ABC project (for example,
follow user-centered design principles [10,25]. In this paper, we standard human-computer interaction or inertial measurement
present our first prototype BCI and a corresponding communica- devices). Operator computers can be used for monitoring and
tion application, and report results of a supporting online study in controlling experimental procedures.
14 end-users with CP.
2.3. The row-column scanning communication board

2. Methods Fig. 1 shows a picture of the GUI. The screen was split into two
parts. The left side contains the grid. On the right side feedback
2.1. User-centered system design adaptations on the selected item was presented. Each row (item) was
highlighted with a red colored box for a predefined time. The
The BCI was designed and remodeled in several iteration steps marker disappeared and after a short break the next row (item)
according to the feedback received from adult CP users, relatives, was highlighted. When the last row (item) was reached, the
caregivers and healthy test users. Firstly, we tested EP-based and marker jumped again to the first element and the sequence
imagery-based BCIs in CP users. We found that CP users could not started again. The selection of a row (item) was reported back to
utilize EP methods for a number of reasons, however, imagery- the user visually by showing an animation sequence of the row
based methods were viable [17,26]. Secondly, for communication (item) dissolving. Additionally, an auditory beep was presented.
we aimed at developing a communication application. Some When an item was selected, the scan cycle started again from the
individuals had previous experience operating row-column first row. In this study a grid with three rows and three columns
scanning communication boards such as The Grid Augmentative was used. Rows (items) were highlighted for 4 s with a 2-s break
and Alternative Communication software (Sensory Software between the markers. Timing can be adapted to fit the user
international, Malvern, UK). The Grid uses one-switch row-column needs, when required. Items included a strawberry, soccer ball,
scanning to select items that are arranged in a grid. Row-column banana, lemon, watermelon, heart, grapes, flower and pineapple
scanning means that each row within the grid is sequentially (Fig. 1).
highlighted until the user selects the row containing the desired
item (for example, letters or icons, Fig. 1) by activating the switch. 2.4. The BCI switch
The columns within the selected row are then scanned until the
target item is highlighted and can be selected by activating the The switch for selecting row (item) was implemented by
switch a second time. Consequently, we aimed at developing a BCI training the BCI to classify between imagery and non-imagery EEG
that robustly generates a binary control signal for replacing the patterns. Signal processing was performed with Matlab/Simulink
switch. To optimize communication speed, we first implemented a (MathWorks, Natick, MA, USA). The standard method of common
maximum-likelihood selection, i.e., letters that are most likely to spatial patterns (CSP) was used to design class specific spatial
be selected appear as the next available item. Using a dynamically filters in user-specific frequency bands, and Fisher’s linear
adapted scanning protocol, however, was confusing for most users. discriminant analysis (LDA) classifier to classify the log-trans-
Thus, we switched back to the slower but familiar row-column formed normalized variance from 4 CSP projections (m = 2, [29]).
scanning mode. Thirdly, to concentrate on continuous feedback, The CSP method projects multi-channel EEG data segments from
which is typically used in BCI, was very demanding for the users. two classes into a low-dimensional spatial subspace in such a way
Therefore, we changed to discrete feedback (Fig. 1). Users were that the variances of the time series are optimal for discrimination
16 R. Scherer et al. / Annals of Physical and Rehabilitation Medicine 58 (2015) 14–22

Fig. 1. Architecture of the BCI-based one-switch row-column communication board. The working principle of the row-column scanning is outlined in the gray box. The
example illustrates the selection of the ‘‘Flower’’ item (3th Row, 2nd Column). The experimental timing is shown in the lower part of the box. The picture in the upper-left area
shows a CP user sitting in front of the communication board during the experiment preparation phase. Raw EEG signals are recorded from the users and artifacts are removed
by using the FORCe method. Clean EEG segments (t = [03] s) are classified (CSP + LDA) as either imagery or not-imagery segments. When enough evidence for decision making
is available, the currently highlighted row (item) is selected. An example evidence accumulation for the selection of the 2nd row is shown in the dotted box. Each time imagery
is detected for the currently highlighted row, the buffer for the specific row is incremented by one. Assuming a 3 out of 5 rule, i.e., three out of the past five selections for a row
have to be classified as imagery, then the 2nd row is selected during the 4th scan cycle. Note, that no imagery was detected during the 3th scan cycle.

[30,31]. LDA projects the CSP filtered signal onto a line and decomposed into independent components (IC) by second-order
performs classification by thresholding in the projected one- blind identification algorithm [33]. Various criteria that charac-
dimensional space [32]. terize artifacts are applied to the IC. Criteria include the amount
of temporal dependency within the signal, the amount of spiking
2.4.1. Fully online and automated artifact removal (FORCe) activity, the kurtosis of the signal, the similarity of the power
Many BCI users with CP exhibit high levels of spontaneous spectral density (PSD) distribution to a 1/f distribution (with f
movement. Therefore an automated method for the removal of denoting the frequency), the PSD of the gamma band (> 30 Hz), the
electromyogram (EMG) artifacts was developed and integrated standard deviation and topographic distribution of the amplitudes,
in the online BCI system. EEG signals are first decomposed via and peak values. ICs that do not meet the criteria represent
the Wavelet decomposition method (‘‘Sym4’’ wavelet). Approxi- artifacts and are excluded. Clean EEG is obtained by reconstructing
mation coefficients (low frequency components) are again the remaining ICs (Fig. 1) [34].
 R. Scherer et al. / Annals of Physical and Rehabilitation Medicine 58 (2015) 14–22 17

Since the row-column scan mode provides visual cues, EEG that imagery was detected while the current row (item) was
signal analysis was synchronized to the marker onset (t = 0 s, Fig. 1, highlighted an auditory beep was presented at the time of the
Highlight event). Four non-overlapping 1-second EEG segments marker-offset. A selection (m out of N imagery class detections)
St = [t 1 t] s were extracted at t = 0, 1, 2, 3 s. The FORCe method triggered the animation as described above (supplementary video).
was applied to each St independently at time t. The reason for This approach increased the selection time, however, also the
applying the method to 1-s segments was its high computational robustness of detection. This strategy, moreover, provides end-
demand. Cleaning of 1-s segments took about 300 ms on a modern users on-demand access to an application.
laptop. Hence, at second t = 3.3 s all segments were cleaned. Since
discrete feedback was provided at t = 4 s, sufficient time was left 2.5. BCI calibration and online item selection runs
for signal processing. Clean 1-s segments were concatenated to one
4-s segment and band-pass filtered in a user-specific frequency Before the BCI can be operated, model parameters need to be
range. The first second of the filtered EEG was discarded to adapted to fit user-specific EEG patterns. To this end EEG trials of
eliminate filter boundary effects. Log-transformed normalized mental imagery were recorded from users. The type of mental
variance from 4 CSP projections were computed from the imagery was defined in agreement with the user prior to the
remaining 3-s EEG segment and classified. calibration. The same experimental paradigm was used for BCI
parameter calibration and online item selection: Users were asked
2.4.2. Evidence accumulation to define a target item and to select it by performing the dedicated
Spasms or involuntary movement may prevent users from imagery each time the related row (item) was highlighted. Users
looking at the GUI - preventing interaction - or even induce EEG were also asked not to perform the dedicated imagery when the
patterns that may mistakenly be interpreted as imagery. To reduce target (row) item was not highlighted. During calibration, the BCI
misinterpretation of EEG patterns, we implemented evidence automatically presented, following m = 3 out of N = 5 evidence
accumulation. Users were asked to repeatedly confirm a selection accumulation procedure, auditory beeps (sham feedback for
before it was accepted by the BCI. More precisely, each row (item) imagery detection). More precisely, for each row (item)
scan step was classified, and the classifier output was stored in a N = 5 scan cycles with m = 3 correct and N-m = 2 erroneous
ring buffer (size N) for each row (item). The highlighted row (item) classifications were presented. The fifth cycle was always correctly
was selected only when a certain number (m) of the last N selected and triggered the animation sequence for the row (item).
classification outputs was classified as imagery class. Due to the The remaining two correct and two erroneous beeps were
sequential scan order, it takes at least m scan cycles to select a row presented in random permutation order during cycles 1–4.
(item). Fig. 1 shows an example row selection. To notify the user Depending on the target item, 13–15 scan steps per row (item)

Table 1
User details.

User Gender Age GMFCS CP type Sensory Dominant Additional information

disturbances hand

S1 M 27 V Dystonic-Spastic – None Anarthria; requires sitting support for trunk control; no

difficulty breathing; yes/no communication by tongue signs;
sometimes w/help of pictograms
S2* F 20 IV Atypical-Spastic – Left Light dysarthria
S3* F 25 V Dystonic-Spastic – Left Anarthria; requires sitting support for trunk control; no
difficulty breathing; yes/no communication to simple
questions by phonemes
S4 M 53 V Dystonic – Left Dysarthria; requires sitting support for trunk control; no
difficulty breathing
S5 F 56 IV Dystonic Myopia Right Dysarthria; uses AAC communicator with ARASAAC system on
Tablet PC by touchscreen; no difficulty breathing
S6 F 38 V Spastic – Right Anarthria; uses SAAC communicator with ARASAAC system on
Tablet PC, row-column scanning, face switch; requires sitting
support for the head and trunk control; uses electric chair with
chin-joystick; no difficulty breathing
S7 M 40 V Dystonic-Spastic – Left Anarthria; breathing difficulties; tracheotomy; neurogenic
dysphagia; user AAC, PCS, over 100 pictograms, accessed by
scanning assisted by the caregiver and bodily rather confirms;
Sitting support
S8 M 33 V Spastic – Left Anarthria; difficulties expectorating neurogenic dysphagia;
requires sitting support; uses AAC, Bliss system, more than
300 pictograms, accessed by scanning and operated by facial
muscle switch
S9 M 24 IV Dystonic – Right Tetraplegic, right hand electric wheelchair control, can stand
on a standing frame. Uses upper limbs to control a portable
personal computer (PC) with a keyboard and Hermes PC
software for communication
S10 M 34 V Spastic Left side None Anarthria; limited ability to keep head, trunk and limbs against
blindness and gravity; can navigate autonomously using electric wheelchair
deafness with head control device
Communication by writing tablet pc, scan and access switch in
the head and speech synthesis
S11 M 39 IV Dystonic-spastic – Left Anarthria; uses electric wheelchair indoors and outdoors short
distances controlled by a rod next to the left cheek. Needs
support to maintain the seated trunk. Uses AAC, Bliss system,
500 pictograms, the grid software in tablet pc and access
scanning

GMFCS denotes the Gross Motor Function Classification System score. Users marked with an asterisk (*) are first-time BCI user. Users S4-S11 participated in [17].
18 R. Scherer et al. / Annals of Physical and Rehabilitation Medicine 58 (2015) 14–22

Fig. 2. Evidence accumulation chance level performance. The curves show the relationship between accuracy and the probability of correct selections as functions of TPR and
TNR for selecting the strawberry (row 1, item 1), ball (row 1, item 2) and banana (row 1, item 3) target items, respectively. The highlighted area illustrates the valid range.

were required. According to this scheme, correct beeps were operate the BCI. Hence, each user received individual instructions
presented in 54-60% of the time. This should simulate real-world and explanations. The experiment consisted of the following steps:
‘‘worst case’’ accuracies and train users to stay relaxed in the Firstly, a dedicated mental task was identified. Relatives and
presence of erroneous BCI detections. Clean 3-s EEG segments caregivers helped identifying appropriate tasks. Possible tasks
belonging to classes’ imagery and non-imagery (available in a included motor imagery, mental arithmetic and word generation
larger number due to the 3  3 grid) trials were used to train CSP [17–20]. Secondly, calibration runs were recorded and BCI
filter and LDA classifier. A 10-times 10-fold cross validation parameters computed. Calibration runs were repeated until binary
statistic was used to estimate the discriminative performance. classification was better than random [35]. Thirdly, online item
Since changes in spontaneous rhythms are user-specific, several selection runs were performed. For both calibration and online
frequency bands were evaluated and the best performing band target item selection runs users were asked to pick a target item
was manually selected for each user individually. Frequency before the run started. After each run the data was analyzed, and
bands included 7–30 Hz, 8–12 Hz, 8–24 Hz, 8–36 Hz, 16–24 Hz, CSP and LDA were updated when performance increase was
16–36 Hz, 24–36 Hz. Identified parameters were used during expected.
online item selection runs to translate EEG patterns into click
actions used to select target items by the row-column scan 2.7. Evidence accumulation chance performance level
communication board.
The accuracy, i.e., the sum of true positive (TP) and true negative
2.6. End-user, signal recording and system performance evaluation (TN) detections divided by the total number of detections, is
commonly used to characterize BCI performance. The evidence
A supporting study in 14 users with CP was performed to accumulation procedure, however, makes interpretation of
evaluate the performance and usability of the implemented achieved accuracy difficult. Consider two cases: Firstly, assuming
system. Experiments were performed at AVAPACE daycare centers the classifier always outputs OFF, no selection is made. However,
in Valencia, Spain and at end-users’ homes in Tübingen, Germany. since 2 out of 3 selections are correct, the resulting accuracy is high
Institutional review board (IRB) ethical approval was obtained for (two-third). Secondly, assuming the classifier always outputs ON
all measurements. All participants gave informed, oral consent. In then the first row (item) is always selected. The accuracy, for
addition, written consent was obtained for every participant. In selecting the first item, however is low (one-third). Fig. 2 illustrates
some cases, written consent had to be provided by the participants’ the relationship between accuracy and selection probability for
legal representatives. Details of participants are summarized in
Table 1. Three out of fourteen users had to be excluded due to Table 2
technical problems that resulted in insufficient EEG quality. Some Summary of experimental runs.
users were naı̈ve to the task and did not receive BCI training before User BCI Row and Item Row BTC Failed Aborted
participating in this study. Calibration BTC (total) (total)
EEG was recorded from 16 electrodes placed over cortical areas S1 3 0 (1) 0 (2) 4 0
according to the international 10-20 system. Electrode positions S2 3 2 (3) 0 (2) 3 1
include AFz, FC3, FCz, FC4, C3, Cz, C4, CP3, CPz, CP4, PO3, POz, PO4, S3 2 1 (1) 0 (1) 0 2
O1, Oz, O2. Reference and ground were placed at position Pz and S4 3 0 (0) 2 (4) 3 2
S5 2 0 (3) 0 (3) 2 0
P5, respectively. The g.GAMMAsys system with g.LADYbird active
S6 1 3 (3) 0 (0) 2 0
electrodes (Guger Technolgies, Graz, Austria) and one g.USBamp S7 3 1 (1) 0 (0) 1 0
biosignal amplifier were used to record EEG at a rate of 512 Hz S8 5 0 (2) 0 (0) 1 0
(Notch 50 Hz, 0.5-100 Hz band pass). S9 6 0 (0) 0 (1) 2 0
Participants were sitting in their wheelchairs. The tablet S10 1 3 (4) 0 (0) 2 1
S11 5 1 (2) 0 (0) 5 0
computer was placed about 80 cm in front of them at approxi- Total 34 11 (20) 2 (13) 25 6
mately eye level (Fig. 1). Before each experiment, participants
For each user, the number of runs recorded for BCI calibration, successful target
received instructions on the task to be performed (both in writing
item selection, successful row (but no item) selection, runs that were not successful
as a slideshow and verbally). Relatives and caregivers explained (time out) and the number of runs that were aborted are listed. BTC denotes the
the aim of the study, how to use the speller application and how to number of runs that are better than chance.
Table 3
Experimental procedure and online performance.

User Experimental run

1 2 3 4 5 6 7 8 9 10 11 12

S1 C C C Flower Banana Banana Bal Grapes Watermelon Pineapple – –

0.57 (56) R, I 0.46 (15) R 0.41 (17) R
U: 1, 2; 8–12 U: 2, 3; 8–12 U: 2, 3, 5, 6;
8–24
S2 C, X C C C Grapes Watermelon Watermelon Flower Pineapple Ball Pineapple Watermelon
0.67 (28) R, I 0.50 (17) R 0.64 (14) R 0.77 (27) R, I 0.78 (19) R, I

R. Scherer et al. / Annals of Physical and Rehabilitation Medicine 58 (2015) 14–22

U: 2, 3, 4; 8–24 U: 2, 3, 4; 8–24 U: 5, 6, 7, 8; U: 8, 9, 10; U: 8, 9, 10;
8–24 8–24 8–24
S3 C, X C C Banana, X Banana Watermelon Watermelon – – – – –
0.63 (19) R, I 0.54 (51) R
U: 2, 3; 24–36 U: 2, 3, 5; 24–26
S4 C C Watermelon Heart C Flower, X Flower Flower Pineapple Heart, X Ball Ball
0.5 (16) R 0.54 (22) R 0.64 (17) R 0.50 (28) R
U: 5, 7; 8–12 U: 5, 7, 8; U: 7, 8, 9;
8–12 8–12
S5 C Flower Flower C Strawberry Grapes Grapes Grapes Banana Pineapple – –
Motor 0.79 (29) R, I 0.78 (41) R, I 0.78 (23) R, I 0.52 (23) R 0.61 (77) R 0.58 (80) R
Execution U: 1; 8–36 U: 1, 2; 36 U: 4, 8–12 U: 4, 5, 6; 8–36 U: 4, 5, 6;
8–24
S6 C Heart Heart Flower Flower Strawberry – – – – – –
(20) R, I 1.0 (17) R, I 0.81 (32) R, I
U: 1, 2; 24–36 U: 1, 2; 24–36 U: 1, 2; 24–36
S7 C C C Pineapple Watermelon – – – – – – –
0.87 (24) R, I
U: 1, 3; 8–24
S8 C C C C C Watermelon Heart Heart – – – –
Mental Mental 0.66 (74) R, I 0.62 (78) R, I
subtraction subtraction U: 4, 5; 8–12 U: 4, 5; 8–12
S9 C C Pineapple Heart C C C C Ball – – –
Mental Mental Mental Mental 0.61 (26) R
subtraction subtraction subtraction subtraction U: 7, 8; 8–12
S10 C Banana, X Banana Heart Heart Lemon Strawberry Pineapple – – – –
0.91 (23) R, I 0.70 (61) R, I 0.76 (21) R, I 0.90 (21) R, I
U: 1, 2; 8–12 U: 1, 2; 8–12 U: 4, 5; 8–12
S11 C C C Pineapple Ball C C Flower Heart Banana Banana Strawberry
0.63 (30) R, I Mental Mental 0.66 (47) R, I
U: 2; 8–24 subtraction subtraction U: 6; 8–24

Calibration runs (marked with letter ‘‘C’’) and online item selection runs are chronologically listed for each user. If not specified otherwise, motor imagery was used. Target items are shown for each online run. For each successful
selection run the achieved accuracy, the number of scan steps needed (in parenthesis) and whether the row (letter ‘‘R’’) and the target item (letter ‘‘I’’) were selected or not are reported. Classifier updates are listed with the letter ‘‘U’’,
followed by the runs used for CSP and LDA training and the selected frequency band. No additional information except the target item is shown for runs that were not successful, i.e., no selection or incorrect selection was performed.
The marker ‘‘X’’ denotes runs that were aborted. Runs that are better than random are highlighted in bold face.

19
20 R. Scherer et al. / Annals of Physical and Rehabilitation Medicine 58 (2015) 14–22

Fig. 3. Power spectral density (PSD) of raw and cleaned 3-s EEG segments, averaged over target and non-target trials, respectively, for successful runs for selected users.
Idealized EEG and electromyography (EMG) PSDs are shown in the lower left corner. Raw EEG PSD shapes resemble the prototypical EMG PSD. After removing artifacts by
using the FORCe method, the EEG PSDs are similar to the ideal linear EEG curve form.

different target items as function of true positive and true negative numbers of successful, failed and aborted experimental runs
rate (TPR and TNR). For correct interpretation of the results, the performed, respectively, are listed in Table 2. Details on the
performance of each item selection run was compared against experimental protocol and online performance are summarized in
random selection. More specifically, the probability that a random Table 3.
classifier selects the target item by using the same number of scan Examples of power spectral density (PSD) estimates of raw and
steps was calculated. Since there are infinitely many possible cleaned target and non-target EEG segments, respectively, for
random classifiers, an unbiased classifier with TPR = TNR = 0.5 was channel C3 are shown in Fig. 3. PSDs show how the variance of the
chosen. The random level probability was computed by summing EEG is distributed over frequency components. Increased power
the probabilities of all correct selections that occur up to the above 30 Hz, characteristic for motor activity, is clearly visible for
number of scan steps required by the user. The probability that row each user in the raw EEG. Clean EEG PSDs show this characteristic
and item are selected was computed by multiplying the probability to a lesser extent.
by row and column selection. Low probability values (P < 0.05)
indicate that selection was unlikely caused by chance.
4. Discussion
3. Results
The aim of this study was to design an imagery-BCI based row-
Six of eleven users succeeded in selecting the target item with a column scanning communication board for users with CP and to
performance that was better than chance level. A further user rate its performance. Online artifact reduction and evidence
succeeded in correctly selecting the row, but not the item. The accumulation was implemented to allow reasonable control with
 R. Scherer et al. / Annals of Physical and Rehabilitation Medicine 58 (2015) 14–22 21

unreliable input. Evidence accumulation required investigation of calibration and control runs. Users may have gained the impres-
chance level performance. The achieved online performances sion that the BCI just worked out of the box. However, operating an
suggest that seven out of eleven users performed better than imagery-BCI is a skill that the user must learn [6]. Hand motor
chance. This is very encouraging, when compared to results from imagery or counting for item selection demand high cognitive
earlier studies [4,17]. Interestingly, user S10 challenged the BCI. In abilities in comprehension, planning, and execution. These abilities
run 2, the user deliberately selected another target item than the have to be trained as well. Nonetheless, these item selection tasks
one agreed on. We stopped the experiment since the wrong row are promising in users with CP when compared to alternative man-
was selected. The user, however, informed us that this was his machine interfaces such as the visual P300 speller involving an
deliberate decision and that he wanted to test whether or not the even more abstract task design [17].
BCI followed his commands. This shows the complex relationship In summary, users with CP were positive about the system, and
between individuals and machines, and also emphasizes a basic users and caregivers provided useful feedback on further
requirement in BCI: user compliance with the task during early improvements. We are currently adapting the system to the
training. feedback received and preparing for the next series of online
PSD of clean EEG segments have 1/f-like frequency scaling for experiments to create useful and effective BCI-based communica-
the majority of users (Fig. 3). In some users (e.g. S1 and S10) high tion devices.
power for f > 30 Hz is visible, and also class-related differences can
be observed. Involuntary movements are especially noticeable in Disclosure of interest
individuals with a dyskinetic form of CP when attempting to move.
Hence, we hypothesize that motor imagery triggered involuntary The authors declare that they have no conflicts of interest
movement. The next major challenge in this regard consists of concerning this article.
decoupling imagery- and involuntary movement-related activities.
When removing artifacts, two crucial questions arise: Firstly,
Acknowledgements
how many artifacts are still in the signal and secondly, how much
cortical activity was removed? The 4–36 Hz PSD of user S10 shows
This work was supported by the FP7 Framework EU Research
enhanced mu band suppression during hand motor imagery
Project ABC (No. 287774). This paper only reflects the authors’
[15,16], which was not clearly recognizable in the raw EEG. This
views and funding agencies are not liable for any use that may be
suggests that the developed method works at a fundamental level.
made of the information contained herein.
Proper evaluation of the artifact reduction method, however,
turned out to be very complex. Only training (How does activity
change with training?) and finding appropriate user-group related Appendix A. Supplementary data
artifact descriptors (How to quantify artifacts?) would allow
appropriate evaluation.
Supplementary data (video) associated with this article can be
Kinesthetic hand motor imagery and mental arithmetic (for
found, in the online version, at http://dx.doi.org/10.1016/j.plantsci.
example, by counting from 5 backward to 1) were selected for BCI
2004.08.011.
operation. For user S5 motor execution was used to illustrate the
concept of motor imagery. Word generation was not selected
because users were not familiar with the Roman alphabet. Since References
the use of motor imagery seems suitable only for a sub-group of CP
[1] Holm VA. The causes of cerebral palsy: a contemporary perspective. JAMA
users [36,37], important issues that need addressing are identifi- 1982;247:1473–7.
cation of appropriate mental tasks and development of related [2] Reddihough DS, Collins KJ. The epidemiology and causes of cerebral palsy. Aust
imagery training exercises. Users need time to train performing J Physiother 2003;49:7.
[3] Evans P, Evans S, Alberman E. Cerebral palsy: why we must plan for survival.
imagery and to learn to focus attention. Feedback-based motor Arch Dis Child 1990;65:1329–33.
imagery training, however, may be useful for rehabilitation to [4] Neuper C, Müller GR, Kübler A, Birbaumer N, Pfurtscheller G. Clinical applica-
enhance motor planning [38]. tion of an EEG-based brain-computer interface: a case study in a patient with
severe motor impairment. Clin Neurophysiol 2003;114:399–409.
Users and caregivers provided important feedback (interviews
[5] Pfurtscheller G, Neuper C. Motor imagery and direct brain-computer commu-
and open questionnaires) on how to improve usability, most nication. Proc IEEE 2001;89:1123–34.
importantly on the ‘‘worst case’’ sham feedback implemented for [6] Wolpaw JR, Birbaumer N, McFarland DJ, Pfurtscheller G, Vaughan TM. Brain-
calibration runs. This was not well received by CP users and was computer interfaces for communication and control. Clin Neurophysiol
2002;113:767–91.
commonly found to be confusing. A more suitable approach is to [7] Müller-Putz GR, Scherer R, Pfurtscheller G, Rupp R. EEG-based neuroprosthesis
present 100% correct sham feedback. This makes instructions control: a step towards clinical practice. Neurosci Lett 2005;382:169–74.
clearer and system behavior less confusing during early training. [8] Scherer R, Müller-Putz GR, Pfurtscheller G. Flexibility and practicality: Graz
Brain-Computer Interface approach. Int Rev Neurobiol 2009;86:119–31.
Important improvements from a usability point of view include [9] Rao RP, Scherer R. Brain-computer interfacing [in the spotlight]. Signal Process
electrode placement and BCI model parameterization. Due to Mag IEEE 2010;27:150–2.
comparability reasons, the electrode setup from a prior study was [10] Millán JDR, Rupp R, Müller-Putz GR, Murray-Smith R, Giugliemma C, Tanger-
mann M, et al. Combining brain–computer interfaces and assistive technolo-
used [17]. To minimize the risk of electrode failures due to head gies: state-of-the-art and challenges. Front Neurosci 2010;4:161.
support systems, occipital electrodes will be replaced by central [11] Nicolas-Alonso LF, Gomez-Gil J. Brain computer interfaces, a review. Sensors
and frontal ones in future studies. Parameters were manually 2012;12:1211–79.
[12] Kaufmann T, Schulz S, Grünzinger C, Kübler A. Flashing characters with famous
selected in the current study. This enabled us to retain control of
faces improves ERP-based brain–computer interface performance. J Neural
the parameterization. This process will be automated [39,40] and Eng 2011;8:056016.
calibration will become easy and convenient for relatives and [13] Pokorny C, Klobassa DS, Pichler G, Erlbeck H, Real RGL, Kübler A, et al. The
auditory P300-based single-switch brain-computer interface: paradigm tran-
caregivers.
sition from healthy subjects to minimally conscious patients. Artif Intell Med
Meticulous care was taken to ensure that each user understood 2013;59:81–90.
the experimental procedure and the mental imagery tasks to be [14] Müller-Putz GR, Scherer R, Neuper C, Pfurtscheller G. Steady-state somato-
performed. One limitation of the study is, however, that the sensory evoked potentials: suitable brain signals for brain-computer inter-
faces? IEEE Trans Neural Syst Rehabil Eng 2006;14:30–7.
cognitive capabilities of the users were not assessed. A particular [15] Pfurtscheller G, Lopes da Silva FH. Event-related EEG/MEG synchronization
problem may have posed the difference of BCI behavior between and desynchronization: basic principles. Clin Neurophysiol 1999;110:1842–57.
22 R. Scherer et al. / Annals of Physical and Rehabilitation Medicine 58 (2015) 14–22

[16] Neuper C, Scherer R, Reiner M, Pfurtscheller G. Imagery of motor actions: [29] Ramoser H, Müller-Gerking J, Pfurtscheller G. Optimal spatial filtering of single
differential effects of kinesthetic and visual-motor mode of imagery in single- trial EEG during imagined hand movement. IEEE Trans Rehabil Eng 2000;8:
trial EEG. Brain Res Cogn Brain Res 2005;25:668–77. 441–6.
[17] Daly I, Billinger M, Laparra-Hernández J, Aloise F, Garcia ML, Faller J, et al. On [30] Müller-Gerking J, Pfurtscheller G, Flyvbjerg H. Designing optimal spatial filters
the control of brain-computer interfaces by users with cerebral palsy. Clin for single-trial EEG classification in a movement task. Clin Neurophysiol
Neurophysiol 2013;124:1787–97. 1999;110:787–98.
[18] Friedrich EVC, Scherer R, Neuper C. The effect of distinct mental strategies on [31] Blankertz B, Tomioka R, Lemm S, Kawanabe M, Müller K-R. Optimizing spatial
classification performance for brain-computer interfaces. Int J Psychophysiol filters for robust EEG single-trial analysis. Signal Process Mag IEEE 2008;25:
2012;84:86–94. 41–56.
[19] Friedrich EVC, Neuper C, Scherer R. Whatever works: a systematic user- [32] Duda RO, Hart PE, Stork DG. Pattern classification, 2nd ed., Wiley-Interscience;
centered training protocol to optimize brain-computer interfacing individu- 2000.
ally. PLoS One 2013;8:e76214. [33] Belouchrani A, Abed-Meraim K, Cardoso J-F, Moulines E. A blind source
[20] Scherer R, Faller J, Balderas D, Friedrich EV, PrPll M, Allison B, et al. Brain–computer separation technique using second-order statistics. Signal Process IEEE Trans
interfacing: more than the sum of its parts. Soft Comput 2013;17:317–31. 1997;45:434–44.
[21] Fatourechi M, Bashashati A, Ward RK, Birch GE. EMG and EOG artifacts in brain [34] Daly I, Scherer R, Billinger M, Müller-Putz G. FORCe: Fully Online and auto-
computer interface systems: a survey. Clin Neurophysiol 2007;118:480–94. mated artifact Removal for brain-Computer interfacing. Neural Syst Rehabil
[22] Daly I, Pichiorri F, Faller J, Kaiser V, Kreilinger A, Scherer R, et al. What does Eng IEEE Trans 2014. http://dx.doi.org/10.1109/TNSRE.2014.2346621.
clean EEG look like. In: Proceedings of the 34th Annual International Confer- [35] Müller-Putz GR, Scherer R, Brunner C, Leeb R, Pfurtscheller G. Better than
ence of the IEEE Engineering in Medicine and Biology Society; 2012. random? A closer look on BCI results. Int J Bioelectromagnetism 2008;10:
[23] Daly I, Billinger M, Scherer R, Muller-Putz G. On the automated removal 52–5.
of artifacts related to head movement from the EEG. Neural Syst Rehabil [36] van Elk M, Crajé C, Beeren MEGV, Steenbergen B, van Schie HT, Bekkering H.
Eng IEEE Trans 2013;21:427–34. Neural evidence for compromised motor imagery in right hemiparetic cerebral
[24] Scherer R, Pfurtscheller G. Thought-based interaction with the physical world. palsy. Front Neurol 2010;1:150.
Trends Cogn Sci 2013;17:490–2. [37] Chinier E, N’guyen S, Lignon G, Ter Minassian A, Richard I, Dinomais M. Effect of
[25] Zickler C, Riccio A, Leotta F, Hillian-Tress S, Halder S, Holz E, et al. A brain- motor imagery in children with unilateral cerebral palsy: FMRI study. PLoS
computer interface as input channel for a standard assistive technology One 2014;9:e93378.
software. Clin EEG Neurosci 2011;42:236–44. [38] Steenbergen B, Crajé C, Nilsen DM, Gordon AM. Motor imagery training in
[26] Daly I, Josef F, Reinhold S, Catherine S-R, Slawomir N, Marting B, et al. hemiplegic cerebral palsy: a potentially useful therapeutic tool for rehabilita-
Exploration of the neural correlates of cerebral palsy for sensorimotor BCI tion. Dev Med Child Neurol 2009;51:690–6.
control. Front Neuroeng 2014;7:20. [39] Faller J, Vidaurre C, Solis-Escalante T, Neuper C, Scherer R. Autocalibration and
[27] Breitwieser C, Daly I, Neuper C, Müller-Putz GR. Proposing a standardized protocol recurrent adaptation: towards a plug and play online ERD-BCI. IEEE Trans
for raw biosignal transmission. IEEE Trans Biomed Eng 2012;59:852–9. Neural Syst Rehabil Eng 2012;20:313–9.
[28] Müller-Putz GR, Breitwieser C, Cincotti F, Leeb R, Schreuder M, Leotta F, et al. [40] Faller J, Scherer R, Costas U, Opisso E, Medina J, Müller-Putz GR. A co-adaptive
Tools for Brain-Computer Interaction: a general concept for a hybrid BCI. Front brain-computer interface for end users with severe motor impairment. PLoS
Neuroinform 2011;5:30. One 2014;9:e101168. http://dx.doi.org/10.1371/journal.pone.0101168.