INFORMATICA Informatica 1822-8844 0868-4952 0868-4952 Vilnius University INFO1243 10.15388/Informatica.2019.229 Research Article Brain Computer Interface Based Communicator for Persons in Locked-in State VlahinićSašasasa.vlahinic@riteh.hr∗

S. Vlahinić received the BSc and PhD degrees in electronic engineering from University of Trieste, Trieste, Italy, in 1998 and 2003, respectively. Since 2000 he had been young researcher at the Department of Automation, Electronics and Computing, Faculty of Engineering, University of Rijeka, Rijeka, Croatia. Currently, he is a full professor at the same faculty and head of Department of Automation and Electronics. His current research interests are concerned with BCI technologies and EEG signal processing.

 BatistićLuka

L. Batistić received his bachelor’s and master’s degrees in computer engineering (years 2014 and 2017, respectively) from Faculty of Engineering, University of Rijeka. Since 2016, he is a junior researcher in Department of Automation and Electronics, Faculty of Engineering, University of Rijeka. Currently he is also a teaching assistant and a PhD student at the same faculty.

 JadavGuruprasad Madhale

G. Madhale Jadav received his BTech in electronics and instrumentation technology from the Visvesvaraya Technological University, Belgaum, Karnataka, India, in 2013. Since 2014 he has been pursuing his PhD as a young researcher at the Department of Automation and Electronics, Faculty of Engineering, University of Rijeka, Rijeka, Croatia. His current field of research deal with BCI technology and EEG signal processing.

 VrankićMiroslav

M. Vrankić received his diploma degree, as well as MS and PhD degrees from University of Zagreb, Faculty of Electrical Engineering and Computing. Since 1999 he has been with the Department of Electronic Systems and Information Processing, Faculty of Electrical Engineering and Computing, University of Zagreb, Croatia. Since 2003 he has been with the Faculty of Engineering, University of Rijeka, Croatia, where he is currently an associated professor.

 Facutly of Engineering, Universtiy of Rijeka, Vukovarska 58, 51000 Rijeka, Croatia Corresponding author. 2019 112019304781798 102017 52019 © 2019 Vilnius University2019 Open access article under the CC BY license.

Brain Computer Interfaces (BCI) are devices that use brain signals for control or communication. Since they don’t require movement of any part of the body, BCI are the natural choice for assisted communication when a person is unable to move.

In this article, BCI based communicator for persons in locked-in state is described. It is based on P300 brain response of the user, thus does not require prior training, movement or imagination of movement. Auditory paradigm is selected in order to apply the communicator in cases where visual ability is also impaired. The communicator was designed to prove also whether low cost hardware with reduced electrode set could be used efficiently in everyday environment, without the need for expert personnel.

The design of the communicator is described first, followed by detailed analyses of the performance when used by either healthy or disabled subjects. It is shown that auditory paradigm is the primary factor that limits the accuracy of communication. Hardware characteristics and reduced electrode set influence the accuracy in a negative way as well, while different questions and answer types produce no major differences.

 BCI communicator P300 ERP auditory stimuli This work was supported by the Croatian Agency HAMAG-BICRO, grant number POC6-2-32. Introduction

Locked-in syndrome (LIS) is the state in which a person can not consciously control his or her own body, nor it can communicate in any way with others (Wolpaw et al., 2002; Birbaumer and Cohen, 2007). This state could be the consequence of some progressive disease, such as Amyotrophic lateral sclerosis (ALS), or the consequence of a stroke or a brain injury due to an accident. For some of these patients it is hard to establish the state of consciousness because they are unable to show to others that they are conscious and that they understand what others are saying. Therefore, the rates of misdiagnosis, where such patients’ states have been confused with either a coma or a semi-coma, are very high (Schnakers et al., 2008; Guger et al., 2018). In these states, complete immobility is a big problem, but an even bigger problem is the inability to communicate with others (Sellers and Donchin, 2006).

Brain Computer Interfaces (BCI) use neurological signals originating in the brain to control external devices or computers (Wolpaw et al., 2002; Birbaumer and Cohen, 2007) and therefore are the natural choice for assisted communication when a person is unable to move. Neurological electrical signals can be recorded in different ways: from the scalp using electroencephalography (EEG), from the dura mater or cortical surface using electrocorticography (ECoG). Other types of signals and recording devices, such as magnetoencephalography (MEG), positron emission tomography (PET), functional magnetic resonance imaging (fMRI), electromiogram (EMG), and optical imaging such as near infrared spectroscopy (NIRS), are also used. The approaches using EMG have the advantage of using rather small equipment, while at the same time, being very sensitive and able to detect the tiniest muscle movements. In Lesenfants et al. (2016) the authors proposed an EMG-based method which enabled them to assess the consciousness of a patient. However, if no voluntary motor control is possible, then an EMG-based approach cannot be used. For the patients with preserved eye movements, often a very successful approach in establishing communication is to use eye tracking devices (Käthner et al., 2015). In recent years eye tracking devices have become increasingly advanced and allow the system to precisely access which part of the computer screen the person is looking at. These devices, in combination with augmentative and alternative communication software (AAC), such as Tobii Dynavox Communicator 5 or Grid 3,1

Tobii AB, Karlsrovägen 2D, S-182, 53 Danderyd, Sweden.

enable a person to communicate efficiently by using their eyes only. EEG is a primary choice for the BCIs for persons with LIS, to be used at home or on a daily basis, due to its non-invasive nature and reduced dimensions, among all the above mentioned technologies.

Researches of BCIs based on EEG signals are primarily using slow cortical potentials (SCPs) (Birbaumer et al., 2000), periodic signals in 8–30 Hz frequency band – mu rhythm or sensorimotor rhythm (SMR) (Kevric and Subasi, 2017; Nguyen et al., 2015) from sensorimotor cortex generated before movement or during imagination of movement, and event-related brain potentials (ERPs), primarily P300 (Birbaumer and Cohen, 2007; Shen et al., 2015; Sellers et al., 2014). P300 is a positive change in EEG signals originating over the parietal cortex of the brain, and occurs around 300 ms after the expected event or stimulus takes place. P300 is the optimal choice for EEG based BCIs for persons with LIS since it does not require neither training nor imagination of movement. The latter feature is important since it is probable that EEG signals related to movement deteriorate and vanish during a prolonged immobility (Birbaumer and Cohen, 2007).

Farwell and Donchin (1988) first used “oddball” paradigm to elicit the P300 response: an “oddball” is a relatively rare but “targeted” stimulus presented within a series of frequently occurring “non-target” stimuli (Sellers and Donchin, 2006; Sellers et al., 2014; Farwell and Donchin, 1988). The most frequently used experimental set-up is a 6×6 matrix of letters where rows and columns are randomly illuminated. The user focuses on a (desired) letter and when the row or column with the desired letter is illuminated, P300 response occurs. Various designs exist, and recently a four-choice display was proposed, where the four letters flash and the participant attends to the letter corresponding to the answer to the question: “Y” (yes), “N” (no), “P” (pass), or “E” (end) (Sellers and Donchin, 2006; Sellers et al., 2014). In this type of visual P300 based systems, the user still must be able to control eye movements and focus. When the visual system is not preserved, auditory (Hill et al., 2005), vibro-tactile (Guger et al., 2018), somatosensory P300 based systems and even hybrid systems (Ortner et al., 2017) could be used. The latter systems are slower and their effectiveness has not been thoroughly tested (Birbaumer and Cohen, 2007; Halder et al., 2010).

In this paper, a BCI based communicator for persons with LIS is described. It is an auditory based P300 design and therefore it can be used in most severe cases of LIS or complete LIS. The design of the communicator is described in the next section, while analyses of system performance is provided in Section 3. Finally, conclusions are given, along with directions of future research and improvements.

 Communicator Design Auditory Paradigm

In our design of EEG based BCI for person with LIS, we use auditory based P300 “speller”, where the user responds to 5 different words. We have 2 types of questions: “yes/no” questions, and questions with names of the persons, cities, etc. In “yes/no” questions, “yes” or “no” could be desired (“target”) answer, while 3 more “dummy” (“non-target”) words are added in order to resemble oddball paradigm. The words were chosen with an intention to keep the duration of the words short and similar. In questions with names, one name is desired or the “target” name, while the others are “non-target” names. More than two answers were used since it was established that P300 amplitude is inversely related to the relative probability of the evoking stimulus, and directly related to its task relevance (Kachenoura et al., 2008).

During the experiment the possible answers or stimuli were normalized and played at a moderate sound level, with an inter stimulus interval of 1 second, and the stimuli were randomly shuffled in a segment. This inter stimulus interval is much longer than the interval usually used in visual experiments (around 300 ms). This is an obvious disadvantage of auditory paradigm, since long inter stimulus interval increases the time needed to obtain the answer. However, in the case of persons with LIS, the primary scope of the communicator is the establishment of elementary communication or even the determination of the state of consciousness, while the speed of communication is of secondary importance. Total time for reproducing answers for one question was around 3 minutes.

“Yes/no” questions are obviously the simplest ones, providing for binary information only. Questions with names are added for two reasons. First, it allows the multiple choice (e.g. questions like “Which part of your body hurts” or “What would you like to drink/eat?” could be asked in the future). Secondly, in cases when it is necessary to establish the state of consciousness of the person, familiar names could elicit response more easily than non-familiar names (Schnakers et al., 2008) and therefore they can serve in the initial phase of using the communicator by the person with complete LIS. Also, several types of attention enhancement were tried. In Schnakers et al. (2008) it was established that in cases when persons were instructed to mentally count the number of occurrence of “target”, the P300 response was more pronounced with respect to cases where subjects were instructed to give a “mental counting”. In order to test other options, finger counting, tapping and mental tapping have been tested as well. In finger counting type of response the subject was asked to finger count the correct answers, and in tapping the subject was supposed to press the key (on regular PC keyboard), using only one finger, every time he heard the correct answer. In mental tapping type of response, the subject was asked to imagine him tapping the key on a keyboard. Although the options involving the movement of hand or fingers are not options for persons with LIS, testing was done in order to establish whether these types of response have any potential for improvement of P300 detection.

 Hardware and Software

The EEG signals were obtained with Emotiv’s Epoc2

EMOTIV, Inc., San Francisco, CA 94102 USA.

EEG headset. This EEG device was selected due to its low cost and mobility but also because of simple and fast electrodes placement suitable to everyday use, while maintaining the 10–20 international standards of electrode placement. The data recorded using the headset are of good EEG quality and comparable to the data obtained with research grade devices (Badcock et al., 2013). Nonetheless, the comparison between the results obtained with Emotiv Epoc and the research grade device has been conducted, and provided in Section 4.

Emotiv Epoc has 14 bit resolution, EEG data is internally sampled at 2048 Hz and then down sampled to 128 Hz, signal bandwidth is 0.2–43 Hz, and digital notch filters at 50 Hz and 60 Hz are used. The EEG headset consists of 14 gold plated electrodes which make direct contact with the detachable electrode tips. These electrode tips have foam which is soaked in saline solution and sits on the scalp, adapted to the scalp topology. The saline solution acts as a good conductive medium for small voltage fluctuations. The electrodes are pre-attached to the headset with fixed arms to point at the locations: AF3, F3, F7, FC5, T7, P7, O1, O2, P8, T8, FC6, F8, F4, AF4, Common Mode Sense (CMS) at the left mastoid M1 and Driven Right Leg (DRL) at right mastoid M2. Since some electrode positions typically used in P300 detection (e.g. Cz, electrodes in the parietal region) are not available, tests are conducted in order to verify whether the present configuration of electrodes could produce reliable results.

Data acquisition and processing is done in OpenVIBE software platform (Renard et al., 2010). The OpenVIBE’s acquisition client is used to obtain raw EEG data and the algorithm for EEG data processing was built using the OpenVIBE Designer. Apart from being a free open source software, OpenViBE allows us to run the classifier and the whole BCI software in real time as the data is getting acquired from the EEG headset. Data preprocessing includes additional filtering using a 4th order Butterworth band-pass filter with a lower cutoff frequency of 1 Hz and a higher cutoff frequency of 20 Hz, with a pass band ripple of 0.5 dB, and down sampling by a factor of 4. An epoch, a slice of 1 second of data after stimulus onset, is considered in following analyses. No artifact removal was done in this stage, although it can improve the overall performance of the system (Minguillon et al., 2017). Manual artifact removal was not done since the proposed communicator should work in everyday situations where data analysis experts are not available. Automatic artifact removal was not applied since all the data processing needs to be done in real time. Also, answers are repeated 30 times in order to be less affected by discarding some of the segments due to artefacts.

Linear Discriminant Analysis (LDA) classifier (Hoffmann et al., 2008) is used to correctly classify responses of the user. LDA classifier computes a discriminant vector that separates the “target” and “non-target” classes. LDA classifier is simple to use because it is nonparametric. Also, it was established that LDA classifier gives good results for P300 classification (Krusienski et al., 2008).

However, this BCI communicator uses auditory paradigm which is less reliable than visual speller paradigm (Lopez-Gordo et al., 2012; Klobassa et al., 2009) and the used headset has a reduced number of electrodes. Therefore, in order to obtain an acceptable accuracy of classification, it is needed to further enhance and preprocess the data prior to the classification. There are numerous contributions dealing with spatial filtering (Rivet et al., 2009; Rivet and Souloumiac, 2013) and blind source separation (Wolpaw et al., 2002; Albera et al., 2008; Xu et al., 2004; Bayliss and Ballard, 1999; Hill et al., 2004; Wang and James., 2006; James and Hesse, 2005) for P300 speller designs, and in this case spatial filtering based on xDAWN algorithm was used (Rivet and Souloumiac, 2013). xDAWN algorithm is specially designed for P300 speller paradigm.

Only the basic idea behind the algorithm is explained, while more detailed explanations can be found in Rivet et al. (2009), Rivet and Souloumiac (2013). The basic idea is to automatically estimate P300 subspace from raw EEG signals. P300 evoked potentials are then enhanced by projecting raw EEG on the estimated P300 subspace. The algorithm derives its name from the following model for the recorded data X: X=DA+N, where X∈R(Nt×Ns) is the matrix of recorded EEG signals, Ns is the number of sensors, Nt is the number of temporal samples, D∈R(Nt×Ne) is the Toeplitz matrix defined from the set of stimuli onsets and estimated durations of the ERP, Ne is the number of temporal samples of the ERP (1 s in our case), and matrix N is the on-going activity of the user’s brain as well as artifacts. DA represents the synchronous response with target stimuli. The least square estimation of response A, Aˆ=(DTD)(−1)DTX , is different than classical epoching of matrix X when synchronous response A extents over several consecutive stimuli. That is generally the case for visual P300 spellers, but not in our case, where inter stimulus interval is 1 s, which excludes the possibility of overlapping responses. xDAWN algorithm estimates Nf spatial filters (with Nf<Ns ) in order to enhance the synchronous response: XU=DAU+NU, where U∈R(Ns+Nf) is the spatial filter matrix. The idea is similar to principal component analysis (PCA) of Aˆ , where recorded signals are projected on the Nf main components associated with the Nf largest principal values. In xDAWN algorithm, the spatial filters U are designed in order to maximize signal to signal plus noise ratio. Finally, the model (1) can be rewritten as: X=DA′WT+N′, where A′ is synchronous response of reduced dimension, W is its spatial distribution over sensors, and N′ is the noise term. The enhanced signals are then computed by: Sˆ=XUˆ=DA′+N″. N″ is the matrix that includes noise and artifacts, transformed by spatial filtering. Spatial filters can be computed by two QR factorizations and one singular value decomposition, or by using generalized eigenvalue decomposition of pair of covariance matrices (Rivet and Souloumiac, 2013).

 Experimental Procedure

The subjects were briefed about the experiment and asked to keep the movements at minimum to avoid artifacts. They were asked to focus on a fixed point, to make sure that there are minimum eye movement artifacts.

The experiment has a training phase and an operational phase. During the training phase, the subject was asked a question the answer of which was known a priori, and such data were used to train both the spatial filter and the classifier. The training phase was conducted only once at the beginning of the session. During the operational phase the responses were classified in real time. The subject was asked questions and the classifier would classify the data based on the subject’s selection. At the end of each trail healthy subjects were asked to verify the answer, while the disabled subjects were asked only those questions that had as a priori known answers.

A series of 5 sessions was conducted with the same subject on different days, and each session comprised of 5 questions. Once a question was asked, the 5 answers were played by the system in the form of 30 random segments, in such a way that the last stimulus of the previous segment was never the same as the first stimulus of the next segment. This and shuffling of stimuli in the segment ensured that the subject was not able to predict the played stimuli. As there were 30 audio events for each kind of answers, there were a total of 150 stimuli per question which means 150 epochs were considered for classification.

 Analyses of Performance

The system was tested on 39 participants, 7 females and 32 males, aged between 22 and 29. 25 participants were healthy subjects that did not suffer any past physical, neurological or psychiatric disorder, while 14 (12 male and 2 female) participants had some form of disability, varying from highly disabled with no motoric control, to locked in (multiple sclerosis, ALS, traumatic brain injury, dystrophy). On average 6 trials (questions) were recorded per each subject (Table 1). Different test conditions will be explained later in text.

Tests performed for each subject. Number of tests performed with different modes of attention “enhancers” (mental counting (mc), mental attention (ma), finger counting (fc), tapping (tt) and mental tapping (mt)), type of EEG device used and type of questions used are given. Conditions: Traumatic Brain Injury (TBI), Amyotrophic Lateral Sclerosis (ALS), Neuronal Ceroid Lipofuscinosis 2 (CLN2), Muscular Dystrophy (MD), Locked-In Syndrome (LIS).

Subject Condition No. of rec. ma mc fc tt mt Epoc/Quickamp “yes-no”/“names”
H-LBZ – 2 0 2 0 0 0 Quickamp 2/0
H-LBN – 5 0 5 0 0 0 Epoc 5/0
H-IM – 2 0 2 0 0 0 Epoc 2/0
H-DM – 8 2 2 2 2 0 Epoc 4/4
H-DS – 10 2 2 2 2 2 Epoc 5/5
H-DM – 8 2 2 2 2 0 Epoc 4/4
H-FS – 8 2 2 2 2 0 Epoc 4/4
D-IO TBI 4 4 0 0 0 0 Epoc 2/2
H-IP – 8 2 2 2 2 0 Epoc 4/4
H-KJ – 8 2 2 2 2 0 Epoc 4/4
H-KB – 8 2 2 0 2 2 Epoc 4/4
H-LB – 10 2 2 2 2 2 Epoc 5/5
H-LR – 10 2 2 2 2 2 Epoc 5/5
H-ZD – 9 1 2 2 2 2 Epoc 5/4
D-ZF ALS 4 2 2 0 0 0 Epoc 2/2
H-VJ – 10 2 2 2 2 2 Epoc 5/5
D-SD CLN2 5 5 0 0 0 0 Epoc 2/3
H-ST – 10 2 2 2 2 2 Epoc 5/5
H-SK – 8 2 2 2 2 0 Epoc 4/4
H-NA – 8 2 2 2 2 0 Epoc 4/4
H-RN – 10 3 2 2 2 1 Epoc 6/4
H-MJ – 8 2 2 2 2 0 Epoc 4/4
H-MK – 8 2 2 0 2 2 Epoc 4/4
H-MP – 8 2 2 2 2 0 Epoc 4/4
H-AGZ – 4 0 4 0 0 0 Quickamp 3/1
H-CZ – 5 2 3 0 0 0 Quickamp 3/2
H-FZ – 6 3 3 0 0 0 Quickamp 4/2
H-MZ – 6 3 3 0 0 0 Quickamp 4/2
D-IO2 TBI 5 5 0 0 0 0 Epoc 3/2
D-LC MD 2 2 0 0 0 0 Epoc 1/1
D-MK MS 4 2 2 0 0 0 Epoc 2/2
D-ML LIS 4 2 2 0 0 0 Epoc 2/2
D-MZ TBI 4 0 4 0 0 0 Epoc 1/3
D-MJ MS 4 0 4 0 0 0 Epoc 2/2
D-ND MS 4 0 4 0 0 0 Epoc 2/2
D-PC MD 3 0 3 0 0 0 Epoc 2/1
D-SC MD 4 0 4 0 0 0 Epoc 2/2
D-VS LIS 3 0 3 0 0 0 Epoc 2/1
D-MS LIS 5 0 5 0 0 0 Epoc 2/3

Disabled subjects had various types of motor impairments. Not all of them were in LIS, but our primary goal was to test proposed communicator in real world situations (in not ideal situations, e.g. subjects unable to control movements, not fully accessible head positions, lying positions, etc.).

In the following, True Positive rate (TP), False Positive rate (FP) and Accuracy are given. The results are obtained from Open WIBE classifier. TP rate is calculated as a ratio of the number of positive answers correctly classified and the total number of positive answers. FP rate is similarly calculated as a ratio of the negative answers incorrectly classified as positive and the total number of negative answers. Accuracy is calculated as a ratio of the sum of true answers (True Positive and True Negative) and the total sum (Positive and Negative).

Overall average accuracy of classification during the operational phase was 75%. This average accuracy is lower than one reported for visual P300 speller (Sellers and Donchin, 2006; Sellers et al., 2014; Krusienski et al., 2008), which was expected for auditory paradigm (Halder et al., 2010; Simon et al., 2015). The minimum accuracy obtained was 65%, while the maximum accuracy achieved was 90%. Minimum, average, and maximum accuracy for each subject is given in Table 2. The usual accuracy is around 70–80% and we can say that trained subjects (subjects which attended to the experiment more than just a few sessions) tend to perform better.

Table 2 also gives information transfer rate (ITR) calculated as: ITR=log2N+Plog2P+(1−P)log2[(1−P)/(N−1)], where ITR is given in bits/question, N is number of possible targets (5 in our case) and P is classifier accuracy (Wolpaw et al., 1998; Yuan et al., 2013).

Low ITR are obtained, but that was expected since our primary goal is to develop a BCI system that can be used to establish basic communication in cases were more efficient communicator types cannot be used.

Number of recordings, minimum, average, maximum accuracy, standard deviation of the accuracy and information transfe rate (ITR) for each subject during testing. Prefix H in subject code indicates healthy subjects, while D indicates disabled ones.

Subject No. of recordings min acc max acc avg stdev ITR
H-LBZ 2 77.0 80.5 78.8 2.5 1.2
H-LBN 5 73.6 83.0 76.1 3.9 1.0
H-IM 2 75.4 76.0 75.7 0.5 1.0
H-DM 8 74.7 90.4 80.7 4.8 1.2
H-DS 10 69.6 77.6 73.7 2.9 1.0
H-DM 8 71.6 80.1 76.1 2.4 1.1
H-FS 8 70.1 77.1 74.3 2.0 1.0
D-IO 4 72.0 76.0 74.8 1.9 1.0
H-IP 8 73.2 78.1 76.2 1.7 1.1
H-KJ 8 70.0 79.6 74.9 3.5 1.0
H-KB 8 65.1 78.4 75.4 4.3 1.0
H-LB 10 68.8 78.5 74.0 2.8 1.0
H-LR 10 69.6 78.8 73.3 3.0 0.9
H-ZD 9 72.3 78.8 75.2 2.2 1.0
D-ZF 4 70.1 78.3 74.7 4.0 1.0
H-VJ 10 69.9 82.0 75.9 3.0 1.0
D-SD 5 71.2 76.1 74.3 2.1 1.0
H-ST 10 64.7 79.3 73.9 4.4 1.0
H-SK 8 71.6 83.1 77.8 4.2 1.1
H-NA 8 73.9 79.1 75.4 1.6 1.0
H-RN 10 68.5 80.1 74.5 3.4 1.0
H-MJ 8 69.7 78.7 73.9 2.8 1.0
H-MK 8 70.0 80.9 75.4 3.5 1.0
H-MP 8 73.2 78.8 75.0 2.0 1.0
H-AGZ 4 74.1 83.9 79.8 4.4 1.2
H-CZ 5 76.3 89.3 80.7 5.0 1.2
H-FZ 6 78.1 85.1 80.7 2.4 1.2
H-MZ 6 76.5 80.5 78.9 1.4 1.2
D-IO2 5 71.9 78.2 74.7 2.7 1.0
D-LC 2 72.3 75.5 73.9 2.3 1.0
D-MK 4 70.7 74.9 72.7 2.0 0.9
D-ML 4 68.4 78.1 72.6 4.1 0.9
D-MZ 4 69.7 75.5 74.0 2.8 1.0
D-MJ 4 72.4 79.6 75.7 3.0 1.0
D-ND 4 68.7 75.8 72.4 3.0 0.9
D-PC 3 72.3 78.0 75.4 2.9 1.0
D-PC 4 68.7 77.9 74.8 4.2 1.0
D-VS 3 70.6 72.9 72.0 1.2 0.9
D-MS 5 71.3 78.8 75.2 2.9 1.0
Average 71.5 79.3 75.5 2.9 1.0
Hardware Comparison

As already mentioned in Section 2.2, Emotiv Epoc was selected due to its low cost, portability and ease of use, while its major disadvantage was the reduced number of electrodes. Therefore, the same system was tested with a different hardware and electrode set, Quick amp with 32 gel electrode set, and 24 bit A/D converter. In this case, there were more electrodes in the central and the parietal region, which could be beneficial to P300 detection. Table 3 gives the classification accuracy for the two devices, as well as the percentage of true positives (TP) and false positives (FP). Both mean values and standard deviations are given for TP and FP.

Mean values and standard deviations for false positives (FP), true positives (TP), and classification accuracy for Emotiv Epoc and Quickamp.

Epoc Quickamp
FP (%) 12.6 ± 2.9 8.0 ± 3.1
TP (%) 25.4 ± 6.1 31.7 ± 7.1
Accuracy (%) 75.0 79.9

As it can be seen, Quickamp device with more electrodes, especially at central and parietal regions has higher accuracy by more than 4%. More detailed results of comparison between the two hardware amplifiers and electrode sets are given in Fig. 1. It shows the false positives versus true positives scores of measurements acquired during classifier training in auditory P300 experiment with Emotiv Epoc and BrainProducts QuickAmp devices. For each trial or question, 5 fold cross validation was done and the average values of all cross validation results for each question are given in Fig. 1. Ideally, the classification should give results located at the left upper position, corresponding to TP of 1 (or 100%) and FP of 0 (0%).

False positives versus true positives of the results acquired during classifier training in auditory P300 experiment with Emotiv Epoc (x) and BrainProducts QuickAmp devices (◇). The dotted line represents a random guess, a 50-50 guess-miss. Measurements above the line have a greater percentage of guess and measurements below it have a greater percentage of miss.

 Audio vs Visual Stimuli Comparison

In order to perform comparison between audio and visual stimuli performance, seven additional healthy subjects were tested with visual P300 speller and EPOC device. Table 4 gives classification accuracy for auditory paradigm and visual P300 speller, as well as the percentage of true positives (TP) and false positives (FP). Both mean values and standard deviations are given for TP and FP. For visual P300 speller, 6×6 matrix was used in standard set-up: a single row or column was randomly illuminated for 200 ms, and inter stimulus interval was 300 ms. Visual P300 speller is much faster than the proposed auditory paradigm and therefore the number of epochs is greater. All other parameters for preprocessing, xDAWN algorithm and classifier have not changed. Visual P300 speller usually has higher accuracy than one obtained here, but to obtain higher accuracy a greater number of electrodes has to be employed.

Mean values and standard deviations for false positives (FP), true positives (TP), and classification accuracy for auditory and visual experiments.

Auditory Visual
FP (%) 12.2 ± 3.2 2.4 ± 0.3
TP (%) 26.0 ± 6.5 15.8 ± 4.4
Accuracy (%) 74.4 84.0

As expected, visual paradigm has better accuracy, by more than 9%. There is however an interesting characteristic of auditory stimuli; the percentage of true positives is significantly greater than the same measure for visual stimuli. It means that in auditory case, the system is better to correctly classify P300 when it occurs. On the other side, during visual stimuli, misclassification of P300 in its absence occur on average in only 2.4% of cases. This is more clearly visible in Fig. 2. This was expected, since it was reported by the examined persons that subjectively it was much easier to spatially focus attention in the visual experiment than in the auditory experiment.

False positives versus true positives of the measurements acquired during classifier training in auditory (x) and visual (◇) P300 experiment. One trial in auditory experiment is data acquired for one question, and one trial in visual experiment is data acquired for 1 character.

Fig. 3 gives Receiver Operating Characteristic curves for targets, for auditory (left) and visual (right) P300 experiment. Upper left corner represents a perfect classifier with no false positives (100% specificity) and no false negatives (100% sensitivity). Again, it is visible that ROC curve for visual experiment is closer to upper left corner of the ROC space, thus indicating better performance for the visual experiment.

ROC curves for auditory (AUD) and visual (VIS) P300 experiment. Area under ROC curve is 0.61 for auditory experiment and 0.73 for visual experiment.

 Healthy vs Disabled Comparison

The BCI communicator was firstly tested on healthy subjects, and later the system was tested on 14 disabled subjects. Since primarily it was designed to help the disabled persons communicate with their caregivers, it was important to test its applicability in realistic situations. Table 5 gives the comparison between healthy and disabled subjects.

The values presented in Table 5 were obtained in same way as in previous comparisons. The accuracy was slightly smaller for disabled subjects, but often the testing conditions couldn’t be arranged in such an optimal way as for healthy subjects. Therefore these results are suggesting that the approach taken in the system presented was suitable for the disabled subjects. This conclusion is confirmed in Fig 4, showing a clear overlapping of results from healthy and disabled persons.

Mean values and standard deviations for false positives (FP), true positives (TP), and classification accuracy for healthy and disabled persons.

Healthy Disabled
FP (%) 11.8 ± 3.3 13.3 ± 2.5
TP (%) 26.5 ± 6.8 24.3 ± 5.1
Accuracy (%) 75.8 74.2

False positives versus true positives of the measurements acquired during classifier training in auditory (x) and visual (◇) P300 experiment. One trial in auditory experiment is data acquired for one question, and one trial in visual experiment is data acquired for 10 letters.

 Question Type Comparison

In Section 2.1. the reasons were given why two types of questions were used, “yes/no” questions, and “names” questions. It was important to see whether the different types of questions have different classification accuracy. Since training “name” questions include familiar names, it was possible that this type of questions result in a more pronounced and easier to detect P300 response. This may be desirable in the training phase, although it might have provided overoptimistic results. Generally, a longer word could have also caused more variability and more difficulty in the classification. Once again, the comparison of “yes/no” questions and “names” questions, as provided in Table 6 and Fig. 5, show that the system performs in a similar manner for different type of questions.

Mean values and standard deviations for false positives (FP), true positives (TP), and classification accuracy for “yes/no” and “names” questions.

“yes/no” “names”
FP (%) 11.9 ± 3.3 12.4 ± 3.1
TP (%) 26.3 ± 7.2 25.6 ± 5.5
Accuracy (%) 75.7 75.2

False positives versus true positives of the measurements acquired during classifier training in auditory P300 experiment for “yes/no” (x) and “names” (◇) questions.

 Selection Type Comparison

Finally, different modes of attention “enhancers” were tested in order to gain more insights in possible improvements of system performance. Mental counting (mc), mental attention (ma), finger counting (fc), tapping (tt) and mental tapping (mt) were tested. Results given in Table 7 and Fig. 6 were tested with analysis of variance (ANOVA) test which confirmed that there were no significant differences between different forms of attention enhancers (p value of 0.92). Mental counting was easy to perform and an effective way of attention enhancer.

Mean values and standard deviations for false positives (FP), true positives (TP), and classification accuracy for different modes of attention “enhancers” (mental counting (mc), mental attention (ma), finger counting (fc), tapping (tt) and mental tapping (mt)).

ma mc fc tt mt
FP (%) 12.3 ± 2.6 12.0 ± 3.5 12.0 ± 3.8 12.5 ± 3.2 12.0 ± 1.9
TP (%) 25.4 ± 5.4 25.9 ± 6.9 27.2 ± 8.8 25.7 ± 5.7 27.1 ± 4.0
Accuracy (%) 75.3 75.5 75.8 75.1 75.9

False positives versus true positives of the measurements acquired during classifier training in auditory P300 experiment for different modes of attention “enhancers” (mental counting (mc), mental attention (ma), finger counting (fc), tapping (tt) and mental tapping (mt)).

 Conclusions

In this paper, a BCI communicator for persons with LIS is described. It is based on P300 ERPs and auditory paradigm which are suitable for most severe cases of LIS when even visual system cannot be used. The system is also based on low cost, portable, mobile and easy to mount hardware, in order to obtain a communicator for everyday use. With the present design choices, the system has several drawbacks, the major ones being the use of auditory paradigm which is slower and less reliable than the visual one, a smaller number of electrodes and the lack of electrodes in standard positions.

In order to test whether the performance of the system is satisfactory, several comparisons were performed and results were given. It was shown that the auditory paradigm and the selected hardware give lower scores when compared with the visual experiment and with a better hardware, but nonetheless the system still can be used. There is no major difference in the results provided by either healthy or disabled persons. Finally, different choices of questions and answer types have similar results, therefore either choice can be used.

Future improvements include the use of other signal processing techniques for blind source separation and possibly the inclusion of other types of EEG signals, in order to enhance the classification accuracy.

 References Albera, L., Ferréol, A., Cosandier-Rimélé, D., Merlet, I., Wendling, F. (2008). Brain source localization using a fourth order deflation scheme. IEEE Transactions on Biomedical Engineering, 55(2), 490–501. Badcock, N.A., Mousikou, P., Mahajan, Y., de Lissa, P., Thie, J., McArthur, G. (2013). Validation of the Emotiv EPOC EEG gaming system for measuring research quality auditory ERPs. PeerJ, 1, 1–17. Bayliss, J.D., Ballard, D.H. (1999). Single trial p300 recognition in a virtual environment. Proceedings of Soft Computing in Biomedicine. In: CIMA’99, Rochester, USA. Birbaumer, N., Kübler, A., Ghanayim, N., Hinterberger, T., Perelmouter, J., Kaiser, J., Iversen, I., Kotchoubey, B., Neumann, N., Flor, H. (2000). The thought translation device (TTD) for completely paralyzed patients. IEEE Transactions on Rehabilitation Engineering, 8(2), 190–193. Birbaumer, N., Cohen, L.G. (2007). Brain-computer interfaces: communication and restoration of movements in paralysis. The Journal of Physiology, 579(3), 621–636. Farwell, L.A., Donchin, E. (1988). Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. Electroencephalography and Clinical Neurophysiology, 70, 510–523. Guger, G., Spataro, R., Pellas, F., Allison, B.Z., Heilinger, A., Ortner, R., Cho, W., Xu, R., La Bella, V., Edlinger, G., Annen, J., Mandalá, G., Chatelle, C., Laureys, S. (2018). Assessing command-following and communication with vibro-tactile P300 brain-computer interface tools in patients with unresponsive wakefulness syndrome. Frontiers in Neuroscience, 12(423), 1–9. Halder, S., Rea, M., Andreoni, R., Nijboer, F., Hammer, E.M., Kleih, S.C., Birbaumer, N., Kübler, A. (2010). An auditory oddball brain-computer interface for binary choices. CClinical Neurophysiology, 121(4), 516–523. Hill, N.J., Lal, T.N., Bierig, K., Birbaumer, N., Scholkopf, B. (2004). Attentional modulation of auditory event-related potentials in a brain-computer interface. In: Proceedings of the IEEE Internacional Workshop on Biomedical Circuits and Systems, Singapore, pp. 17–19. Hill, N.J., Lal, T.N., Bierig, K., Birbaumer, N., Scholkopf, B.N. (2005). An auditory paradigm for brain-computer interfaces. In: Advances in Neural Information Processing Systems. MIT Press, USA, pp. 569–576. Hoffmann, U., Vesin, J.M., Ebrahimi, T., Diserens, K. (2008). An efficient p300-based brain-computer interface for disabled subjects. Journal of Neuroscience Methods, 167(1), 115–125. James, C.J., Hesse, C.W. (2005). Independent component analysis for biomedical signals. Physiological Measurement, 26, 15–39. Kachenoura, A., Albera, L., Senhadji, L., Comon, P. (2008). ICA: a potential tool for BCI systems. IEEE Signal Processing Magazine, 25(1), 57–68. Käthner, I., Kübler, A., Halder, S. (2015). Comparison of eye tracking, electrooculography and an auditory brain-computer interface for binary communication: a case study with a participant in the locked-in state. Journal of NeuroEngineering and Rehabilitation, 12(76), 1–11. Kevric, J., Subasi, A. (2017). Comparison of signal decomposition methods in classification of EEG signals for motor-imagery BCI system. Biomedical Signal Processing and Control, 31, 398–406. Klobassa, D.S., Vaughan, T.M., Brunner, P., Schwartz, N.E., Wolpaw, J.R., Neuper, C., Sellers, E.W. (2009). Toward a high-throughput auditory P300-based brain-computer interface. Clinical Neurophysiology, 120(7), 1252–1261. Krusienski, D.J., Sellers, E.W., McFarland, D.J., Vaughan, T.M., Wolpaw, J.R. (2008). Toward enhanced P300 speller performance. Journal of Neuroscience Methods, 167, 15–21. Lesenfants, D., Habbal, D., Chatelle, C., Schnakers, C., Laureys, S., Noirhomme, Q. (2016). Electromyographic decoding of response to command in disorders of consciousness. Neurology, 87(20), 2099–2107. Lopez-Gordo, M.A., Fernandez, E., Romero, S., Pelayo, F., Prieto, A. (2012). An auditory brain-computer interface evoked by natural speech. Journal of Neural Engineering, 9, 1–9. Minguillon, J., Lopez-Gordo, M.A., Pelayo, F. (2017). Trends in EEG-BCI for daily-life: requirements for artifact removal. Biomedical Signal Processing and Control, 31, 407–418. Nguyen, T., Khosravi, A., Creighton, D., Nahavandi, S. (2015). Fuzzy system with tabu search learning for classification of motor imagery data. Biomedical Signal Processing and Control, 20, 61–70. Ortner, R., Allison, B.Z., Pichler, G., Heilinger, A., Sabathiel, N., Guger, C. (2017). Assessment and communication for people with disorders of consciousness. Journal of Visualized Experiments, 126, 1–8. Renard, Y., Lotte, F., Gibert, G., Congedo, M., Maby, E., Delannoy, V., Bertrand, O., Lé, A. (2010). OpenViBE: an open-source software platform to design, test, and use brain-computer interfaces in real and virtual environments. Presence: Teleoperators and Virtual Enviornments, 19(1), 35–53. Rivet, B., Souloumiac, A., Attina, V., Gibert, G. (2009). xDAWN algorithm to enhance evoked potentials: application to brain computer interface. IEEE Transactions on Biomedical Engineering, 56(8), 2035–2043. Rivet, B., Souloumiac, A. (2013). Optimal linear spatial filters for event-related potentials based on a spatio-temporal model: asymptotical performance analysis. Signal Processing, 93(2), 387–398. Simon, N., Käthner, I., Ruf, C.A., Pasqualotto, E., Kübler, A., Halder, S. (2015). An auditory multiclass brain-computer interface with natural stimuli: usability evaluation with healthy participants and a motor impaired end user. Frontiers in Human Neuroscience, 8(1), 1–14. Schnakers, C., Perrin, F., Schabus, M., Majerus, S., Ledoux, D., Damas, P., Boly, M., Vanhaudenhuyse, A., Bruno, M.A., Moonen, G. et al. (2008). Voluntary brain processing in disorders of consciousness. Neurology, 71, 1614–1620. Sellers, E.W., Donchin, E. (2006). A P300-based brain-computer interface: initial tests by ALS patients. Clinical Neurophysiology, 117(3), 538–548. Sellers, E.W., Ryan, D.B., Hauser, C.K. (2014). Noninvasive brain computer interface enables communication after brainstem stroke. Science Translational Medicine, 6, 257–272. Shen, J., Liang, J., Shi, J., Wang, Y. (2015). A dynamic submatrix-based P300 online brain-computer interface. Biomedical Signal Processing and Control, 15, 27–32. Wang, W., James., C. (2006). Enhancing evoked responses for BCI through advanced ICA techniques. In: MEDSIP 06, 3rd International Conference on Advances in Medical, Signal and Information Processing, Glasgow, UK, pp. 38–41. Wolpaw, J.R., Ramoser, H˜.N., McFarland, D.J., Pfurtscheller, G. (1998). EEG-based communication: improved accuracy by response verification. IEEE Transactions on Rehabilitation Engineering, 6(3), 326–333. Wolpaw, J.R., Birbaumer, N., McFarland, D.J., Pfurtscheller, G., Vaughan, T.M. (2002). Brain-computer interfaces for communication and control. Clinical Neurophysiology, 113(6), 767–791. Xu, N., Gao, X., Hong, B., Miao, X., Gao, S., Yang, F. (2004). BCI competition 2003-data set iib: Enhancing p300 wave detection using ICA based subspace projections for BCI applications. IEEE Transactions on Biomedical Engineering, 51(6), 1067–1072. Yuan, P., Gao, X., Allison, B., Wang, Y., Bin, G., Gao, S. (2013). A study of the existing problems of estimating the information transfer rate in online brain-computer interfaces. Journal of Neural Engineering, 10(2), 1–11.