Detected Alpha Waves - ROC Curves

In my previous post, I improved my Alpha detection processing by introducing "guard bands", which allowed me to reject most of the non-Alpha activity that looked like Alpha waves.  In finding the best thresholds to use for detecting and rejection, I had to find the right balance between keeping good sensitivity (ie, detecting most of the true Alpha activity) while minimizing false alarms (ie, avoiding detecting non-Alpha activity as if it were Alpha).  Finding the right balance can be tough and can involve lots of guesswork.  In this post, I'll show how that process can be formalized...by making a ROC curve!

Example EEG data (top) and some example processing (middle and botom).  True Alpha activity is limited to the "Eyes Closed" period.  I used the amplitude in the Alpha band (blue trace) to detect the Alpha activity.  I used the amplitude in the Guard Band (green trace) to reject activity that is not specific to the Alpha band.  For my given detection threshold (Alpha > 3.5 uVrms) and rejection threshold (Guard > 2.5 uV), the red circles show when my algorithm detects Alpha activity.

Background:  The plot above shows the final result from the last post.  It shows the raw EEG data in the spectrogram at the top, as recorded from the back of my head (O1 with earlobe reference).  As you can see, when my eyes are closed, the horizontal stripe of color shows my Alpha rhythm.  The middle plot shows the amplitude of the EEG signal in the Guard band (I picked 3-6.5 Hz and 13-18 Hz) and the bottom plot shows the amplitude of the EEG signal in the Alpha band (7.5-11.5 Hz).  You can also see that, by eye, I picked a Guard rejection threshold of 2.5 uVrms and an Alpha detection threshold of 3.5 uVrms.  The red circles show the "detections" when the EEG signal satisfies the Alpha detection threshold (ie, is > 3.5 uVrms) and avoids the Guard rejection threshold (ie, is < 2.5 uVrms).  This works algorithm works way better than the Alpha-only detection approach that I used in my first attempt.  Theis Alpha+Guard algorithm isn't perfect, however.  You see that I still have a pair of false detections around 84 seconds.

Choosing the Thresholds, By Eye:  One of the key decisions in this algorithm is what value to use for the Alpha detection threshold and what value to use for the Guard rejection threshold.  Initially, I choose them by eye from the plots above and then used trial-and-error to tune them further.

Alpha-Guard Scatter Plot:  Another approach for choosing these thresholds is to use a dedicated plot such as the one below.  Here, I take the exact same Alpha and Guard values as plotted above and I re-plot the data.  But, instead of plotting the data as a smooth trace as a function of time, I plot the the data as a scatter plot of the Alpha and Guard value for each moment of time.  For each moment in time when my eyes are closed (which mean that true Alpha waves are likely present), I plotted the data point in blue.  For each data point outside of this time period (so Alpha are likely not present), I plotted the values in red.

For each moment in time, I measure the amplitude of the EEG signal in both the Alpha band and the Guard band.  Each dot in this plot represents the Alpha and Guard values for one moment in time.  The blue dots are from the time when my eyes were closed, so they should contain Alpha activity.  You can see how the Alpha threshold (black line) and Guard threshold (red line) can be used to try to discriminate between the two types of activity.

Choosing the Thresholds, Graphically:  With the Alpha-Guard scatter plot shown above, it is much easier to visually move the Alpha threshold (the horizontal black line) up-and-down, or to move the Guard threshold (the vertical red line) left-and-right.  Any points in the upper-left quadrant satisfy our detection rules (measured Alpha > Alpha Threshold and measured Guard < Guard Threshold).  The thresholds are currently shown at the values from my previous post -- Alpha Threshold at 3.5 uVrms and Guard Threshold at 2.5 uVrms.  We see that the vast majority of the detections are blue dots, which is good because that is the eyes-closed activity.  We also see the two false detections as the two red circles.  With this plot, we clearly see that if we raise our Alpha band a little bit, we could eliminate those two false detections...at the expense of losing more of the desired (blue dot) detections.  Again, we see that it's a balance between high sensitivity and a low rate of false alarms.

Try All Alpha and Guard Threshold Values:  To get a better picture of the effect that these two threshold values have on the sensitivity and false alarm rate, we can have the computer brute-force reprocess the data trying a wide range of threshold values.  By stepping through all the combinations of Alpha and Guard thresholds, the computer forms a dense table of the number of correct detections and the number of incorrect detections for our given EEG recording.  We can then use a contour plot to visualize this dense table of values.  These are shown below.

Effect on Sensitivity: The first plot below shows how the Alpha and Guard thresholds affect our detection sensitivity (the fraction of eyes-closed EEG activity was correctly detected).  The black "x" shows our current threshold values (3.5 uV for Alpha, and 2.5 uV for the Guard).  At the end of my previous post, I said that this achieved a sensitivity of 65%, which agrees with the plot below.  If I want higher sensitivity, this plot clearly shows that the ideal is toward the upper-left, which means lowering my Alpha detection threshold and raising my Guard rejection threshold.

After trying all combinations of Alpha and Guard thresholds (as shown by the black dots), we can see how the threshold values affect our sensitivity in detecting the true Alpha activity.  The highest sensitivity is in the top-left of this plot, meaning we want the lowest threshold for Alpha detection and the highest threshold for Guard rejection.

Effect on False Alarms:  The desire for high sensitivity, however, must be balanced by the need for a low false alarm rate.  The plot below shows how the threshold values affect the number of false alarms for this data set.  As before, the black "x" shows our current threshold values: 3.5 uV for Alpha, and 2.5 uV for the Guard.  In my previous post, I said that this point resulted in 2 false alarms, and the plot below agrees with that finding.  This plot says that if we want to further lower the false alarm rate, we need to move down and to the right, which is to lower our Guard rejection threshold and to raise the Alpha detection threshold.  This is exactly the opposite direction that was needed to increase our sensitivity.  Again, we're seeing that it is a struggle to optimize sensitivity versus false alarms

After trying all combinations of Alpha and Guard thresholds (as shown by the black dots), we can see how the threshold values affect the number of  false alarms (incorrect detection of non-Alpha activity).  The fewest false alarms are seen in the bottom-right of this plot, meaning we want the highest threshold for Alpha detection and the lowest threshold for Guard rejection.

Best Sensitivity for a Given False Alarm Rate:  Given that it is a trade between sensitivity and false alarm rate, how does one choose where to set the detection and rejection thresholds?  One approach would be to simply start by picking a number of false alarms that would seem to be be acceptable.  Then, one can search through the threshold values to find the combination that achieves the best sensitivity at the given false alarm rate.  That's what I did to make the plot below.  I used the dense grid of values before and directly plotted the sensitivity and false alarm count for every combination of threshold values.  Clearly, there is an upper limit running across this plot.  So, I added the green line to define the maximum sensitivity (for this detection algorithm) that can be achieved for any given false alarm rate.

A plot of the sensitivity versus number of false alarms for every combination of threshold values analyzed earlier.  There is clearly a maximum sensitivity value that occurs for each given false alarm rate.  This is Receiver Operating Characteristic (ROC) curve for this detection algorithm.

ROC Curve:  The green line in the plot above is a very useful tool for understanding the performance of the detection algorithm.  It is so useful that it has a name -- it is the "receiver operating characteristic" (ROC) curve for this detection algorithm.  The name is kinda funny because it originated in World War II during the development of radar.  In radar, one is trying to detect pulses of radio energy that have been reflected off distant aircraft.  When comparing different radar receivers, one can always appear to be more sensitive, though at the cost of increasing the rate of false alarms.  So, they developed the ROC curve as a way to make it easier to understand, compare, and optimize detection systems.  It was such an effective tool, that it has became a core technique in the general field signal detection and classification.

Comparing Alpha Detection Systems:  As an example of how useful it is in comparing detection "systems", the ROC curves below show the performance of "Alpha+Guard" algorithm versus my original "Alpha Only" detection algorithm.  As can be seen, the ROC curve for the Alpha+Guard algorithm is higher at point.  It is superior at any allowed false alarm rate.  Alpha+Guard is clearly superior.  Interestingly, this plot also exposes how the "Alpha Only" algorithm was never able to have less than 10 false alarms in this EEG recording.  If one cares about a low false alarm rate (which I certainly do), the ROC curve exposes this critical information.

ROC curves for two different detection algorithms: "Alpha+Guard" detects in the Alpha band and rejects based on the Guard bands, while "Alpha Only" does not include the rejection based on the Guard band.  The ROC curve for "Alpha+Guard" is higher at all false alarm rates, thereby showing its performance is superior.

Selecting the Operating Point:  The ROC curve is an excellent visualization for deciding which detection algorithm to use.  It is also helpful for understanding where the steepest trade-off exists between sensitivity and false alarms.  For the Alpha+Guard algorithm (unlike the Alpha Only algorithm), we see that the curve is generally quite flat, so there is not a severe penalty in targeting a very low false alarm rate.  As an experiment, let's target zero false alarms.  The plot above tells me that, at zero false alarms, it is possible to achieve a sensitivity of 56%.  Fine.  But what threshold values should I use to get this performance?  The ROC curve itself does not tell me that information.

Finding the Threshold Values:  When I made the ROC curve, I built it up by plotting the data generated through a brute-force assessment of all pairs of Alpha and Guard threshold values. These were the green circles shown two plots ago.  The ROC curve came from a subset of those green circles.  So, I can go back and find the threshold values that correspond to the green circles that just fall on the ROC curve.  These threshold values would be the optimal threshold values for a given false alarm rate.  The figure below plots these optimal threshold values as a function of allowed false alarms (ie, allowed number of incorrect detections).  If we are targeting zero false alarms, this plot says that we should choose an Alpha detection threshold of about 3.75 uV and a Guard rejection threshold of about 2.55 uV.

For a given number of allowed incorrect detections (ie, false alarms), this plot shows the Alpha and Guard threshold values that result in the highest sensitivity (ie, most number of correctly identified Alpha activity).  As with all figures in this analysis, the curves above are derived from a single EEG recording, which is why the curves are noisy.

Re-Process the EEG Data:  If we use these two threshold values and reprocess our EEG data, we get the detection performance seen below.  Again, the top plot is the raw EEG recording.  The middle plot shows the EEG amplitude in the Guard band along with (in red) the Guard rejection threshold.  The bottom plot shows the EEG amplitude in the Alpha band along with (in black) the Alpha detection threshold.  The red circles show those points that satisfy both thresholds -- the are the "detections" resulting from our algorithm.  You'll note that there are only detections when my eyes are closed -- there are no false detections.  This looks really good!  I think that I'm getting close to being able to deploy this into my OpenBCI GUI for real-time operation in detecting Alpha waves.

Detection Performance Using the Detection Thresholds for Zero False Alarms.  The detections (rec circles) resulting from this approach look pretty good!

Summary:  In this post I introduced a number of new ways of analyzing and plotting EEG data so that we can optimize the performance of our Alpha detection algorithm.  Specifically, I showed how the ROC curve is a great way of visualizing system performance in a way that smoothly handles the inherent trade-off between sensitivity and false alarms.  I showed how the ROC curve is a simple way to quickly see how one detection algorithm is better than another ("Alpha+Guard" is clearly superior to "Alpha Only") and to quickly see that there is little penalty for driving to an even lower false alarm rate.

Next Steps:  Up until now, we've just been looking at one EEG recording.  Therefore, all of our conclusions on which algorithm to use and which detection thresholds to use have been based upon a very small amount of data.  The results seen here might not generalize well, which means that my algorithm might not perform well when faced with future data.  To address this possibility, my next step is to bring in some of my other EEG recordings, all of which are noisier and more challenging.  With the additional data, we can better optimize our algorithm and have greater confidence that it'll work well on future data that has not been part of its training.

Follow Up:  I examine the performance on six different EEG recordings here.