Sunday, June 8, 2014

Controlling a Hex Bug with my Brain Waves

Ever since my effort with OpenBCI began, I've been looking to control something with my brain.  Sure, a while back, I was successful in lighting an LED with my brain waves, but that's pretty simple.  I wanted something more.  And now I can do it.  I can control a robot with my mind!  Yes!


Approach:  My robot has just a few actions that it can do...turn left, turn right, walk forward, and fire.  To make this brain-controlled, I need a way to invoke these commands using signals from my brain.  Ideally, I'd just think the word "Fire!" and the robot would respond.  Unfortunately, those kinds of brain waves are too hard to detect.  Instead, I need to use brain waves that are easy to detect.  For me, "easy" brain waves include the Alpha waves (10 Hz oscillations) that occur when I close my eyes, as well as the brain waves that occur when I watch my blinking movies (a.k.a. visual entrainment).  So, my approach is to use OpenBCI to record my brainwaves, to write software to detect these specific types of brain waves, and to issue commands to the robot based on which brain waves are detected.

Here are all the pieces that you see in the video
Hardware Setup:  The core hardware for this hack is similar to my usual OpenBCI setup: EEG electrodes, an OpenBCI board, an Arduino Uno, and my computer. Added to this setup is the Hex Bug itself and its remote control, which I hacked so that the remote can be controlled by an Arduino.  So, as shown below, my brain wave signals go from my head all the way to the PC.  The PC processes the EEG data looking for the Alpha waves or the visually-entrained waves.  If any are detected, it decides what commands to give the robot.  The commands are conveyed back to the Arduino, which then drives the remote control, which the Hex Bug receives over its usual IR link.

Here is the schematic of how the pieces work together.

EEG Setup:  I'm going to be measuring my Alpha waves and I'm going to be measuring the brain waves induced through visual entrainment.  Based on my previous experience, I know that both are best recorded using an electrode on the back of the head (at the "O1" position, if you're into your 10-20 electrode placement standard).  I do not need electrodes all over my head.  That's the only sensing electrode that I'm using.  That's it.  Of course, EEG also requires a reference electrode, which I put on my left earlobe.  And, finally, EEG often has a third electrode ("bias" or "driven ground"), which I placed on my right earlobe.

Looking at the Frequency of my Brain Waves:  As mentioned above, my approach is to control my robot by detecting Alpha waves and by detecting visually-entrained brain waves.  These are easily detectable because they occur at specific frequencies.  Alpha occur around 10 Hz and the visually-entrained brain waves occur at the blink rate(s) of whatever movies I use (my best results were from 5 Hz and 7.5 Hz movies). So, to control my robot, I will be looking for EEG signals at these frequencies: 5 Hz, 7.5 Hz, and 10 Hz.  I'm going to "look" for these frequencies by writing some EEG processing software that'll look at the frequency content of my EEG signal to see if these frequencies are present.


EEG Processing:  The flow chart above shows the steps that I use to process the EEG signal (my software is here).  Once the PC gets EEG data from the OpenBCI board, the first step is to compute the spectrum of the signal, which tells me the content of the EEG signal as a function of frequency.  I then search through the relevant part of the spectrum (4-15 Hz) to find the peak value.  I note both its frequency value and its amplitude.  In parallel, I also compute the average EEG amplitude across the 4-15Hz frequency band.  This average value is my baseline for deciding whether my peak is tall (strong) or short (weak).  By dividing the amplitude of my peak by this baseline value,  I get the signal-to-noise ratio (SNR) of the peak.  The SNR is my measure of the strength of the peak.  The output of the EEG processing, therefore, are two values: the frequency of the peak and the SNR of the peak.


Deciding My Robot's Action:  Once my EEG processing finds the frequency and SNR of the peak in my EEG spectrum, I now have to decide how to act on that information.  After some trial and error, I settled on the algorithm shown in the flow chart above.  It's got three steps:
  • SNR Check:  First, I decide whether the current peak in the spectrum is legitimate, or if it is likely to be just noise.  I don't want to issue a command if it is just noise because then my robot will be taking all sorts of actions that I didn't intend.  That is not what I want.  So, to decide if the peak is likely to be legitimate, I look at the SNR of the peak.  If it has a big SNR, I'll accept it as a legitimate peak.  If it is too small, I'll take no further action.  Right now, my threshold for this decision is at 6 dB.  Setting a higher threshold results in fewer false commands (which would be good), but it also makes the system less sensitive to legitimate commands (which is bad).  This 6 dB threshold resulted in an OK (but not great) balance.
  • Frequency Check:  If the peak seems legitimate, I decide how to command the robot based on the frequency of the peak.  If the peak is between 4.5-6.5 Hz, I must be looking at the right-side of my 2-speed blinking movie (ie, the portion that blinks at 5 Hz), so the computer prepares the "Turn Right" command.  Alternatively, if the EEG peak is 6.5-8.5 Hz, I must be looking at the left-side of my 2-speed blinking movie (ie, the portion that blinks at 7.5 Hz), so it prepares the "Turn Left" command.  Finally, if the EEG peak is 8.5-12 Hz, it must be my eyes-closed Alpha waves, so the computer prepares the "Move Forward" command.
  • New Command Check:  Before issuing the command, I check to see whether this command is the same as the last command that was extracted from my brain waves.  If the latest command is different, I hijack the command and, instead, issue the "Fire!" command.  If the latest command is the same, I go ahead and issue the left / right / forward command like normal.  The reason for this hijack is that I have no other type of easily-detected brain wave that I can use for commanding the robot to fire.  This approach of issuing "Fire!" on every change in command seemed like a decent way of getting a 4th command out of 3 types of brain waves.
Putting It All Together:  As you can see in the movie, I eventually able to get all of these pieces working together to allow me to command the Hex Bug using just my brain waves.  Of course, it didn't work the first time.  Even once I got all the hardware working, I still needed to tune a bunch of the software parameters (FFT parameters and the detection threshold) until I got something that worked somewhat reliably.  To help with this tuning process, I used the spectrum display that is in my Processing GUI.  Some screen shots are below.

Example EEG spectrum when I stared at the right side of my two-speed blinking
movie.  It induced 5 Hz brain waves.  I programmed 5 Hz to mean "Turn Right".
The SNR here is between 6 and 7 dB.

Here's an example EEG spectrum when I stared at the left side of my two-speed
blinking movie.  It induced 7.5 Hz brain waves.  When the GUI detected 7.5 Hz,
it issued a "Turn Left" command to the Hex Bug.  The SNR is only 6-7 dB.

Finally, here's an example EEG spectrum with my eyes closed so that I was
exhibiting Alpha waves, which are near 10 Hz.  When it detected 10 Hz, I
programmed it to issue a"Forward" command.  The SNR is > 8 dB.

Weak Peaks:  In the screenshots above, the red line shows the current EEG spectrum.  The heavy black circle shows the spectral peak that my software algorithms have detected.  The black dashed line is the "background noise" from which the SNR is computed.  To be declared a legitimate detection, the peak must be 6 dB higher than the black dashed line (unfortunately, I don't show this on the plot...sorry!).  As can be seen, the 5 Hz and 7.5 Hz examples are not very strong (the SNR is only 6-7 dB).  Other peaks within the plots are very close to being the same size, which would cause false commands to be sent to the robot.  In my movie at the top of this post, there were several false commands.

Balancing Sensitivity with False Commands:  To reduce the number of false commands, I could raise my detection threshold above 6 dB. Unfortunately, as see in the first two spectrum plots above, my 5 Hz and 7.5 Hz peaks are usually pretty weak (<  7 dB).  Therefore, any attempt to raise my detection threshold above 6 dB would cause me to no longer detect my legitimate brain waves.  I know because this is exactly the tuning process that I tried.  Bummer!  So, if I want more reliable performance, I'll need to develop a fancier signal processing beyond this simple FFT-threshold approach.  Future challenges!

Wrapping Up:  Even with the false commands seen in my movie, I was still able to command the robot to move around the table.  I could get it to go (roughly) where I wanted it to go.  And, I did it all with just my brain waves.  I think that this is pretty exciting!  Yay!  What are the next steps?  Well, maybe now that I have this under my belt, I can move on to control flying fish, or maybe a quadcopter!  Do you have any other cool ideas for things I can control with my brain?

Coolness:  This hack got picked up by IEEE Spectrum as part of an article on OpenBCI.  Cool!  Check it out here.

More Coolness:  This hack also got picked up by Wired.  Fun!

Follow-Up: I got to share this hack with Joel and Conor of OpenBCI.  You can see their luck with controlling the robot here.

11 comments:

  1. Hi Chip,
    It's a great demo. You are getting there. I'm amazed by your work putting in the Processing sketch for other to use. I probably try your approach with Matlab to see how will it turn out.
    Also, how about reducing the detection threshold and adding a confirmation method (feedback) to the system before issuing a command? For example, the system will ask for a double eye blink in a certain interval before issuing the command. It will not be a purely EEG system though.

    ReplyDelete
    Replies
    1. Hey, thanks for reading! I appreciate your feedback!

      Regarding the use of a confirmation method, I think that's a great idea. Unfortunately, my experience with lowering the detection threshold below 6 dB is that there are SO MANY detections, that you'd be having to continually confirm/reject candidate actions. That would be such a pain.

      The best approach is to improve the SNR of the of the signal that you want to detect...in this case, it is the SNR of my visually-entrained brain waves (the ones at 5 Hz and 7.5 Hz) that need to be made stronger. One can always improve the SNR through additional averaging, but that slows down the response time. Other ideas that I've had include:

      * Brighter blinking lights
      * Blinking lights with more light-dark contrast
      * Blinking lights closer to my eyes, such as on eye glasses

      Or, maybe the SNR could be improved through using multiple EEG electrodes. If I used multiple electrodes on the back of my head (say O1 and O2), perhaps the visually-entrained signal would be coherent between the two electrodes, whereas the "noise" (background activity) might not be coherent. If I exploit the coherence, I might get another dB or two (best case for two electrodes would be +3 dB on SNR).

      So, those are the ideas that I have for improving the reliability of the commands. Keep sending me your ideas!

      Delete
    2. I just read an IEEE paper on steady-state visual evoked potential (SSVEP) like I'm doing here with my blinking movies. Like I mentioned in my own comment above, they suggested that multiple EEG electrodes could improve performance.

      In my suggestion above, I said that I could seek to use additional channels where the SSVEP signal might be coherent (ie, common) between the channels. By adding the channels together, I could boost the amplitude of the SSVEP signal, which would increase its SNR and make its detection more reliable.

      Conversely,iIn the IEEE paper, they suggested using channels where the SSVEP signal is NOT common but where the background EEG activity (ie, the "noise") might be common. They suggest pairing O1 with PO1, Oz with POz, or O2 with PO2. By differencing these pairs, you can maybe cut the noise amplitude, which would be an equally effective way to achieve my goal of increasing the SNR.

      Cool idea!

      Delete
  2. Fantastico Chip: as I can I will try for a dancing robot!
    All the best!
    Giorgio

    ReplyDelete
    Replies
    1. Thanks for the kind words. If do you make a dancing robot, be sure to share a video! I can't wait!

      Chip

      Delete
  3. Chip, hey cool. Nice going.

    Checkout this video posted by the g.tec guys. They use flashing checkerboard icons in the 4 corners, and it looks like a single sensor at POz or Pz. (Not Oz !) Not sure where their reference is, video said ground was approx. Fz.

    http://vimeo.com/channels/miranda/88151780

    ReplyDelete
    Replies
    1. Great link!

      Clearly, we're both using ssvep as our input method. I like how they're doing 4 blink rates and not just 2, like I'm using.

      I also like hoe they're using the bci to select whole musical phrases. That's the only approach that makes sense, given the speed (slowness) of bci's, but it is an approach that had never occured to me. Very smart.

      Finally, I like how they had the bci-decided musical phrases go to human players. They could have easily had the computer play the musical phrases, but the human players make the whole endevour more compelling.

      Thank for sharing!

      Chip

      Delete
  4. Wonder what it is about the complementary checkerboard patterns they find effective? Could it be that is more "attention grabbing" as far as the various levels of visual processing? Or less subject to interference from the neighboring patterns? The checkerboard matrix dimensions may also be an important variable.

    They may be sampling farther from Oz to tap into these higher level abstraction effects.

    re: human players, yeah I bet the resultant mix is ten times better than with strictly mechanical playback. Because of all the nuances quartet players would add to segue, blend, harmonize with each other. In other words, they listen and respond to the other players, huge. I'm a big advocate of music improv, see lightfield.com/iii.htm .

    ReplyDelete
    Replies
    1. My understanding is that the checkerboard, with all of its horizontal and vertical lines, stimulates the edge-detection portion of the brain's circuitry in addition to the simple light-dark contrast-detection circuits. As a result, people think that they get a stronger overall response.

      I did some testing of my own using a checkerboard, but I didn't see much difference. The large-area light-dark blink worked better for me. But, maybe my checkerboard sizing wasn't ideal.

      Re: human players, I totally agree that the resultant music was far more engaging than it would have been with computerized playback. For us mere mortals without g.tec funding to pay the musicians, it is fun to think how we all could hack together something similar, as long as we're OK using computerized playback of the EEG-directed music.

      Chip

      Delete
  5. Hey Chip. Really great stuff man! Check your SoundCloud...or email me at mike.tate@me.com
    I have something I would like to talk with you about.

    ReplyDelete
  6. "Do you have any other cool ideas for things I can control with my brain?" - Yes I do. It has not been done yet and is waiting for someone like you to help develop it! Get in touch with me.
    Mike Tate

    ReplyDelete