Two Brains - One Robot

After my success with sharing the brain-controlled hex bug with Conor and Joel, we brainstormed on how we could make this hack even more fun.  We decided that the main problem with this hack is that only one person gets to participate -- the person driving the robot.  The solution?  Let's hook up multiple people at the same time to control the one robot.  It'll be like that 3-legged race, where you tie your leg to the leg of another person, and then you stumble together in slapstick hilarity until you both get to the finish line.  We are going to do the same thing, but with brain-controlled robots.  Here's how far we've gotten so far...

The Plan:  Our goal is to have multiple people control one robot via their brain waves.  To do this, we aimed to connect multiple people to a single OpenBCI board.  I have never connected multiple people to one EEG system before, so this was pretty exciting for me.  As shown in the figure below, the idea is that each player is responsible for just one of the robot's actions -- one player is responsible for "Turn Left", another for "Turn Right", etc.  Since the robot has four actions (Left, Right, Forward, Fire), we can have up to four players.

The Hexbug robot has four commands (Left, Right, Forward, Fire), so for multi-player fun,
connect four people to one OpenBCI board and work cooperatively!

Commanding the Robot:  In setting up this hack, I wanted to make it as easy as possible for the players to command the robot with their brain waves.  The easiest brain waves to generate and the easiest brain waves to detect are Alpha rhythms (ie, 10 Hz oscillations), specifically the Alpha rhythm that naturally occurs when you close your eyes.  So, with the setup above, we have the computer looking for Alpha waves in each person's EEG signal.  If the computer sees Alpha waves from Player 1, the computer issues a "Turn Left" command to the robot.  If the computer sees Alpha waves from Player 2, it issues a "Forward" command.  And so on...

EEG Setup:  To detect these eyes-closed Alpha waves, we put one electrode on the back of a player's head over the visual cortex (position "O1" in the 10-20 system).  We put another electrode on one ear lobe to act as the EEG reference.  Finally, we put a third electrode on the other ear lobe to act as the EEG Bias.

Individual Reference:  To allow each person to use their own reference electrode, we configured the software on the OpenBCI board to put the ADS1299 EEG chip into per-channel differential mode.  Unlike our normal mode of operation, which uses a common reference electrode via SRB1 or SRB2, this differential mode allows each channel (ie, each player) to have its own reference.  This is what we want!  We simply plug the O1 electrode into the channel's "P" input and the ear lobe reference into the channel's "N" electrode.

Common Bias:  The only tricky part is that we want all four players to be connected to the OpenBCI Bias.  This is tricky because the OpenBCI board does not have four Bias pins.  Well, as you can see below, all it takes is a soldering iron and you can connect a piece of pin header to turn the single Bias pin into four Bias pins.  Now we're hacking!

OpenBCI V3 Board With Extra Pins Soldered to the Bias Output

Connecting the Pieces:   The picture below shows all the connections to the OpenBCI board assuming three players.  On the lower left, we've got three pairs of wires (one pair for each player) plugged into the "P" and "N" inputs of three different channels.  Then, in the upper-left, you see three wires plugged into three of the four new Bias pins.  Finally, in the upper-right, you see five wires that go off to command the hacked Hexbug remote control.

OpenBCI Board with Connections Ready for Three Players

Making It Happen:  Being a rare thing that me and Joel and Conor are all together, it was really fun that we could work together to make this hack happen.  Joel worked the soldering iron to attach the pins and he modified the Arduino code running on the OpenBCI board to enable the per-channel differential mode.  Conor further modified the Arduino code as well as the Processing GUI to enable slower turning of the robot (originally, it was turning WAY too fast).  Then, I modified the Processing GUI to enable Alpha detection on the four individual players.  We did all this in parallel.  I'd never really done group-hacking before.  It was definitely fun.

Conor and Joel working through the details of the connecting the Hexbug remote control.

Testing It:  Once we pulled together all of the pieces, Conor and I began to test the complete setup (see pic below).  After a little tweaking, we got the whole system working, as shown in the video at the top of this post.  It was a group effort that worked out.  Pretty sweet.

Conor and Chip -- Two Brains, One Robot.

Breaking Robots:  So our original vision was to get this hack working so that we could have *two* 4-person teams, with each team controlling their own robot.  Luckily, we had multiple robots and multiple OpenBCI boards, so we thought that we could make it happen.  Unfortunately, as soon as Conor and I made our video, the robots started to break.  They don't like being stuffed in suitcases, I guess.  So, we were left with just one working robot.  Bummer.

Recruiting a Team:  At the AF LabHack, there were lots of folks doing their own hacking.  By the time we got our system working (with the one healthy robot), the other teams were scrambling to get there last results prior to presenting to the group...so we had a tough time recruiting volunteers for being part of a robot-control team.  In the short time we had left, we did get three enthusiastic folks step up.  We got them all equipped with EEG electrodes, tuned the system a bit and let them play!

Our Fine Volunteers.  Three Brains, One Robot.  

No Video:  At this point, we should be presenting a triumphant video.  Unfortunately, we don't have one.  If we did, what you'd see is that two of the three players could easily and repeatably use their eyes-closed Alpha waves to command the robot.  It was cool to see.

No Alpha:  The third player, though, did not have much luck controlling his part of the robot.  At first, I assumed that it was a problem with our system, but after a little debugging, I came to the conclusion that his brain simply wasn't generating eyes-closed Alpha.  He could have been trying too hard (you must be relaxed, without concentrating or being overly focused), or he could have been part of the 11% of the normal, healthy population that simply does not generate Alpha upon closing their eyes [Ref 1].  For these folks, I've got to come up with an alternate robot-control methodology...perhaps by the concentration signature of counting-backwards-by-three.

Next Steps:  The next steps are clear -- I have to get a bunch of people together, hook them up, and enjoy the shenanigans of many brains trying to control a single robot.  Should be fun!

Ref [1]: Gibbs FA, Gibbs EL, Lennox WG. Electroencephalographic classification of epileptic patients and control subjects. Arch Neurol Psychiatry. 1943;50:111–28, as referenced by http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3927247/

Follow-Up:  We used a similar approach to get a 5-person team to brain-control a swimming shark balloon.  It's cool.  Check it out here.

Sharing the Brain-Controlled Hex Bug

I got to meet up with Joel and Conor (of OpenBCI) for some hacking over the weekend.  We were at a hackathon sponsored by the Air Force Research Laboratory.  During some of the down-time between hackathon events, we got to do some hacking of our own.  Since Joel and Conor had never seen my brain-controlled hex bug up close, I brought out my stuff.  And nerdy-fun shenanigans ensued.

Brain-Controlled Robots Rule!

The EEG and computer setup was exactly the same as when I did it earlier -- one electrode on the back of the head near O1, the reference electrode on one ear lobe, and the bias electrode on the other electrode.  We used my same blinking movies to induce brain waves at 5 Hz and 7.5 Hz, and we the normal eyes-closed response to induce Alpha waves (which are near 10 Hz).  After playing around with the detection thresholds, we were able to get the system to work for both Joel and Conor.

As a spectator, I really enjoyed the tension and drama provided by Joel's showmanship:

And I also enjoyed the authority of Conor's brain-control skills:

...until his skills failed...

It was really fun to share this hack with Joel and Conor.  Personally, I find that making something move out here in the real world (like this toy robot) is way more fun than simply making traces move on a computer screen.  Sharing hardware hacks is where it's at.  Hardware hacking, FTW!

Controlling a Hex Bug with my Brain Waves

Ever since my effort with OpenBCI began, I've been looking to control something with my brain.  Sure, a while back, I was successful in lighting an LED with my brain waves, but that's pretty simple.  I wanted something more.  And now I can do it.  I can control a robot with my mind!  Yes!

Approach:  My robot has just a few actions that it can do...turn left, turn right, walk forward, and fire.  To make this brain-controlled, I need a way to invoke these commands using signals from my brain.  Ideally, I'd just think the word "Fire!" and the robot would respond.  Unfortunately, those kinds of brain waves are too hard to detect.  Instead, I need to use brain waves that are easy to detect.  For me, "easy" brain waves include the Alpha waves (10 Hz oscillations) that occur when I close my eyes, as well as the brain waves that occur when I watch my blinking movies (a.k.a. visual entrainment).  So, my approach is to use OpenBCI to record my brainwaves, to write software to detect these specific types of brain waves, and to issue commands to the robot based on which brain waves are detected.

Here are all the pieces that you see in the video

Hardware Setup:  The core hardware for this hack is similar to my usual OpenBCI setup: EEG electrodes, an OpenBCI board, an Arduino Uno, and my computer. Added to this setup is the Hex Bug itself and its remote control, which I hacked so that the remote can be controlled by an Arduino.  So, as shown below, my brain wave signals go from my head all the way to the PC.  The PC processes the EEG data looking for the Alpha waves or the visually-entrained waves.  If any are detected, it decides what commands to give the robot.  The commands are conveyed back to the Arduino, which then drives the remote control, which the Hex Bug receives over its usual IR link.

Here is the schematic of how the pieces work together.

EEG Setup:  I'm going to be measuring my Alpha waves and I'm going to be measuring the brain waves induced through visual entrainment.  Based on my previous experience, I know that both are best recorded using an electrode on the back of the head (at the "O1" position, if you're into your 10-20 electrode placement standard).  I do not need electrodes all over my head.  That's the only sensing electrode that I'm using.  That's it.  Of course, EEG also requires a reference electrode, which I put on my left earlobe.  And, finally, EEG often has a third electrode ("bias" or "driven ground"), which I placed on my right earlobe.

Looking at the Frequency of my Brain Waves:  As mentioned above, my approach is to control my robot by detecting Alpha waves and by detecting visually-entrained brain waves.  These are easily detectable because they occur at specific frequencies.  Alpha occur around 10 Hz and the visually-entrained brain waves occur at the blink rate(s) of whatever movies I use (my best results were from 5 Hz and 7.5 Hz movies). So, to control my robot, I will be looking for EEG signals at these frequencies: 5 Hz, 7.5 Hz, and 10 Hz.  I'm going to "look" for these frequencies by writing some EEG processing software that'll look at the frequency content of my EEG signal to see if these frequencies are present.

EEG Processing:  The flow chart above shows the steps that I use to process the EEG signal (my software is here).  Once the PC gets EEG data from the OpenBCI board, the first step is to compute the spectrum of the signal, which tells me the content of the EEG signal as a function of frequency.  I then search through the relevant part of the spectrum (4-15 Hz) to find the peak value.  I note both its frequency value and its amplitude.  In parallel, I also compute the average EEG amplitude across the 4-15Hz frequency band.  This average value is my baseline for deciding whether my peak is tall (strong) or short (weak).  By dividing the amplitude of my peak by this baseline value,  I get the signal-to-noise ratio (SNR) of the peak.  The SNR is my measure of the strength of the peak.  The output of the EEG processing, therefore, are two values: the frequency of the peak and the SNR of the peak.

Deciding My Robot's Action:  Once my EEG processing finds the frequency and SNR of the peak in my EEG spectrum, I now have to decide how to act on that information.  After some trial and error, I settled on the algorithm shown in the flow chart above.  It's got three steps:

• SNR Check:  First, I decide whether the current peak in the spectrum is legitimate, or if it is likely to be just noise.  I don't want to issue a command if it is just noise because then my robot will be taking all sorts of actions that I didn't intend.  That is not what I want.  So, to decide if the peak is likely to be legitimate, I look at the SNR of the peak.  If it has a big SNR, I'll accept it as a legitimate peak.  If it is too small, I'll take no further action.  Right now, my threshold for this decision is at 6 dB.  Setting a higher threshold results in fewer false commands (which would be good), but it also makes the system less sensitive to legitimate commands (which is bad).  This 6 dB threshold resulted in an OK (but not great) balance.

• Frequency Check:  If the peak seems legitimate, I decide how to command the robot based on the frequency of the peak.  If the peak is between 4.5-6.5 Hz, I must be looking at the right-side of my 2-speed blinking movie (ie, the portion that blinks at 5 Hz), so the computer prepares the "Turn Right" command.  Alternatively, if the EEG peak is 6.5-8.5 Hz, I must be looking at the left-side of my 2-speed blinking movie (ie, the portion that blinks at 7.5 Hz), so it prepares the "Turn Left" command.  Finally, if the EEG peak is 8.5-12 Hz, it must be my eyes-closed Alpha waves, so the computer prepares the "Move Forward" command.

• New Command Check:  Before issuing the command, I check to see whether this command is the same as the last command that was extracted from my brain waves.  If the latest command is different, I hijack the command and, instead, issue the "Fire!" command.  If the latest command is the same, I go ahead and issue the left / right / forward command like normal.  The reason for this hijack is that I have no other type of easily-detected brain wave that I can use for commanding the robot to fire.  This approach of issuing "Fire!" on every change in command seemed like a decent way of getting a 4th command out of 3 types of brain waves.

Putting It All Together:  As you can see in the movie, I eventually able to get all of these pieces working together to allow me to command the Hex Bug using just my brain waves.  Of course, it didn't work the first time.  Even once I got all the hardware working, I still needed to tune a bunch of the software parameters (FFT parameters and the detection threshold) until I got something that worked somewhat reliably.  To help with this tuning process, I used the spectrum display that is in my Processing GUI.  Some screen shots are below.

Example EEG spectrum when I stared at the right side of my two-speed blinking
movie.  It induced 5 Hz brain waves.  I programmed 5 Hz to mean "Turn Right".
The SNR here is between 6 and 7 dB.

Here's an example EEG spectrum when I stared at the left side of my two-speed
blinking movie.  It induced 7.5 Hz brain waves.  When the GUI detected 7.5 Hz,
it issued a "Turn Left" command to the Hex Bug.  The SNR is only 6-7 dB.

Finally, here's an example EEG spectrum with my eyes closed so that I was
exhibiting Alpha waves, which are near 10 Hz.  When it detected 10 Hz, I
programmed it to issue a"Forward" command.  The SNR is > 8 dB.

Weak Peaks:  In the screenshots above, the red line shows the current EEG spectrum.  The heavy black circle shows the spectral peak that my software algorithms have detected.  The black dashed line is the "background noise" from which the SNR is computed.  To be declared a legitimate detection, the peak must be 6 dB higher than the black dashed line (unfortunately, I don't show this on the plot...sorry!).  As can be seen, the 5 Hz and 7.5 Hz examples are not very strong (the SNR is only 6-7 dB).  Other peaks within the plots are very close to being the same size, which would cause false commands to be sent to the robot.  In my movie at the top of this post, there were several false commands.

Balancing Sensitivity with False Commands:  To reduce the number of false commands, I could raise my detection threshold above 6 dB. Unfortunately, as see in the first two spectrum plots above, my 5 Hz and 7.5 Hz peaks are usually pretty weak (<  7 dB).  Therefore, any attempt to raise my detection threshold above 6 dB would cause me to no longer detect my legitimate brain waves.  I know because this is exactly the tuning process that I tried.  Bummer!  So, if I want more reliable performance, I'll need to develop a fancier signal processing beyond this simple FFT-threshold approach.  Future challenges!

Wrapping Up:  Even with the false commands seen in my movie, I was still able to command the robot to move around the table.  I could get it to go (roughly) where I wanted it to go.  And, I did it all with just my brain waves.  I think that this is pretty exciting!  Yay!  What are the next steps?  Well, maybe now that I have this under my belt, I can move on to control flying fish, or maybe a quadcopter!  Do you have any other cool ideas for things I can control with my brain?

Coolness:  This hack got picked up by IEEE Spectrum as part of an article on OpenBCI.  Cool!  Check it out here.

More Coolness:  This hack also got picked up by Wired.  Fun!

Follow-Up: I got to share this hack with Joel and Conor of OpenBCI.  You can see their luck with controlling the robot here.

Follow-Up: Follow-Up:  We used a similar approach to get a 5-person team to brain-control a swimming shark balloon.  It's cool.  Check it out here.

Arduino Control of a Hex Bug

Based on my previous success with visual entrainment, I'm moving forward with my plans on making a brain-computer interface (BCI) using my blinky lights.  To really kick this project into high gear, though, I need a good goal -- I need something that would be really fun to control with my brain.  Luckily, my OpenBCI friend Conor, found these cool remote-controlled 6-legged robots that can walk around a fire a little gun.  Bingo!  As you can see below, today's post shows how to hack the robot's remote to make it controllable from an Arduino.  Once it is controllable from an Arduino, it's only one more step until it's controllable from my brain!

The Hex Bug Battle Spider:  The remote-controlled robot that Conor found is a Hex Bug Battle Spider.  They are available in two colors and you can have them do battle.  Hex Bug makes smaller and cheaper versions of this robot, but I believe that only the Battle Spider has the ability to do battle.  I'm looking forward to facing off cerebro-a-cerebro with Conor, so I'm sticking with the Battle Spider.

Hex Bug Battle Spider - My Hacking Target for Today

IR Remote Control:  The Hex Bug is commanded using an infrared (IR) remote-control.  It is the remote control that I will hack so that the Hex Bug can be commanded from an Arduino.  As you can see in the picture below, I was so anxious to hack the remote, that I never got a picture of it while it was still in one piece...I just couldn't wait to smell the solder!  Regardless, as you can see, the remote is simply a bunch of plastic pieces, a couple of coin cell batteries (not shown), and a printed circuit board (PCB).

The Hex Bug Infrared Remote Control (in Pieces)

Thank You, Test Points!  Flipping over the PCB, I was very pleased to see that this board has test points for everything.  Oh, the joy!  Because of these test points, it is much easier to probe the board with my multimeter or with an oscilloscope to figure out how this thing works.  The test points also makes it much easier to attach wires to control this thing from the outside (like from my Arduino).  It turns out that there is a test point for each of the four user buttons on this board.  The relevant test points are shown in the picture below.  These test points will be the focus of my work.

All those test points enable easy hacking.  The test points
with the arrows are for the user buttons.

How the Buttons Work:  After a bit of probing of these test points, I learned that the buttons are used like most buttons in small devices like this (see "Button" demo by the Arduino folks for more info).  The buttons on this remote control are simply switches that are normally open-circuit.  When you press a button, the switch becomes closed-circuit.  The "low" side of each button is tied to ground.  The "high" side is connected to 3.3V via a pull-up resistor (probably inside the microcontroller).  The microcontroller is continually sensing the voltage on the high side of the switch.  When the button is not pressed, no current flows through it nor through the pull-up resistor, so the voltage seen by the micro is high.  When the button is pressed, current flows through the button, which drops the voltage seen by the micro.  As a result, the micro knows that the button was pressed.  Easy!

Hacking Approach:  Based on this discussion, it is clear that the microcontroller on the remote control knows nothing about the buttons...it only knows about the voltage being controlled by the buttons.  When one of those lines goes from high to low, the microcontroller thinks that a button has been pressed.  My hacking approach, therefore, is to wire the Arduino to the remote control so that the Arduino can pull the lines low for me.  This requires me to attach a wire to the high side of each button and to a attach a wire to the remote control's ground.  Normally, I'll keep the Arduino's pins in a high-impedance state so that no current flows.  When I want it to "press" a button for me, I'll command the relevant pin to go into a low-impedance state to allow it to conduct current to ground.  Electrically, this will mimic the behavior of the buttons themselves.  No extra components will be necessary!

Connecting Ground:  OK, we're mostly done talking.  Now let's start soldering.  First, I connected a wire to the remote's ground.  After looking around the PCB, I decided that I liked the solder that was on the low side of the "fire" button.  So, as you can see below, I sneaked a black wire into that location and soldered it in place.

A blurry picture showing where I soldered a black wire to
attach to the remote control's ground.

Connecting Each Button:  Then, I flipped the board over and soldered a colored wire to each button's test point.

All of my wires are now soldered to the test points.

Snip a Pass-Through for the Wires:  While it is not necessary to do this, I like the idea of re-assembling the remote so that I can use it with my fingers (as if it were not modified) or so that I can use it with the Arduino.  To enable the reassembly of the remote, one simply has to cut a hole in the plastic housing to get the wires out.  I used a "nibbler" tool to cut a small hole.  As you can see below, it worked really well!

For extra credit, I used a nibbler to cut a hole in the plastic housing so that
I can get the wires out, even after I fully re-assemble the remote control.

Attach a Pin Header:  To ease the connection of these 5 wires to the Arduino, I decided to solder the free ends of the wires to a piece of basic pin header.  With these pins, I can easily insert the five wires as a single unit into the sockets on the Arduino board.

To make it easier to connect the wires to an Arduino, I attach the wires to
a basic pin header.

The Hacked Remote:  The picture below shows the hacked remote after I reassembled it.  I tested it by pressing the buttons with my fingers and the robot moved.  So far, so good!

My hacked remote control is now re-assembled and ready for testing.

Software:  If I'm going to command this robot from my Arduino, the Arduino needs software.  So, I plugged in my remote to the Arduino (because the Arduino is really flexible, I used the Analog Input pins even though these signals are neither Analog nor are they Inputs...but that doesn't matter, you can use the Analog pins as Digital ins and outs, too) and then began coding.  My code is available on my GitHub as "TestHexBugController".  This code tells the Arduino to listen to commands coming over Serial from the PC.  I assign one ke on the PC's keyboard to each of the Hex Bug's four functions: "forward", "turn left", "turn right", and "fire".  When the Arduino receives one of these commands, it toggles the relevant pin to pull it LOW for 500 msec.  That's all it takes!

Using the hacked remote so that I can use an Arduino to control my Hex Bug via
commands entered from the PC.

Testing it Out:  As you can see in the video at the top of this post, this setup works pretty well for controlling the Hex Bug.  I can make it walk anywhere and shoot its gun on command.  It's pretty fun.  If I cared to, I could no script a whole series of maneuvers.  If I attached a pen to the Hex Bug, maybe I could make it walk around on a big piece of paper so that it would draw out a funny picture.  That could be fun.  Or, I could combine it with some computer vision on my PC and have it chase my cat around.  That would definitely be fun.  Sadly, I don't know anything about computer vision.

Brain Control:  Really, though, my next step is to control this with my brain.  So, I'll attach my OpenBCI shield to this same Arduino and I'll have the Arduino pipe my EEG signal to the PC.  On the PC, I'll process the EEG signal and, if it detects the right brainwave signatures, I'll have the PC send robot commands back to the Arduino.  The Arduino will then convey those commands to the robot via the IR remote.  All of the hardware pieces are in place...now it's time to put it all together!

Follow-Up:  I finally did put all the pieces together.  I can now control the Hex Bug with my brain waves!