Brain-Controlled Shark Attack!

Visiting my friends at OpenBCI HQ, we got together to do some hacking.  Since I'm always looking to control new things with my brain, I was really excited to see that someone had brought a remote-controlled shark-shaped balloon (an "Air Swimmers").  This is a very cool toy -- it swims through the air in a wondrous way.  But, I can't just leave a good things alone.  So, after a few hours of hacking, Joel and I were able to turn this simple toy into a 5-person, brain-controlled, SHARK ATTACK!

Approach:  Our approach to this hack is extremely similar to the approach that we used for our multi-person control of a toy robot.  As shown in the figure below, the idea is that you get multiple players hooked up to a single EEG system (OpenBCI, in my case).  The computer processes the EEG data looking for each person's eyes-closed Alpha waves.  Depending upon which person's Alpha waves are detected, the computer sends commands to the shark.  The commands are conveyed to the shark via an Arduino, which is driving the shark's remote control.  The end result is that the shark swims because of one player's brain waves. I think that's pretty cool.

Two-Person Demo:  As Joel and I were pulling this hack together, we started to test it using just the two of us.  Being just two people, we could only do shark two commands, not all five.  It was still pretty fun, though.  I love the sense of excitement that happens when a hack first starts to work.

Hacking the Remote Control:  To make this shark controllable from the computer, we needed to hack into the shark's remote control.  Like when I hacked the remote for the toy robot, Joel found that the remote for the shark was simply a few push buttons that were wired to pull one side of the switch down to ground whenever the button was pushed.  So, to make this controllable from my computer, Joel soldered some wires to the circuit board (to the high side of each switch) to allow an Arduino to pull it down to ground instead of having to push it with your finger.  As a result, we can now send a command to the Arduino and cause the shark to move.  Our Arduino code for this hack is on GitHub here.

We modified the shark's remote control by adding a wire to the non-grounded side of each push button.
We brought the wires out and connected them to an Arduino.

An Arduino drives the shark's remote control.

EEG Electrode Setup:  It's quite easy to record a person's eyes-closed Alpha waves.  You need three electrodes.  Put one electrode on the back of your head (O1 or O2, if you know the 10-20 system), put the EEG reference electrode onto your earlobe, and put the EEG bias electrode on your other earlobe.  You can see some examples in the photo below, where we had three people controlling the shark.  We used the gold cup electrodes and Ten20 electrode paste that came with the OpenBCI kit.

Three-Man "Team Alpha!" Controlling the Shark.  You can also see
where we put the electrodes -- back of head and both earlobes.

OpenBCI Setup:  We are going to wire up multiple people to control this shark.  And to be clear, it is not normal to hook multiple people to one EEG system.  But that is what we are going to do.  This is definitely using EEG in a non-traditional way.  That's why this is called "hacking".  The trick to making it work is to tell the EEG system (in this case, OpenBCI) that each person has his own EEG reference electrode.  OpenBCI enables this by allowing each EEG channel to be run in "differential mode", where you use each channel's the "P" and "N" inputs as a differential pair.  This is in contrast to the more-usual "common reference mode", where we use one of the SRB inputs as a common EEG reference for all EEG channels.  To change OpenBCI to differential mode, you use the OpenBCI GUI, via the "Chan Set" tab, to change each channel's "SRB1" and "SRB2" setting to "off".  Then, in the main window, turn off all of the other channels that you are not using.

Screenshot Showing How to Configure for Five Channels in
Differential Mode...simply Turn Off SRB2.

Plugging Into OpenBCI:  Once you've got the EEG electrodes on the individual players, you've got to hook them into the OpenBCI board.  Because we're in "differential mode", each player will get one "P" input and one "N" input.  For this hack, we put the electrode from back of the head into the "N" input.  We then put the left earlobe into the corresponding "P" input.  Finally, the right earlobe was connected to a bias pin.  Because we had five players, there aren't enough bias pins available on the OpenBCI board.  I used a 16-channel OpenBCI board because it has 4 bias pins, so that covered four players.  The fifth player simply plugged into the analog ground pin ("AGND"), which is not as good as using a bias pin, but it worked well enough.

Wiring Electrodes to the OpenBCI Board for Five Players.  Note that the board only
has 4 bias connections, so one player is attached to AGND instead.

EEG Processing Algorithms:  As mentioned earlier, the PC does all of the EEG processing...no processing occurs on the OpenBCI board itself.  For the software on the PC, we started with the stock OpenBCI GUI.  Then, I extended it by (1) adding the Alpha detection algorithms and by (2) adding code to send shark commands to the Arduino.  This variant of the OpenBCI GUI is currently saved here on GitHub as a branch of the main repository.  As you can tell from the class names shown in the code, the code is based heavily upon the previous work with the HexBug robot...which you may find confusing since we're controlling a shark and not a HexBug.  Sorry for the confusion!

Three-Person Testing:  After Joel and I did our two-person testing, we roped in a couple of other players to join Joel.  That got us to a three-man shark attack!

Three-Person Brain-Controlled Shark

Shark Food:  While we were all hacking the shark to make it brain-controlled, Conor was busy doing his own hacking.  Once we finally got our brain-controlled shark into fighting condition, we couldn't resist swimming it over to harass Conor.  Conor was pretty sure that his teeth were sharper than the shark's, so he wasn't much afraid.

Conor Faces Off Against the Shark.

Four-Person Shark Control:  Having successfully used three people to control the shark, we wired up a fourth person.  Have you ever tried to get four people doing anything in a smooth and coordinated fashion?  It's hard!  But, we did have success...

Five-Person Shark Attack:  OK, if four people working together is hard, five people is just chaos. In case you can't seem him, notice below that the fifth guy is in the center of the crowd, kneeling so that you just see his head popping above the bench.  The wiring on the OpenBCI electrodes seems generously long when you're just attaching one person.  With 5 people, though, you really need longer wires...or you simply need a little creativity on how you pack the people together.

Five-Person Brain-Control of the Swimming Shark.  The fifth person is kneeling
and you can only see his head.  We need longer wires!

Swimming Away:  By the time that we got this all working, it was really late at night.  The time stamps on the pictures show that it was about midnight, and we'd been at OpenBCI HQ since about 10AM.  So, between the fatigue and all the caffeine, I think that we were getting some funny brain wave behavior.  For example, one guy was making some weak Alpha even with his eyes open.  Wacky!  Regardless, we were able to get the shark to swim around...until we swam the shark too far to one end of the OpenBCI HQ...

Swimming the Shark Off Into the Sunset.

...at which point the shark's IR remote control could no longer communicate with the shark.  Stranded Shark!  And so our night of EEG hacking ended.  Still, it was mighty fine work,  Go Team Alpha!

Update 2015-03-09: I just saw that someone has already made a BCI for controlling this very same swimming shark (Chen et al.  "Recreational devices controlled using an SSVEP-based Brain Computer Interface (BCI)").  Note that they used one person to control the shark via SSVEP, which is exactly what I did with my brain-controlled Hex Bug!

Update 2015-09-24: Wow!  I was given an opportunity to write an article for IEEE Spectrum for their Oct 2015 issue.  How cool is that?  You can check it out here: "OpenBCI: Control An Air Shark With Your Mind".

Update 2015-11-08: I see that Wired (magazine) posted their nicely-done video on our shark hacking.  It's quite an enjoyable piece.  Good work, Wired!.

Arduino Control of a Hex Bug

Based on my previous success with visual entrainment, I'm moving forward with my plans on making a brain-computer interface (BCI) using my blinky lights.  To really kick this project into high gear, though, I need a good goal -- I need something that would be really fun to control with my brain.  Luckily, my OpenBCI friend Conor, found these cool remote-controlled 6-legged robots that can walk around a fire a little gun.  Bingo!  As you can see below, today's post shows how to hack the robot's remote to make it controllable from an Arduino.  Once it is controllable from an Arduino, it's only one more step until it's controllable from my brain!

The Hex Bug Battle Spider:  The remote-controlled robot that Conor found is a Hex Bug Battle Spider.  They are available in two colors and you can have them do battle.  Hex Bug makes smaller and cheaper versions of this robot, but I believe that only the Battle Spider has the ability to do battle.  I'm looking forward to facing off cerebro-a-cerebro with Conor, so I'm sticking with the Battle Spider.

Hex Bug Battle Spider - My Hacking Target for Today

IR Remote Control:  The Hex Bug is commanded using an infrared (IR) remote-control.  It is the remote control that I will hack so that the Hex Bug can be commanded from an Arduino.  As you can see in the picture below, I was so anxious to hack the remote, that I never got a picture of it while it was still in one piece...I just couldn't wait to smell the solder!  Regardless, as you can see, the remote is simply a bunch of plastic pieces, a couple of coin cell batteries (not shown), and a printed circuit board (PCB).

The Hex Bug Infrared Remote Control (in Pieces)

Thank You, Test Points!  Flipping over the PCB, I was very pleased to see that this board has test points for everything.  Oh, the joy!  Because of these test points, it is much easier to probe the board with my multimeter or with an oscilloscope to figure out how this thing works.  The test points also makes it much easier to attach wires to control this thing from the outside (like from my Arduino).  It turns out that there is a test point for each of the four user buttons on this board.  The relevant test points are shown in the picture below.  These test points will be the focus of my work.

All those test points enable easy hacking.  The test points
with the arrows are for the user buttons.

How the Buttons Work:  After a bit of probing of these test points, I learned that the buttons are used like most buttons in small devices like this (see "Button" demo by the Arduino folks for more info).  The buttons on this remote control are simply switches that are normally open-circuit.  When you press a button, the switch becomes closed-circuit.  The "low" side of each button is tied to ground.  The "high" side is connected to 3.3V via a pull-up resistor (probably inside the microcontroller).  The microcontroller is continually sensing the voltage on the high side of the switch.  When the button is not pressed, no current flows through it nor through the pull-up resistor, so the voltage seen by the micro is high.  When the button is pressed, current flows through the button, which drops the voltage seen by the micro.  As a result, the micro knows that the button was pressed.  Easy!

Hacking Approach:  Based on this discussion, it is clear that the microcontroller on the remote control knows nothing about the buttons...it only knows about the voltage being controlled by the buttons.  When one of those lines goes from high to low, the microcontroller thinks that a button has been pressed.  My hacking approach, therefore, is to wire the Arduino to the remote control so that the Arduino can pull the lines low for me.  This requires me to attach a wire to the high side of each button and to a attach a wire to the remote control's ground.  Normally, I'll keep the Arduino's pins in a high-impedance state so that no current flows.  When I want it to "press" a button for me, I'll command the relevant pin to go into a low-impedance state to allow it to conduct current to ground.  Electrically, this will mimic the behavior of the buttons themselves.  No extra components will be necessary!

Connecting Ground:  OK, we're mostly done talking.  Now let's start soldering.  First, I connected a wire to the remote's ground.  After looking around the PCB, I decided that I liked the solder that was on the low side of the "fire" button.  So, as you can see below, I sneaked a black wire into that location and soldered it in place.

A blurry picture showing where I soldered a black wire to
attach to the remote control's ground.

Connecting Each Button:  Then, I flipped the board over and soldered a colored wire to each button's test point.

All of my wires are now soldered to the test points.

Snip a Pass-Through for the Wires:  While it is not necessary to do this, I like the idea of re-assembling the remote so that I can use it with my fingers (as if it were not modified) or so that I can use it with the Arduino.  To enable the reassembly of the remote, one simply has to cut a hole in the plastic housing to get the wires out.  I used a "nibbler" tool to cut a small hole.  As you can see below, it worked really well!

For extra credit, I used a nibbler to cut a hole in the plastic housing so that
I can get the wires out, even after I fully re-assemble the remote control.

Attach a Pin Header:  To ease the connection of these 5 wires to the Arduino, I decided to solder the free ends of the wires to a piece of basic pin header.  With these pins, I can easily insert the five wires as a single unit into the sockets on the Arduino board.

To make it easier to connect the wires to an Arduino, I attach the wires to
a basic pin header.

The Hacked Remote:  The picture below shows the hacked remote after I reassembled it.  I tested it by pressing the buttons with my fingers and the robot moved.  So far, so good!

My hacked remote control is now re-assembled and ready for testing.

Software:  If I'm going to command this robot from my Arduino, the Arduino needs software.  So, I plugged in my remote to the Arduino (because the Arduino is really flexible, I used the Analog Input pins even though these signals are neither Analog nor are they Inputs...but that doesn't matter, you can use the Analog pins as Digital ins and outs, too) and then began coding.  My code is available on my GitHub as "TestHexBugController".  This code tells the Arduino to listen to commands coming over Serial from the PC.  I assign one ke on the PC's keyboard to each of the Hex Bug's four functions: "forward", "turn left", "turn right", and "fire".  When the Arduino receives one of these commands, it toggles the relevant pin to pull it LOW for 500 msec.  That's all it takes!

Using the hacked remote so that I can use an Arduino to control my Hex Bug via
commands entered from the PC.

Testing it Out:  As you can see in the video at the top of this post, this setup works pretty well for controlling the Hex Bug.  I can make it walk anywhere and shoot its gun on command.  It's pretty fun.  If I cared to, I could no script a whole series of maneuvers.  If I attached a pen to the Hex Bug, maybe I could make it walk around on a big piece of paper so that it would draw out a funny picture.  That could be fun.  Or, I could combine it with some computer vision on my PC and have it chase my cat around.  That would definitely be fun.  Sadly, I don't know anything about computer vision.

Brain Control:  Really, though, my next step is to control this with my brain.  So, I'll attach my OpenBCI shield to this same Arduino and I'll have the Arduino pipe my EEG signal to the PC.  On the PC, I'll process the EEG signal and, if it detects the right brainwave signatures, I'll have the PC send robot commands back to the Arduino.  The Arduino will then convey those commands to the robot via the IR remote.  All of the hardware pieces are in place...now it's time to put it all together!

Follow-Up:  I finally did put all the pieces together.  I can now control the Hex Bug with my brain waves!