Raspberry pi robot: real-time object detection using remote Tensorflow server

Finished product first:

The video above shows real-time objection detection using a neural network model called Single Shot MultiBox Detector (SSD). The neural network model was trained on (and runs with) Tensorflow.

The “AI” model works much better than older Computer Vision (OpenCV) techniques. For example, here’s an example of a previous attempt to track objects with the color “tomato”. The way that worked was by filtering a camera frame for a specific color, then masking out that color to find the contour of an object.

There were many problems with this approach. For example, color-based detection is sensitive to lighting changes. It performs differently depending on whether it’s day or night time. Also, we get false detections because many other objects in the scene may have the same color.

But, one advantage of classic OpenCV techniques is they’re simple and fast. Fast enough to run in real time on a live camera feed on a tiny Raspberry Pi processor.

On the other hand, AI models are more accurate and powerful. They can detect many different objects in various orientations (even if they’re partially hidden or chopped out of frame). They’re less sensitive to lighting changes. 

But the downside is they’re expensive to run. The Raspberry Pi doesn’t have quite enough horsepower. I tried it and it’s possible… but I wasn’t happy with the performance. I found some interesting USB AI accelerators like Google’s Coral but they’re pricey. 

So I decided to go with the “Mars Rover” approach and use a remote inference server (running on a PC in my living room). This buys us “infinite” computing power for a fixed cost of network latency. As long as your network is fast, this scheme works well.

One thing going for us is even though the Raspberry Pi camera can capture HD images, the SSD AI model only needs 300×300 pixel inputs. So we can downsize the images before sending them over the network to improve performance.

Here’s the Raspberry Pi code. All it does is capture frames, encode them as Base64 strings and makes requests to our remote inference server: 

Now for the object detection neural network. Google provides a set of pre-trained models for object detection in their Model Zoo. The model I picked was ssdlite_mobilenet_v2_coco. You’ll also need this file to convert the detection IDs to text labels: coco-labels-paper.txt 

I made a Python module to load and run the pre-trained Tensorflow object detection model. The class Predictor in coco_predictor.py  below loads the saved model and provides a predict() function for running inference on new input images.


Finally, the Flask inference server that runs on a PC. The Raspberry Pi sends it images and it replies with detections:


Oh, and if you’re curious about the Robot Kit I’m using, it’s made by Adeept. They call it the Mars Rover PiCar-B:

Mars Rover PiCar-B

The hardware is excellent and well-designed. All the pieces fit together perfectly and mount onto a solid acrylic chassis with bolts. There is a steering rack and a RWD drivetrain that sends power from one motor to both rear wheels. You can pretty much follow the instructions and everything fits together like a Lego set. And when you’re not using it in robot mode, it’s a good looking “case” that lets you use it like a regular Raspberry Pi on your desk.

The software, however, is so-so. You definitely need programming experience. Some things might not work right out of the box, and some things need fiddling around with. On the plus side, they do provide tons of working code for all the various sensors, servos and components. So as long as you are comfortable with Python, you can use their code as excellent references. You can pick and choose and mix and match what you need.

I like this robot kit because it uses a Raspberry Pi (which I already have). So it uses standard Linux stuff. You can use state-of-the-art software and write regular programs like a civilized person. And because it uses a Pi, you get WiFi, Ethernet, Bluetooth, USB, HDMI, etc for free. No painful caveman Arduino programming. It runs on CR 18650 lithium ion batteries (which I also have from salvaging old laptop batteries). It can also run on the regular Pi USB power source.

“Behind the scenes”: Driving school. Programming the head.

I hope to make it autonomous and self-recharging one day!

Happy hacking!

How to make a webapp/server that can read & react to emails

In the last post, we talked about how to teach your Google Home device (or Google Assistant on your Android phone) to send emails.

But that itself is not too interesting (unless you really enjoy spamming yourself). What would be more useful is if you create a program to read and act on the emails automatically. Because, then, you have the ingredients for building your own Virtual Assistant.

The simplest (and cheapest way) I found to do this is using Google App Engine. It has a generous daily free quota. For all my home automation projects, I haven’t run above the free limit.

Start by following the “hello world” instructions to make your first web app:

Then, add the files/code below to enhance your web app to handle emails. (Note: the code I provided below is an example of how I use email to control my home sprinkler system. This lets me say “Turn on sprinkler 1” to my Google Home Mini, which triggers IFTTT to send an email to my web app, which reads the email and triggers my home sprinkler system to turn on zone 1.)




After you set up the above, you can send emails to your Google App Engine web app using its special address:

Now that you have a web app, there’s much more you can do beyond handling emails. It opens up a couple more opportunities for your home automation. E.g.:

  • Use this web app as a trusted HTTP server that your smart devices can fetch instructions from
  • Use this web app to serve a web GUI for your devices/projects
  • Use this web app (which supports HTTPS) to authenticate users and securely control your smart devices

Give it a go. I think the code I gave you above should work. If I missed something, please let me know at aaron@secretsciencelab.com. Happy hacking!

How to teach Spyno to spy on (almost) anything

Spyno likes JSON. It’s a JSONivore. So when you make a Spyno agent, you need to feed it a source URL in JSON format.

But what if you want to spy on things that are not in JSON? No worries. We have two tricks:

Trick 1. Yahoo Query Language YQL free service

“Use YQL to convert XML to JSON & vice versa. Access atom, rss, micro formats and more. You can even load CSV files from anywhere.”

So whether your favorite site is in RSS or XML, no problem. Just send it to Google Feed to transform it to JSON.

For example, say one of your guilty pleasures is Perez Hilton (don’t ask):

  1. Find the RSS feed URL (E.g., http://i.perezhilton.com/?feed=rss2)
  2. Drop the following query into this YQL form:

  3. That returns you the following Endpoint, which you can copy-paste into Spyno

Trick 2. Google Feed API

“With the Feed API you can download any public Atom, RSS, or Media RSS feed using only JavaScript, so you can easily mash up feeds with your content and other APIs.”

Trick 3. Kimonolabs API

Kimono is like the Swiss Army kitchen sink that turns any website into a JSON feed. It’s designed for scraping websites. But what’s special about it is that you can teach it what you want scraped. All you have to do is open the website you’re interested in using Kimono, then click elements on the page you want:

And these two tricks are about all you need to make JSON feeds for your new Spyno Agent.

Happy scraping, Spyno trainers!

How to view Spyno from anywhere

When you’re on your PC, open Spyno by clicking on him at the top right of Chrome:



But what if you’re not at your PC? Visit your Public URL, of course!

Here’s how you get the Public URL for your Spyno page:

  1. Type chrome://extensions in your Chrome’s address bar. Hit Enter.
  2. Visit Spyno’s options page
  3. Click


  4. Scroll to the bottom. Click “Get code”:


  5. Sign in with Google and you’ll see this:
    Code Page

    The link marked (1) is your public URL. Send that to your phone to view Spyno from your phone. Copy the code marked (2) to your PC’s Spyno option page. Spyno uses that code to update your public page.

    Paste your code here and click Save

    Paste your code here and click Save

  6. Ta-da!

Now you can take Spyno with you anywhere.

Next: How to teach Spyno to spy on (almost) anything

Secret Spyno: How to customize your Spyno page layout

Spyno Agents appear in columns, like a newspaper. The number of columns you see depends on the width of your screen. Agents are drawn left to right until the edge of the screen. Then they continue on the left in the next row, like a typewriter.


1 | 2 | 3 | 4
5 | 6 | 7 | 8
9 |

To control the order in which your Agents are drawn, edit the Layout in the options page. Layout takes 2 formats:

1) a list

2) a list of tabs

Make your own Spyno page

This is what the tabs layout above gives

Tip: Spacers

Sometimes you’ll find that some columns end up longer than others. You may add “Spacer” agents to skip long columns. Just add a “spacer” in the layout like I did in the tabs example above.

Next: How to view your public Spyno page from anywhere

Secret Spyno: How to add a new agent

Your Spyno Agents do all the work, so you don’t have to. Train agents for each website you want to gather information from. You tell each agent what information to grab, as well as how to present the information to you. Here’s how…

  1. Visit in Chrome’s address bar
  2. Click

    Open the options page

  3. You'll see this Spyno options page

    You’ll see this Spyno options page

  4. Scroll to the bottom and click in the box for "Add New Agent"

    Scroll to the bottom and click in the box for “Add New Agent”

Agent example (options explained below):

Line 2 name: unique name for your Spyno agent (required)
Line 4 periodInMinutes: how often you would like Spyno to fetch from the source URL
Line 6 url: the source URL that has the info you want Spyno to fetch
Line 7 extract: this section tells Spyno what info to grab and how you want it presented
Line 8 root: JSONPath to the list of records you’re interested in
Line 9 fields: JSONPath to the fields you’re interested in, and what name you want to give them
Line 14 template: Handlebars template to tell Spyno how to draw the info you fetched. Spyno automatically packages your info into a list of “rows”, so your template should always have

You may include any html in your template, including ‹style›. E.g.,

Next: How to design a layout for your Spyno page