Tim Poulsen

Shelly Home Assistant-compatible relay

2024-09-19T00:00:00-04:00

Using a Shelly 1 Mini Gen 3 as an MQTT-controllable relay

For my Christmas lights, I've been using homemade MQTT-controllable relays. I've been wiring up an ESP2866, a relay, and a AC-DC converter. While this works, it's a bit of a rat's nest of wiring. So I decided to try using a Shelly 1 Mini Gen 3 relay instead. These package basically the same components I've been using into a small, self-contained little box with convenient wiring connections. In my bench tests so far, I think this will work great.

In this post, I'll explain how I wired up the Shelly and how I can control it using MQTT.

Wiring it up

First up, I needed to wire up a Shelly. All of the diagrams I could find assumed you were using these to control a light and you'd have a light switch in the system. I will not be using them that way. After some reading and tinkering, this is the way I wired up my Shelly without a light switch involved:

Here's a picture of it on my bench with some short extension cord pigtails:

This should be much neater and easier to package in a 3D printed case. Next up was getting MQTT to work.

Enable MQTT and get details about your Shelly

You'll need various details about your Shelly, such as the IP address, ID, and other information. The easiest way to find this info is to use the Shelly mobile app. With your Shelly wired up and powered on, it should show up in the app. You can tap into it to get these details. You'll need the ID to use with MQTT or the IP address to use with HTTP URLs to control the relay.

You'll also need to enable MQTT. I'd swear I did this from the app. But that was a while ago and other blogs say you have to do this via the web interface. In your browser, enter the IP address you got from the app. This will bring up the adminstration pages for your Shelly. The option to enable MQTT will be under the Internet & Security > Advanced - Developer Settings option.

(If you don't have or want to install the Shelly app, you can check your router to find the IP address the Shelly is using. Then, use the administration page as described above to find the Shelly ID.)

Controlling via MQTT

I found some web pages describing how to control the Shelly over MQTT. However, they were apparently all for older models because most of the directions in those guides failed to work for me. Here's what I was able to figure out on my own.

First, to make the later commands a bit easier, let's set some shell variables. This will work on Linux or Mac. I'm not sure about Windows, though perhaps if you use the Linux-like shell rather than the Command Prompt or PowerShell.

It's handy to open two terminal windows so that you can subscribe to the Shelly's topic and get error information in one and issue commands to the Shelly in another. So, open two terminals and enter the following in both.

export MQTT_SERVER="your_mqtt_server.local"
export MQTT_PORT=1883
export SHELLY_ID="shelly1minig3-dcda0ce69534"

Subscribe to the Shelly

In one of the terminals, subscribe to the Shelly's RPC topic like this:

mosquitto_sub -h ${MQTT_SERVER} -p ${MQTT_PORT} -t ${SHELLY_ID}/rpc

Or subscribe to all topics, like this:

mosquitto_sub -h ${MQTT_SERVER} -p ${MQTT_PORT} -t '#' -v

Any error messages will be output to this terminal, and you can see the commands that have been issued to the Shelly. This isn't required, but it can be helpful for debugging.

Get status info

In the second terminal, enter the following:

mosquitto_pub -h ${MQTT_SERVER} -p ${MQTT_PORT} -t ${SHELLY_ID}/rpc \
    -m '{"id":"Shelly.GetStatus", "src":"devices/'${SHELLY_ID}'/messages/events", "method":"Shelly.GetStatus"}'

The method parameter specifies which command you're executing on the Shelly. There are many you could use. Check the Shelly docs, but be aware that the docs cover all the Shelly models so some of the pages won't apply to the Mini 1 Gen 3 (or whatever you're using).

This will return a bunch of information about your Shelly. In fact, all the same info you were able to get from the app.

Turn on the relay

mosquitto_pub -h ${MQTT_SERVER} -p ${MQTT_PORT} -t ${SHELLY_ID}/rpc \
    -m '{"method": "Switch.Set", "params":{"id":0,"on":true}}'

The method in this case is Switch.Set. There's a Light.Set command as well, but it's not used or supported by the Mini 1 Gen 3. Next note the params key that has two additional sub-keys. The id parameter is required. Some Shelly models come with multiple relays, and this identifies which relay you're controlling. The Mini 1 Gen 3 has a single relay, but you still have to tell the Shelly that you're addressing the first one.

Finally, the on parameter specifies whether the relay should end up being on or off — true for on, false for off.

Turn off the relay

No surprise then that this command turns off the Shelly:

mosquitto_pub -h ${MQTT_SERVER} -p ${MQTT_PORT} -t ${SHELLY_ID}/rpc \
    -m '{"method": "Switch.Set", "params":{"id":0,"on":false}}'

Controlling via HTTP

You can also control the Shelly over HTTP, in other words with your browser. Let's say your Shelly is at 192.168.0.122. You can visit these URLs to control your Shelly:

http://192.168.0.122/ — access the management interface
http://192.168.0.122/relay/0?turn=on — turn on the Shelly
http://192.168.0.122/relay/0?turn=off — turn off the Shelly

Additional Resources

https://shelly.guide/webhooks-https-requests/
https://shelly-api-docs.shelly.cloud/gen2/ComponentsAndServices/Mqtt
https://kb.shelly.cloud/knowledge-base/shelly-1-mini-gen3#Shelly1MiniGen3-Basicwiringdiagrams
https://sequr.be/blog/2020/10/getting-started-with-mqtt-and-home-assistant-and-shelly/ (This is one of the blogs that had info that didn't work for me. Still, you'll find some useful information here, plus Home Assistant integration info.)

Pi Zero Birdcam

2024-06-15T00:00:00-04:00

Nestbox camera with a Pi Zero W and PiCamera2

In an early post on this blog, I detailed my steps for creating a nestbox streaming camera. I've updated my setup so I thought a new post on the topic would be in order.

A Raspberry Pi Zero W and its camera are small enough to fit inside a nestbox without taking up too much room from the birds. You do need to supply good power to the Zero, so you'll have one cord running out of the box. Make sure to run the cable in such a way that it doesn't cause water to drip into the nestbox. Also, make sure a snake can't use the cord as a means to scale their way up and into the box. Snakes will eat bird eggs if they can get to them.

Pi setup and configuration

It's pretty easy to install all the needed software to a headless Raspberry Pi Zero W. There's a good article over at Tom's Hardware that describes the process, plus gives you tips for accessing the Pi Zero when it's plugged into your computer's USB port. Install the "Lite" version, which doesn't include the GUI. During the installation, make sure to set a unique hostname for your Pi Zero.

I'm using the newer PiCamera2 library for my streaming server. It does not support the legacy Pi camera interface. So, if you're running Bullseye or another non-current OS version, make sure to use raspi-config to disable the legacy camera interface. Bookworm has removed the legacy interface altogether, so you're set if you're using Bookworm.

Speaking of Bullseye, if you're running that older version of the Raspberry Pi OS, you'll need to tweak the /boot/config.txt file. You need to set the dtoverlay to ov5647 in order for the camera to be detected on the new camera interface. You can add this line at the end of the config.txt file. If a dtoverlay line already exists, as shown here, comment it out by adding a # to the beginning of the line:

# dtoverlay=vc4-kms-v3d
dtoverlay=ov5647

With those configurations out of the way, reboot your Pi Zero. Then, ssh into it in order to perform the remaining configurations.

PiCamera2 set up

Install PiCamera2 if it's not already installed. On my Bullseye-based Pi, it was already installed. But it was not installed on the newer Bookworm Lite version. The --no-install-recommends argument specifies to not install various helper apps, which are mostly GUI-based. You won't need them, and if you installed the Lite OS version, you wouldn't be able to run them anyway.

sudo apt install -y python3-picamera2 --no-install-recommends

If you'll be installing any other Python libraries, I recommend you create a virtual environment. However, you'll need to include the system packages in this environment or you won't have access to PiCamera2 that you just installed. Use this command to create the virtual environment:

python3 -m venv --system-site-packages venv

# then activate the virtual environment
source venv/bin/activate

# now you can pip install packages without interfering
# with other Python setups on your system

The streaming server

The PiCamera2 project provides various example scripts, including a handy streaming MJPG script. It's long, so I won't include a copy here. Grab it from the link provided. I'm using the script mostly unmodified. I named my file rpi_cam.py. I updated the title and H1 text to be a bit more applicable to my use-case. I also increased the stream dimensions from 640 x 480 to 800 x 600 (there are two spots in the file to change that).

Getting it all to run at boot

Of course, you'll want the script to run when you boot your Pi Zero. Create a shell script, named rpi_cam.sh, with the following contents:

#!/usr/bin/env bash

# activate the virtual environment if you created one
source /home/pi/venv/bin/activate
# run the camera script
python /home/pi/rpi_cam.py

You need to use the full path to the virtual environment and script if you'll be using the crontab technique I describe next. Don't forget to make the script executable with chmod +x rpi_cam.sh

To run the script automatically when the PiZero boots, run sudo crontab -e and add this line at the end:

@reboot /home/pi/rpi_cam.sh &

You need the full path to the shell script. The crontab will be executed as a system user that won't otherwise "know" about your home directory path. The & at the end runs the script as a detached process (i.e. in the background) so that it will keep running when the crontab process finishes.

Loading the resulting page

Once you've done everything above, reboot the Pi Zero one more time. Once it has booted, use your browser to visit http://the_hostname.local:8000/ and you should see the streaming output from your camera!

We had a pair of house finches build a nest in a wreath hanging on our front porch. Here's a shot of the camera rigged up to watch the nest, and a picture of the nest itself.

Weather display on an LED matrix

2024-04-24T00:00:00-04:00

Using CircuitPython on a HUB75 LED matrix

I bought a couple of surplus LED matrix panels a while back, and I finally got around to playing with them. My panels are 16x32 HUB75-compatible panels. I planned to "stack" them to create an effective 32x32 matrix. Adafruit makes a neat little ESP32-based microcontroller, the MatrixPortal that features HUB75 output and built-in WiFi support. I picked up one of them and then had everything I needed.

I wanted to do something useful with the matrix. Eventually I settled on showing the current weather conditions on it. I have an Ambient Weather weather station. It uploads data to the Ambient Weather site. Nicely, they provide a simple API where I can pull my data back down.

Given the really low resolution of my matrix, I planned to show one weather stat at a time, then cycle to the next. For example, show the temperature, then a moment later show the wind speed, and so forth. I planned to poll the API every few minutes to get updated data.

CircuitPython

While I could program the MatrixPortal in C, I'm a lot more comfortable with Python. Adafruit publishes a version of CircuitPython for this board. CircuitPython is a slimmed-down version of Python meant to run on microcontrollers. I downloaded the version for my board. The install process is pretty simple. You plug the board into a USB port and copy the CircuitPython files to the board. That's it.

It's important to note again that CircuitPython is a subset of Python. You could probably install some Python (PyPI) libraries, but more commonly you'll use the libraries that Adafruit publishes. These are provided as files that you copy as needed to the lib folder on the MatrixPortal (or other board you're using).

The MatrixPortal has pretty limited memory and storage. You need to be careful to install only the libraries you need to reduce the chance of running out of memory for your code. Make sure to delete any libraries you're not using off the board, too.

There's no compile step with CircuitPython. Simply copy your code.py file to your microcontroller. It will reboot and load your new version.

Challenges

Like Python, CircuitPython includes an automatic garbage collector to free memory when it's no longer used. However, the MatrixPortal was so limited in memory, I found I needed to force collection regularly in my program. Without that, the program would crash with an out of memory error.

Liberal use of gc.collect() helped a lot, but didn't fully resolve the issue. For a while, I was convinced that the requests library was too heavy for the MatrixPortal. I even considered giving up on the project till I realized the silly error I was making. By default, the Ambient Weather API call I was making was returning dozens of data readings when I wanted only the most recent one. Once I added the &limit=1 param to the URL, I was able to get the program working.

The other challenge I had to overcome was how to handle the two panels I was trying to use as if it were one larger panel. The trick turned out to be the serpentine parameter. In the following call, tile_rows specifies I have two panels. serpentine=False indicates I want them laid out serially, not in a serpentine fashion. You'd think that wouldn't matter with just two panels. But without that, I could not get the orientation of the two panels to work correctly.

matrix = Matrix(width=32, height=32, tile_rows=2, serpentine=False)

Adding a background graphic

Once I had the basic text version working, I decided to get fancy and show a GIF as a background. I used a graphics editor to create a 32x32 pixel image (to match my matrix size). The gifio built-in library can be used to show static or animated GIFs. Note that the OnDiskGif() function loads the GIF, but not the image within it until you call the next_frame() method. (That's the method you'd call to load subsequent frames in an animated GIF.)

odg = gifio.OnDiskGif('/background.gif')
odg.next_frame()  # critical or no image will appear
# Depending on your display the next line may need Colorspace.RGB565
#   instead of Colorspace.RGB565_SWAPPED. If the colors are wonky,
#   try changing this.
mygif = displayio.TileGrid(odg.bitmap,
                        pixel_shader=displayio.ColorConverter
                        (input_colorspace=displayio.Colorspace.RGB565_SWAPPED))
group.append(mygif)

My GIF is pretty simple. Some blue at the top to represent the sky. Some green at the bottom to represent grass. It's simple, but it made me happy to have it there.

Full code

Below is the full contents of my code.py file. The secrets.py file contains a simple dict named secrets whose keys are my SSID, WiFi password, Ambient Weather keys, and so forth.

import board  # for pin definitions
import busio  # to access SPI bus
import displayio
import gc
import gifio

from adafruit_esp32spi import adafruit_esp32spi_wifimanager
from adafruit_display_text.label import Label
from adafruit_esp32spi import adafruit_esp32spi
from adafruit_matrixportal.matrix import Matrix
from digitalio import DigitalInOut
from rtc import RTC
from terminalio import FONT  # Provides the font we use
from time import sleep

gc.collect()

# Get wifi details and more from a secrets.py file
try:
    from secrets import secrets
except ImportError:
    raise

print(f"Free memory (start): {gc.mem_free()}")

url = f"https://rt.ambientweather.net/v1/devices/{secrets['device_mac']}?applicationKey={secrets['awn_application_key']}&apiKey={secrets['awn_api_key']}&limit=1"

esp32_cs = DigitalInOut(board.ESP_CS)
esp32_ready = DigitalInOut(board.ESP_BUSY)
esp32_reset = DigitalInOut(board.ESP_RESET)
spi = busio.SPI(board.SCK, board.MOSI, board.MISO)
esp = adafruit_esp32spi.ESP_SPIcontrol(spi, esp32_cs, esp32_ready, esp32_reset)

requests = adafruit_esp32spi_wifimanager.ESPSPI_WiFiManager(esp, secrets, None)
requests.connect()

try:
    print("Connected to", str(esp.ssid, "utf-8"), "\tRSSI:", esp.rssi)
    print("My IP address is", esp.pretty_ip(esp.ip_address))
    print(f"Free memory (network connected): {gc.mem_free()}")
except:
    pass

matrix = Matrix(width=32, height=32, tile_rows=2, serpentine=False)
display = matrix.display

# --- Drawing setup ---
group = displayio.Group()  # Create a Group
bitmap = displayio.Bitmap(32, 32, 1)  # Create a bitmap object,width, height, bit depth
color = displayio.Palette(2)  # Create a color palette

# Create a TileGrid using the Bitmap and Palette
tile_grid = displayio.TileGrid(bitmap, pixel_shader=color)
group.append(tile_grid)  # Add the TileGrid to the Group
display.root_group = group

odg = gifio.OnDiskGif('/background2.gif')
odg.next_frame()
# Depending on your display the next line may need Colorspace.RGB565
#   instead of Colorspace.RGB565_SWAPPED
mygif = displayio.TileGrid(odg.bitmap,
                        pixel_shader=displayio.ColorConverter
                        (input_colorspace=displayio.Colorspace.RGB565_SWAPPED))
group.append(mygif)

# Text area 1
label_1 = Label(FONT)
label_1.text = "LOADING"
label_1.color = 0xFF0000
_, bby, bbwidth, bbh = label_1.bounding_box  # _ is bbx
label_1.x = 2  # round(display.width / 2 - bbwidth / 2)
label_1.y = display.height // 2 - bbh // 2 - (-bby) // 2 + 3
group.append(label_1)

# # Text area 2
label_2 = Label(FONT)
label_2.text = "##"
label_2.color = 0xFF0000

_, bby2, bbwidth2, bbh2 = label_2.bounding_box
label_2.x = 2  # round(display.width / 2 - bbwidth2 / 2)
label_2.y = display.height - bbh2 // 2 - (-bby2) // 2 - 1
group.append(label_2)

display.show(group)


def twoDigits(n):
    d2 = n % 10
    d1 = int(n / 10)
    return str(d1) + str(d2)


def parse_weather_data(resp):
    gc.collect()
    print(f"Free memory (parse_weather_data): {gc.mem_free()}")
    if resp:
        updated_weather = resp.json()
        gc.collect()
        return updated_weather[0]
    return current_weather


def get_weather_conditions():
    gc.collect()
    print(f"Free memory (get_weather_conditions): {gc.mem_free()}")
    resp = requests.get(url)
    print(f"response code: {resp.status_code}")
    if resp.status_code == 200:
        return resp


"""
0xFF0000  # red
0x00FF00  # green
0x0000FF  # blue
0x9933FF  # purple
"""
keys = [
    "tempf",
    "feelsLike",
    "humidity",
    "windspeedmph",
    "windgustmph",
    "winddir",
    "dailyrainin",
]
labels = {
    "tempf": ("Temp.", 0xFF0000),
    "feelsLike": ("Feels", 0xFF0000),
    "humidity": ("Humid", 0x0000FF),
    "windspeedmph": ("Wind", 0x9933FF),
    "windgustmph": ("Gust", 0x9933FF),
    "winddir": ("Direc", 0x00FF00),
    "dailyrainin": ("Rain", 0x0000FF),
}
current_key = 0
current_weather = parse_weather_data(get_weather_conditions())

def publish_weather_conditions(current_weather: dict, current_key: int):
    key = keys[current_key]
    value = str(current_weather[key])
    try:
        value_int = int(float(value))
    except ValueError:
        value_int = 0
    label = labels[key][0]
    color = labels[key][1]
    print(key, value)
    label_1.text = label
    label_2.text = value
    label_1.color = color
    label_2.color = color
    if current_key == 0 and value_int < 40:
        # if the temp is below 40, blue instead of red for Temp
        label_1.color = 0x0000FF
        label_2.color = 0x0000FF


while True:
    if current_key == len(keys):
        current_key = 0
    ts = RTC().datetime
    mins = twoDigits(ts.tm_min)
    secs = twoDigits(ts.tm_sec)
    if int(mins) % 5 == 0.0 and int(secs) == 0:
        # retrieve data every 5 mins
        print("Retrieving fresh data & updating display...")
        current_weather = parse_weather_data(get_weather_conditions())
        publish_weather_conditions(current_weather, current_key)
    if int(secs) % 3 == 0:
        # update display every 3 seconds
        print("updating display...")
        publish_weather_conditions(current_weather, current_key)
    current_key += 1
    sleep(1)

How to find outdated Python requirements

2024-04-07T00:00:00-04:00

Finding outdated Python packages

A quick Python technique this time. You can get a list of outdated Python packages installed on your system with a simple command:

pip list --outdated

That will list all the packages, including their dependencies. That's nice, but sometimes it makes it hard to know what to update first. So instead, use this command:

pip list --outdated --not-required

This will list only top-level packages, not their dependencies.

Do you use developer-only ("dev dependencies") packages, such as unit testing packages? There does not seem to be a simple command to find outdated dev-only packages. The best I can come up with is to install the dev dependencies, then use pip list, like this:

python3 -m venv temp_venv  # create a temp virtual environment
source temp_venv/bin/activate
pip install -r requirements-dev.txt  # install the dev dependencies
pip list --outdated --not-required
# then deactivate and delete the temp_venv folder

Without the --outdated flag, the pip list command will list the packages installed in your current environment. You can use the --not-required flag to again list only top-level packages.

You can use the --format json flag to output the list of packages as JSON. This could be useful if you're going to parse the list programatically. As long as you're not using the --outdated flag, you can use --format freeze option to output the list of packages in the pip freeze style.

You can learn about these and other pip list flags at the Pip documentation site.

Pixels part 3: FastLED

2024-01-30T00:00:00-05:00

In the first post in this series, I introduced RGB pixels, discussed the various types, and compared analog and true RGB pixels. In the second part, I covered wiring and shared my tips for reliably connecting to your microcontroller. In this third part, I will discuss the FastLED library, one of the most popular libraries for programming pixels.

FastLED

FastLED is a hugely popular library for programming all sorts of RGB pixels. The intro on that linked page summarizes the library nicely: fast, easy to use, widely used, and under active development. See the GitHub page for code and an excellent wiki. Get help in the FastLED Reddit sub.

Let's start with the basics. For this, I think it will help to see code. The comments inline explain what's going on.

#include "FastLED.h"           // include the FastLED library
#define LED_TYPE      WS2812B  // specify which type of pixels you're using
#define DATA_PIN      5        // which GPIO pin is outputting the data signal
#define NUM_LEDS      32       // how many LEDs in your setup
#define COLOR_ORDER   RGB      // the order of red-green-blue values in the data
#define BRIGHTNESS    128      // from 0 - 255

CRGB leds[NUM_LEDS];           // create an array to hold LED values

uint8_t startIndex;            // a couple of variables for tracking info
uint8_t colorIndex;

void setup() {
    delay(1500);
    // Set up our LED configuration, specifying the type, data pin, and color order
    // Then we pass in the array of values and the number of LEDs. Finally, and this
    // is optional, we apply a color correction to help account for color variations
    // across LED strips.
    FastLED.addLeds<LED_TYPE, DATA_PIN, COLOR_ORDER>(leds, NUM_LEDS).setCorrection(TypicalLEDStrip);
    // Next we set the brightness level
    FastLED.setBrightness(BRIGHTNESS);
}

void loop() {
    colorIndex = startIndex;   // this is a counter value, we're setting a start point
    for( int i = 0; i < NUM_LEDS; i++) {
        // for each LED in the array, set its color to a value picked from the
        // predefined (by FastLED) palette of colors -- in this case a rainbow
        // The `colorIndex` specifies which color to grab from the palette.
        // The function also expects a brightness value, and a way to blend
        // colors from one pixel to the next.
        leds[i] = ColorFromPalette( RainbowColors_p, colorIndex, BRIGHTNESS, LINEARBLEND);
        colorIndex = colorIndex + 10;  // increment our index
    }

    // You have to call show() to actually write the values to the physical LEDs and
    // light them up
    FastLED.show();
    // For delays, best to use the FastLED-provided functions instead of built-ins
    FastLED.delay(10);
    // Finally, increment our counter so that the next time through the loop, the
    // colors will be assigned one LED ahead of where they were this time.
    startIndex = startIndex + 1;
}

Phew, that was a lot. I hope you read through the comments and they were helpful. I'll expand on a few points. Starting from the top, the LED_TYPE is a constant representing the type of pixels you're using. FastLED supports many, though not all types. The supported pixels are listed here.

Next is the COLOR_ORDER. This varies by pixel type. Some expect the data to have the red color bits first, followed by green, then blue. But others will be in BGR or GRB or whatever order. I'm sure there's a better way, but when I don't know, I'll set all the LEDs to be red. If they show up blue or green, I know to start with a different order. Repeat for green and blue till you get the order correct.

Next I'll jump down to the FastLED.addLeds call. There are two primary versions of this function: one for 3-wire pixels and the other for 4-wire pixels. Shown above is the 3-wire variant, where there is no dedicated clock line. If you're using 4-wire pixels, the function you'll call is FastLED.addLeds<LED_TYPE, DATA_PIN, CLOCK_PIN, COLOR_ORDER>(leds, NUM_LEDS).

Colors and palettes

In the example above, I demonstrated using one of the pre-defined palettes of color. These define a set of colors that you can apply across the array of LEDs to get a pleasing mix of colors, such as a rainbow or set of ocean (blue-green) tones. That might be considered a bit more advanced than setting a specific color.

FastLED gives you many ways to specify a color for an LED. You can do so using one of the pre-defined named colors, by RGB value, HSV value, and more. For example, to set an LED to display 100% red, you could use any of the following:

leds[i] = CRGB::Red;
leds[i] = 0xFF0000;
leds[i] = CRGB(255, 0, 0);
leds[i].setRGB(255, 0, 0);
leds[i] = CHSV(HUE_RED, 255, 255);  // HUE_RED happens to be 0 on the color wheel
leds[i].setHSV(HUE_RED, 255, 255);
// or
leds[i].red   = 255;
leds[i].green = 0;
leds[i].blue  = 0;
// ... and probably more!

There's one more trick up the sleeve ... let's say you wanted to set all of the LEDs to red. You could loop through, set the color using one of the above techniques, then call FastLED.show(). Or, you could call FastLED.setColor(CRGB::Red) (or specify a hex value, HSV value, etc.). With setColor() you do not call .show()!

Fading out

I'll end with two techniques for fading to black. In both cases, you would put the call to the function inside the loop() function so that it's called with each iteration of the program loop.

// Technique 1: brute force
#define BRIGHTNESS    255

// Assumes there's something that sets all the LEDs to some color, like:
FastLED.setColor(CRGB::Red);

void fade_to_black() {
    // put the call to this function inside loop()
    static int brightness = BRIGHTNESS;
    static int fadeAmount = 2;
    if (brightness <= 0) {
        return;
    }
    brightness -= fadeAmount;
    FastLED.setBrightness(brightness);
}

Or:

// Technique 2: use one of the built-ins

void fade_to_black() {
    for (int i = 0; i < NUM_LEDS; i++) {
        // Dim a color by 25% (64/256ths)
        // eventually fading to full black
        leds[i].fadeToBlackBy( 64 );

        // Or, reduce color to 75% (192/256ths) of its previous value
        // eventually fading to full black
        // leds[i].nscale8( 192);
    }
    FastLED.show();
}

There's a ton more to explore in the FastLED library. There's support for matrices, controlling multiple strips (even of different types), built-in fast math functions for varying pixels by sine waves and such, and so much more. I hope this gets you off to a good start. Happy exploring.

Pixels part 2: the hardware

2023-12-07T00:00:00-05:00

In the first post in this series, I introduced RGB pixels. I discussed the various types and compared analog and true RGB pixels. I also covered power requirements. In this post, I'll cover wiring and share my tips for reliably connecting to your microcontroller. I also include a basic sketch for displaying some colorful blinkies.

Microcontrollers

I personally prefer to use ESP32 or ESP8266 microcontrollers. They're small, cheap, and readily available from Amazon or Aliexpress. They even feature built-in wifi and Bluetooth connectivity. But any Arduino-compatible microcontroller will work.

I recommend powering your pixels directly. Don't pull power from a 5v pin on the microcontroller. As discussed in the previous post, pixels draw a lot of power so you could easily burn out your microcontroller by drawing too much power through it.

Another consideration is powering the microcontroller itself. You could power it through its USB port. Most microcontrollers do just fine running off any old 5v power brick you have laying around. But, wiring that way means you'll essentially need two power sources: one for the microcontroller and another for the pixels. So, instead of powering over USB, you can supply power and ground directly to the Vin and Gnd pins on your microcontroller.

Finally, there's wiring up the data (signal) line. Wiring 3-wire or 4-wire pixels is essentially identical. With 3-wire pixels, you'll run a wire from a digital output pin to the Di (data in) pad on the pixel. For 4-wire pixels, you'll also need to run another wire to the Clk (clock) pad.

Consult the pinout documentation for your specific microcontroller. You want to choose pins that are not pulled high or low during boot (i.e. "strapped" pins). If you do, you might have odd difficulties like not being able to flash your microcontroller or having it not boot. Also, don't choose the pin whose GPIO port corresponds to an on-board LED. If you do, you may experience some flashing during boot or other odd lighting behavior as general operations light up the on-board LED.

In the diagram above, I illustrate connecting pin D1 to the data line. This pin on the ESP32s I use corresponds to GPIO pin 5 which is an unused GPIO port on that particular board. Again, consult the documentation for your microcontroller.

Soldering the strip

I have never found quick-connectors to work reliably with pixel strips. For that reason, I always solder connections to the pads. Here's a tip: tin the pads first, then solder on the wires. By "tinning" I mean put solder on the pad. Clean the pads or dab a small amount of liquid flux on the pad. Heat it carefully, quickly, though you don't need to rush. Add a small bead of solder once the pad is hot.

Next, tin the wire. Again, flux it, heat it, tin it. Once that's done, you can more easily connect the two. Heat the pad and when the solder melts, bring the wire to the pad. Its solder will melt quickly, fusing the wire to the pad.

There's no easy way to recover if you destroy a pad, say by over-heating or scraping off the thin copper coating. In those cases, you'll need to cut off that pixel and work with a slightly shorter strip.

If your microcontroller has header pins soldered to it already, I recommend you use jumper wires with Dupont connectors. Connect the female end to the microcontroller and cut off the other end and solder that to the pixel strip. Soldering to a header pin is tough, plus once you do you'll never be able to insert the pins into a breadboard or socket. You can super-glue the connector to the board. If you ever need to remove it, you can dissolve the glue with acetone. If your microcontroller has through-board holes or pads, you can solder the other end of the wire directly to your microcontroller.

I went so far as to create a custom PCB carrier board. I can solder the ESP8266/ESP32 and a few components to the board then have easy screw-connect headers to make connecting the wires easy and reliable.

As you see on my board, I typically wire in a capacitor (500–1000 µF at 6.3V or higher) across the + and – terminals. Adafruit also recommends a small resistor, 300-500 Ohms, between the output pin and the LED strip. My board above shows that resister. However, in general I don't include such a resistor. I have not found it necessary and sometimes my strips don't light if I include one. Your mileage may vary.

Simple blinky script

Now that you have your LED strip connected to your microcontroller, you need to flash some code to get the lights blinking. A very popular library for working with pixels is FastLED. It contains all the core functionality you need plus many helpful functions and constants. In a future article, I will dive more deeply into FastLED. Here we'll go with a simple example from the FastLED_examples repo.

Below is a simple example of a FastLED script that will show a moving rainbow palette of colors. This script is for the 3-wire WS2812B type pixels, with the data line connected to GPIO 5. In essence, each time through the loop, the script assigns a color to each LED in the strip from a rainbow palette. After showing the colors, it delays 10 milliseconds, advances the starting point in the palette for the colors, and repeats.

//***************************************************************
// Basic palette example using one of FastLED's palettes.
// From https://github.com/marmilicious/FastLED_examples
// Marc Miller,  March 2019
//***************************************************************

#include "FastLED.h"
#define LED_TYPE      WS2812B
#define DATA_PIN      5
#define NUM_LEDS      32
#define COLOR_ORDER   RGB
#define BRIGHTNESS    128

CRGB leds[NUM_LEDS];

uint8_t startIndex;
uint8_t colorIndex;

void setup() {
    delay(1500);
    FastLED.addLeds<LED_TYPE, DATA_PIN, COLOR_ORDER>(leds, NUM_LEDS).setCorrection(TypicalLEDStrip);
    FastLED.setBrightness(BRIGHTNESS);
}

void loop() {
    colorIndex = startIndex;
    for( int i = 0; i < NUM_LEDS; i++) {
        leds[i] = ColorFromPalette( RainbowColors_p, colorIndex, 255, LINEARBLEND);
        colorIndex = colorIndex + 10;
    }

    FastLED.show();
    FastLED.delay(10);
    startIndex = startIndex + 1;
}

That should get your lights blinking!

An intro to RGB pixel LEDs

2023-11-27T00:00:00-05:00

RGB LEDs, sometimes called pixels, are lots of fun. Who doesn't like blinky lights?! They're not hard to use, though there's enough complexity to make these LEDs seem a bit daunting. In this post, I'll go through the basics to get you started on your blinky fun.

Traditional LEDs display a single color. Bicolor and tricolor variations can show multiple colors. RGB LEDs, on the other hand, let you show any color in the rainbow (ignoring some color theory stuff that limits their actual gamut). Traditional LEDs need just the right polarity power to the appropriate pins in order to light up. RGB pixels require some sort of microcontroller to specify what color to show and to specify when to show it.

Addressable vs non-addressable

To be clear, in this article I'm talking about addressable pixels. With this type, the microcontroller can specify a different color for every LED in the set. There are also non-addressable LEDs, sometimes called analog LEDs. With those, you can display any color but all of the LEDs in the set will show the same color.

With non-addressable LEDs, a separate signal is fed to red, green, and blue LEDs (tiny ones, built into a single integrated circuit component). By varying the power sent to each, you can represent any color in the rainbow. But again, keep in mind that all the LEDs in the set will show the same color.

You can identify analog, non-addressable LEDs by their inputs -- if you see connectors for R, G, B, and a voltage (typically 12v), then you're dealing with a non-addressable LEDs set. As shown here, sometimes these strips also include a white channel (W) to give brighter, purer white output.

In contrast, addressable LEDs will have pins for voltage, ground, and data (or signal). Sometimes, an additional clock pin is used. Addressable LEDs come in 5v and 12v varieties, and sometimes even in other voltages. So, the typical inputs will be labeled +5, D, and GND; or +5, D, C, and GND. In and out matter. So, you will probably see labels like DI and DO for data in and data out. Sometimes, you'll just see an arrow on the strip. The arrow points to the out side. You must put your inputs into the in side or the strip will not light.

WS2812B vs APA102C vs ... oh my

You will see many different identifiers that describe the type of addressable LEDs. Most times, these are product numbers, such as WS2812B. Some vendors have assigned their own names, like neopixels or dotstar, which in my opinion just adds confusion.

In broad terms, all the addressable LEDs can be broken down into roughly two camps: 3-wire vs 4-wire. 3-wire pixels required power, data, and ground connections. Common examples are the WS2812B and SK6812. Those are 5-volt examples, though 12v versions, such as the WS2811 are also available. Adafruit calls their 5 volt, 3-wire pixels "neopixels" (SK6812).

4-wire pixels need power, data, clock, and ground connections. The most common are SK9822 and APA102C. Again, there are 5v and 12v versions. Adafruit calls their 5 volt, 4-wire pixels "dotstar" (SK9822). Because of the clock signal, 4-wire pixels can change colors more rapidly and more uniformly.

In essence, with 3-wire pixels, the data sent down the line is a series of values for each pixel in line. As the signal arrives at the IC, the chip "takes off" the value at the top of the list, lights up according to that signal, then passes the remaining signal down the line. Thus, each pixel in the strip lights up at a slight delay after the LEDs that precedes it. (The delay is barely noticeable on even the longest strips.)

With 4-wire pixels, again the data is sent as a series of values and each pixel takes off its value and passes the remainder down the line. However, the pixels don't show the new color value until the clock signal line switches. In this way, the pixels all change in unison. The real benefit of this technique is that you can more rapidly change colors since the timing is closely controlled by the clock signal.

While you can mix and match types controlled by a single microcontroller, it's easiest to stick with a single type for a given project.

More power!

Pixels draw a lot of power. For example, the WS2812B draws 50 milliamps per pixel at full brightness. That adds up. A strip of 30 pixels will draw as much as 1.5 amps. Your Arduino might run off some random phone charger but since the typical phone brick puts out 1-2 amps, you won't be driving too many pixels off of it. At the extreme, 5 meters of 144 pixel per meter strip will need as much as 36 amps at 5v! For that, you'll need a specialty power supply.

With longer strips, you'll also want to consider power injection. In addition to supplying power at the beginning of the strip, with power injection you supply the same voltage farther down the strip. For example, you might connect your power source to the +5 line every meter on a long strip.

You can buy high amperage power supplies off Amazon or other online retailers. Another handy way is to repurpose an old laptop power supply. They typically output some higher voltage, say 19 volts. You can use a DC to DC converter to step down the power to the voltage you need. Amazon, Adafruit, and others sell DC-DC step-down converters in various form factors.

Coming up

This post is getting too long so I'll end it here. In the next post, I'll cover wiring, connecting to your microcontroller, and show a basic sketch for displaying some colorful blinkies. I'm planning a third post in the series to cover the libraries -- FastLED in particular -- commonly used with pixels. Stay tuned!

This code is crap

2023-08-31T00:00:00-04:00

Throughout my career, I've frequently heard developers lament that "this code is crap!" (well, they usually say ). It's usually some mid or senior level engineer looking at the existing codebase whining about how "bad" the old code is. To those situations, I say there's never a valid time to say the code is "crap."

Code represents the sum of:

the acceptance criteria, goals, and deadlines that inspired the code to be written.
the knowledge of the problem domain at the time it was written.
the balance between Done & Good-enough at the time it was written. Put another way, the balance between get it out the door versus make it perfect.
the knowledge, skill, attention, and diligence of the reviewers.
the knowledge and skill of the author. (This is intentionally last.)

Ultimately, code reflects the team of people who specified it, wrote it, reviewed it, and tested it, as well as the priorities, deadlines, and understanding at the time it was written. Attacking code is attacking people, many of whom may still be your fellow employees.

So, don’t say “this code is crap.” Instead, say given what we know now, given the people we have to work on it, and the time we have to work on it now, we can make it better.

An anecdote

"This code is crap" said the reviewer to my face at my first ever professional code review. What a way to start a career! Now, he was right. I was new and had tons to learn. To his credit, he showed me the many ways my code could be improved. In the end, what was merged into the codebase was much better code. I learned a lot. And despite dealing with a rather harsh personality, I learned to appreciate what a code review can do for me and the team.

Still, don't ever say that to a new engineer!

Dad Jokes As a Service

2023-07-24T00:00:00-04:00

Everyone loves a good dad joke

... now, if only there were some good dad jokes!

The ICanHazDadJoke site claims to maintain the internet's largest collection of dad jokes. GitHub user CrossNox has made a nice wrapper around their API. Together, this lets you retrieve a random dad joke programmatically.

Why, you ask, would you want to get a dad joke programmatically? You could include it in your web site or app, showing random dad jokes to delight your users. You could write a simple script to retrieve and show you the joke from the command line. You could even prank someone and have their computer speak a dad joke every day.

You'll need the library. A simple pip install dadjokes will take care of that.

Then, you'll need this simple script:

from dadjokes import Dadjoke
dadjoke = Dadjoke()
print(dadjoke.joke)

Whenever you need a laugh, from the directory where you saved that script, run python dadjoke.py

Some simple enhancements (for *nix systems)

You can make this file executable, so you can run it with having to put the python part on the command. First, let's find out where your python executable is located. Enter which python3 and note the path that is output. Next, edit your script. As the very first line, you'll add a "hashbang" that points to your python executable. Like this:

#!/path/to/your/python

from dadjokes import Dadjoke
dadjoke = Dadjoke()
print(dadjoke.joke)

Once you've saved your changes, you can mark the file as executable with chmod +x dadjoke.py From now on, you can type ./dadjoke.py to run the script. Wow, that saved you from typing like 5 whole characters!

For you Mac users, you can use the built-in say program and have your system speak the dad joke. Enter ./dadjoke.py | say (If you can believe it, the jokes are even less funny when spoken by the monotone computerized voice.)

It would be unprofessional of me to suggest you sneak over to a coworkers computer when they leave it unlocked, and set up a cron job to have a dad joke spoken to them throughout the day. So, I won't suggest that.

Getting a Boolean from any value in Python

2023-07-14T00:00:00-04:00

Getting a Boolean value from any data type

Recently, I was building a webhook that needed to accept and operate on varied data from outside sources. Included in that data were values meant to be Booleans. However, some callers were sending values like "Yes" or "no" or even "-1". I needed to convert each of those to a Boolean value based on their semantic meaning. In other words, we humans know that the string "false" should represent a False value. But on it's own, Python would treat the non-empty string as a True value.

I'll give background and explanation below. But for the impatient readers out there, here's the code I came up with.

import ast
from numbers import Real
from typing import Any

def get_bool_from_any(val: Any) -> bool:
    # CC0 license https://creativecommons.org/publicdomain/zero/1.0/
    try:
        # First, try treating the input as a number so that we can
        # handle values such as 0, 1, -1, 0.1, etc.
        return float(val) > 0
    except:
        # Converting to a float failed, so, let's see if it's a string
        if type(val) is str:
            try:
                # Use ast.literal_eval() rather than eval() to safely evaluate the input string
                # lowercase/capitalize to handle strings that aren't lead-capitalized
                # This statement will handle strings like True, true, False, FALSE, etc.
                return ast.literal_eval(val.lower().capitalize())
            except ValueError:
                # String value isn't parseable by ast, so check for custom falsey strings
                return val.lower() not in ['no', 'none', 'null']
            except SyntaxError:
                # empty strings and multi-word strings will raise a syntax error, treat that as False
                return False
        else:
            # Finally, it's not a number or string so rely on Python's built-in coercion rules
            return bool(val)

In the actual project, I added a suite of tests around this function. For simplicity in this post, let's just manually check some values.

truthy_values = [True, "True", "true", "TRUE", "Yes", 1, "1", 0.1, {"a": False}]
falsey_values = [False, "False", "false", "FALSE", "No", False, "To be or not to be", 0, "0", -1, None, {}]

for val in truthy_values:
    print(f"`{val}` is {get_bool_from_any(val)}")

for val in falsey_values:
    print(f"`{val}` is {get_bool_from_any(val)}")

Changing data types

To understand what's going on in that function, we need to consider data types. Python is not a strongly-typed language like C or Java. However, the language does define data types, and generally requires you to operate on one type at a time. For example, you can't add an integer and a list.

When data types don't match, the Python interpreter will do its best to convert values so they do match. However, its rules for this coercion don't always match what you might expect.

Originally, Python did not have a Boolean data type. One was added in PEP 285 (way back in 2002!). In Python, bool inherits from int in order to maintain compatibility with code that was written before this type was added. This is why 1 == True evaluates to be a true statement. It also means you can add 1 + True to get 2. Weird.

Coercion rules in Python

That PEP and its implementation defined how the interpreter will coerce values into Booleans, which is summarized in this table.

Input value	Coerced to
An empty string	`False`
The string `"False"`	`False`
Any other non-empty string	`True`
The number `0` or `0.0`	`False`
Any other number, positive or negative	`True`
An empty list, object, or set	`False`
A list, object, or set with members	`True`

For most situations, those rules are great. However, you'd probably expect bool("false") to return False when in fact it returns True. Lots of situations treat -1 as a falsey value. And silly humans use strings like "Yes", "No", and so forth to represent Boolean-like values. We can do better!

Digging into get_bool_from_any()

We finally have the background we need to understand the get_bool_from_any() function above. The try block that starts on line 7 handles numbers, as well as strings that can be converted to numbers like "0.1". For my purposes, it made sense that any value greater than 0 would be True and the rest False.

The float(val) call will raise an exception if val can't be converted to a number. So the next thing the function handles is strings. As the comment says, I use ast.literal_eval() rather than plain eval() since it's less vulnerable to injection attacks. (This function comes from the ast or Abstract Syntax Tree built-in library.) To handle upper- and lowercase variations, I convert to lowercase, then lead-capitalize the string. In this way, "false" will be parsed and converted to False as we'd expect.

The ast.literal_eval() function will raise an exception in a couple of cases. If it can't parse the word — in other words, it's not "False" or "True" — I catch the ValueError and test to see if the string is in my short list of falsey words. If so, the function returns False otherwise it returns True.

The other except block takes care of empty strings and multi-word strings. On its own, ast.literal_eval() raises a SyntaxError in those situations. I just treat those cases as falsey values and return False.

Finally, I handle input values that are something other than a string or number. Line 27 uses Python's bool() function to explicitly convert the input value to a Boolean. Following the rules described in the table above, this handles cases like empty or populated lists, objects, and so forth.

Conclusion

I really didn't need a function that was this comprehensive. The callback payloads I was handling always contained strings, and the values were always one of a short list of variations. I mean, my function could have been as simple as:

def is_it_true(val):
    return val in ["true", "True", "yes", "Yes", "1"]

But, where's the fun in that? With my function, I handle just about any data type as an input value. I handle human-intuitive values like -1 being a falsey value. And, I got to explore some library functions I would rarely use otherwise. Figuring out the logic of the if-else and try-except blocks gave my brain a little exercise too.

I hope you find the function useful, or you at least learned something from this post. If you find an error or an edge case, hit me up on Mastodon or put in a PR on this blog's repo; links are in the sidebar.

Mr. Huntley's 1914 Chandler Light Six

2023-03-09T00:00:00-05:00

Identifying the car in an old photo

One of the greatest parts of living in an old house is identifying the history of its occupants. In this photo, we see A.E. Huntley, the original owner of our house, looking rather dapper in his car. His grandaughter shared the photo with us a few years after we bought the house. While she knew who was in the photo, and where it was taken, she had no other information. I thought it would be great to identify the car, and therefore the age of the photo. But, the picture didn't give me much to go on.

Guessing a date based on the age of our house, a rough age for Mr. Huntley, and the general styling of the car, I figured the photo would have been taken sometime in the early 1900s, most likely in the teens or early 20s. Even under a magnifying glass, I could not make out any identifying marks on the car. There are no medallions, name plates, or really hardly any identifying features to the car. The shape of the doors is perhaps the most distinctive feature.

I started searching with the most well-known cars of the era -- Ford, Chevrolet, etc. Nothing looked close. While Mr. Huntley was a successful dairy farmer, he was not wealthy. He certainly could not have afforded a Cord, Duesenberg, Pierce Arrow, etc. (Of course, it was fun looking at photos of those gorgeous vehicles.) Eventually I moved on to lesser-known and regional makers, of which there were many.

I found out that a few car makers were based in the Rochester area in the early 1900s, including Seldon (PDF), Foster, Cunningham, and Jenkins. One very promising candidate was the Mora, built not too far down the road in Newark, NY.

Look at those doors! But the overall shape was not correct. Also, the model year of that car is 1907, which was about a decade older than my guess on the photo's age. Still, I kept on researching Mora cars. There's not a lot of info or photos online. From what I found though, they made cars that looked like Mr. Huntley's.

I tried asking friends and fellow-car enthusiasts, including co-workers at an automotive tech business I used to work for. My posts on Facebook and Reddit got me various answers, but none were very helpful (or correct). I didn't give up, but I did put the search on the back-burnder. The other day, someone replied to one of my rather old Reddit posts and pointed me to the Chandler. A little googling and bingo!

The 1914 Chandler Light Six is a dead-ringer for Mr. Huntley's car. Compare the vents on the side of the hood, shape of the seats, spoke patterns on the wheels, the running board shape, and of course, those doors.

The Chandler Motor Car Company

The Chandler Motor Car Company was located in Cleveland, OH. It was formed in 1913 by executives and other employees of the Lozier Motor Company. Lozier made high-end cars, in fact, some of the most expensive US-made cars of their day. Frederick C. Chandler, former-chief designer at Lozier, wanted to build quality mid-priced cars. So he left Lozier and started his own company.

Chandler sold a 1914 six-cylinder touring car, probably what Mr. Huntley owned, for $1,785. According to the CPI inflation calculator, that would be approximately $53,700 in 2023 dollars. I have no idea if he bought it new or used, though the car looks to be in great condition in the photo. I think it's very interesting that he bought a car from an Ohio manufacturer when there were quite a few makers of similar-class cars much closer to home.

Chandler had a decent run, and by 1919 was outselling Nash and Oldsmobile. They ranked seventh in export sales among U.S. automakers that year. Unfortunately, they expanded too quickly, ran out of money, and sold to Hupp Motors. While Hupp promised to keep the Chandler lines going, about a year later they shut down Chandler and instead used their factories to produce Hupp vehicles.

I'll probably never know what happened to this car. He probably sold it or traded it in. Maybe it was lost in the fire that destroyed the barn. There are just a few Chandlers known to still exist. If this one were still around, it would be worth more than our house.

My first custom PCB

2020-03-25T00:00:00-04:00

I had many troubles this year with my Christmas light electronics. Solder connections went bad multiple times. Wires got snagged and tangled during setup. It was tough re-soldering connections while sitting in a snowbank with the circuit hanging by its wires from a yard decoration.

I decided I could fix these problems by using a custom printed circuit board. With a custom PCB, components would be securely soldered to the board; there would be no wires to snag or pull free; and the final product would be easier to package in a waterproof container. Of course, I've never created a PCB before.

My custom PCB with components mounted

There are quite a few PCB design apps, many of them free or free for personal use. After checking out gEDA, KiCAD, and a few others, I settled on EasyEDA. It seemed to be fairly simple to use. Conveniently, they are affiliated with a PCB prototyping company. So, right from EasyEDA, you can upload your finished file to place an order (more on that later).

Since this was my first PCB, I'm not the best person to teach how to design PCBs or even use EasyEDA. So, I'll point you to the tutorials that got me started.

The Beginners guide to PCB design with EasyEDA Part 1 (and parts 2 and 3) YouTube videos were probably the most helpful.
EasyEDA has some videos on their site. They vary in quality. Check them out on the EasyEDA YouTube channel
EasyEDA also has many written tutorials that explain a lot of the terminology and back-details that you'll probably need to know. I read through many of these as the need arose.

Prototype first

It really helps to know what circuit you're going to create before you start designing your PCB. I'm sure expert circuit designers could start from scratch. But, I suggest you use a breadboard and jumpers to make sure your circuit design will work as intended before you try to turn it into a PCB. In my case, I had working prototypes of my circuit.

My pixel controller prototype

EasyEDA

This software is available as either a download or a web app. Either way, you need to create a free account on their site because all the data is stored on their systems. The bright side of this is that you can switch back and forth between the installed app and the web app without issues.

Working from your prototype, you draw your circuit using tools fairly similar to a normal drawing program. To start, you place components that you find in their library of items onto a schematic view (shown below). At this stage, you don't need to worry about arranging components as you would want them on the final PCB. Instead, you put them wherever and connect them via "net ports." These are like labeled flags specifying which parts will be electrically connected. For example, you would tag anything connected to ground with the GND net port.

Schematic view of EasyEDA

Once you have all the components on the schematic, and all your net ports defined, EasyEDA creates the PCB view. This is the stage where you arrange components in a functional manner. You want to make sure "traces" (the electrical connections) can be neatly and logically drawn. You can't have traces cross, though you can draw traces on different layers to make connections where otherwise they'd have to cross. (Traces connect from one layer to another with a "via" basically a wire between layers.)

PCB designer view of EasyEDA

The basic PCB purchase includes a two layer board. To save money, you'll want to stick to just two layers if you can. I spent a bunch of time moving components around so that my traces wouldn't need more than two layers.

There are actually more layers that come with all boards. These include the top and bottom silk layers, which is where you can add labels and drawn features that are silk screened onto your PCB. There are others too, that are part of the PCB manufacturing process and which you probably don't need to worry about, such as the top and bottom paste layers. You can see all of these in the EasyEDA PCB view.

The other cost constraint is that the cheapest board prices are for a 100mm by 100mm PCB. Conveniently, you can actually fit multiple PCBs in that space in what's called a "panelized layout." In my case, I fit all the components and traces in a roughly 50mm by 100mm area so I could get two PCBs in the allotted space. I paid for 10 PCBs, but since I got two boards in each, I ended up with 20 actual boards to play with. Note that you need to leave about 3mm dead space between the panels to allow space for the V-grooves (where the two PCBs snap apart.)

Panelized finished board

Check and check again

I spent a bunch of time checking and re-checking my layout. You can hide various layers to help focus on certain connections. For example, I'd hide all but the top-layer traces to be sure each connected the pins I needed connected. Then I'd turn on other layers and re-check. Once your PCB is fabricated, you can't change the traces. So if you get it wrong, you'll be adding jumper wires or scraping off traces to break unwanted connections.

When you're ready to go, you click an upload button which shows a screen like the following.

Export/Upload design to JLCPCB

They make the "Order at JLCPCB" button extra obvious, and it is convenient. But if you want to use a different fab house, click the inconspicuous Generate Gerber button. Gerber files are the standard that most fabricators will need to create your PCB.

In my case, I clicked that big blue order button. This uploads your gerber file to the JLCPCB web site. At that site, you choose options like board color, number to order, trace and over-solder materials to be used, and so forth. They offer a pretty generous discount price on your first order. Frankly, for a hobbyist like me, their normal prices are also quite reasonable.

The Audit

Once you've submitted your order, JLCPCB will review your circuit. They're not going to do any sort of in-depth checking of your traces or whether the PCB will function (at all, let alone how you want it to). Their audit is simply looking to confirm your PCB can be produced. In my case, my original upload failed the audit. I had panelized by design but failed to leave the required minimum space between panels. In a day or so, I got the following note:

JLCPCB's audit says I failed :(

If this happens to you, don't place a new order like I did. By doing so, you're no longer placing your "first order" and won't get that same discount. This time, instead of uploading from the EasyEDA app, export the gerber file. Then, go directly to the JLCPCB web site. There will be a "replace file" option under your failed order. Use it and re-submit. You should get the order at the original price using that method.

Their shipping options aren't great. DHL is probably your best bet. It's not particularly cheap, or fast. Still, even during the early Coronavirus/Covid-19 disruption timeframe, my order showed up in roughly two weeks.

It's Alive!

The most important thing, of course, is that the board worked as intended (well, almost). I soldered on the components — in my case, that was the NodeMCU Lolin board, a level shifter, capacitor, resistor, and three header connectors. I wired it to some NeoPixels and powered it up ... nothing. Turns out that there's nothing wrong with the board other than the resister in-line with the data out connection. I threw that in at the last minute after reading a resistor was necessary to prevent damage to the pixels. I've never used one before, and I don't think I will in the future. I soldered a shunt (wire) across the resister and it works just fine.

In use — it works!

Looking ahead

My custom PCB does exactly what I wanted it to do. Still, I may refine this board in a future order. Next time, I won't include the resistor in the design. I'll make the pin holes for the ESP8266 module larger (it was tough getting the pins through the holes) and those for the capacitor smaller (they're huuuge).

I already have other boards in mind to create — relay boards, similar boards for other chips, like the ESP32, and more. This was fun and uber-geeky to have created my own PCB.

Remote work

2020-03-15T00:00:00-04:00

As I write this, Coronavirus / Covid-19 is on everyone's mind. For many, social distancing and self-quarantines will mean working from home for perhaps the first time. I've worked remotely for over 20 years. Let me share some of what has worked, and what hasn't for me.

I'll start with some general recommendations that concern mental and physical health as well as motivation. Then, I'll touch on some very specific tactics for remote working and teleconferencing. I'll finish up with some further tips to follow once life returns to a more normal (or "new normal") routine.

Health & well-being

Keep a regular routine — Remember, you're still working even if you're doing it at home. Get up, shower and get dressed, then get started on work just like you would if you were going into the office. Try to maintain consistent start and stop times for work. Take regular breaks, at least as often as you would at the office. Eat lunch on a regular schedule. Stop work when the day is done. Of course, you can be a little more flexible, but you'll keep up your motivation and focus if you keep a regular work routine.

Keeping a routine helps maintain a separation between work and personal time. Office workers get that separation thanks to the commute and physical separation between the office and home. You are not your job. Be sure to live your life.

Have a dedicated work space — Another way to maintain work/life separation is to have a dedicated work area in your home. It's best to have a separate room, with a door, especially if you live with family or others. Of course, not everyone has a spare room they can work in. Set up a dedicated area, a desk or table as separated from your relaxing spaces as you can manage. My preference is to not work from a bedroom or family room. Make sure to communicate to your family that when you're in that work spot, you shouldn't be disturbed because you're "at work."

Maintain your physical and mental health — Exercise, sleep right, eat well, and so forth. Get outside, even if for just a short time on your lunch break. Maintain your body, feed your soul.

Make time for socializing — Going hand-in-hand with the previous, socialize as much as you can, at least like you would have at the office. Chat with your co-workers, have virtual lunches (over a video link), set up a watercool chat mechanism, and so forth. I'm in multiple Slack groups — my work channel, a community group of fellow techies, a dedicated group for the robotics team I mentor, and so forth. Across those groups, I have access to channels to discuss hobbies, tech topics, and yes, even politics. These conversation channels help me maintain personal connections and not feel isolated.

If your job typically involves face-to-face interactions (meetings, one-on-ones, even office socializing) bias towards video conferencing over voice or chat. Even if you normally just hunker down in your cube, get in some face time (over video) with co-workers. Video conferencing helps maintain the personal connections you have with your co-workers, customers, and vendors.

Tactical tips

Mute yourself — Regardless of whether you're using audio or video channels, mute yourself if you're not talking to the group! As soon as you're done speaking, mute yourself again. Most of the videoconferencing systems enable you to mute attendees by default as they join the call. Use it.

Use a good headset — with a good mic, preferably a noise-canceling mic, for audio & video calls. Your co-workers will appreciate being able to hear you clearly and not hearing a ton of background noise and static. On the other hand, I like listening to music as I work. Specifically, I like to use speakers not headphones for this. YMMV.

Document and share — Document all decisions and take & share meeting minutes. It's harder to maintain focus and keep track of everything that goes on in a virtual meeting than an in-person meeting.

Start meetings on time — Whether you’re onsite or remote, show up on time. Attending meetings remotely is confusing enough already without the mess of random start times and late joiners.

When life returns to normal

Eventually, life will return to something close to its old routines. I hope this pandemic will foster more remote-friendly workplaces. With that in mind, you'll probably end up in a situation with a mix of on-site and remote workers. Here are some suggestions for when that time comes:

Get good A/V equipment for conference rooms — the mic on your laptop doesn't cut it. You need a proper room mic.
Make sure everyone in the office has proper headsets for those times when they have to participate in calls/meetings from their desks.
If you're the organizer or sitting closest to the room mic, use the good presenter technique of repeating questions so that those on the call can hear.
Avoid "hallway meetings" as a way of making decisions. And when that happens, make sure to inform your remote employees. Try to work in ways that include the remote employees (call them, quick grab a conference room and start a video chat, etc.)
Conduct all meetings as if everyone is remote. Try to not all gather in one room (meet from your desks).
Regularly check in with remote employees to make sure they know what tasks they're assigned, what roadblocks they've hit, etc.
Whatever you use for video conferencing, use it consistently. I worked at one place where it was random, sometimes Slack calls, sometimes Hangouts, sometimes Zoom, sometimes whatever freebie service the organizer had discovered someplace. That confusion sucks for everyone involved.
Practice good video conference hygiene — avoid stray noises, background conversations, mute yourself when not talking, let the remote person get a word in, ask them if they have input/questions. And for blessed sake, figure out connectivity and tech ahead of time so that not every call begins with “can you hear me?” and fumbling to get sound/video working. If a meeting must start late or early, make sure to let the remote folks know (and why).
If the office gets together for socializing, be sure to do something nice for the remotes. For example, send them a dinner gift certificate or movie tickets.
Avoid creating or working on teams where only one person is remote. It's too easy to leave them out or forget them, or bias your processes against remote employees.

There are literally whole books about working remotely and fostering remote-friendly workplaces. So, I'll end this here. Working remotely can be great. But it can also suck if it's not done right. Good luck! If you get totally down on this temporary situation, reach out. My social links are in the sidebar. We might be "socially distanced" but we're not alone.

Dark mode with CSS variables

2020-02-09T00:00:00-05:00

Dark mode is the hot schnitzel these days. Not wanting to be left behind, I recently updated my site to use a dark or light mode depending on your operating system preferences. Thanks to the magic of CSS media queries and CSS custom properties (otherwise known as CSS variables) this turned out to be rather easy.

CSS Media Queries

The Level 5 version of CSS media queries lets us specify styles based on the user's color scheme preferences. According to CanIUse.com, this query type is supported by all major browsers including Chrome, Firefox, Edge, and Safari. Internet Explorer and some Linux browsers do not support it.

To use this media query, we'd add something like the following to our CSS file:

@media (prefers-color-scheme: light) {
    /* light mode styles here */
}
@media (prefers-color-scheme: dark) {
    /* dark mode styles here */
}

This is great ... except I don't want to duplicate all my CSS rules to specify both light and dark mode colors. Thankfully, we can use CSS variables to limit duplication.

CSS variables

If you're not familiar with CSS variables, I highly recommend you check out some of the great articles at CSS Tricks. In a nutshell, we can define variable values, then use those variables in our rules. We need to specify a scope (where the variable is valid) and prefix the variable name with two dashes, like this:

:root {
    --link-color: rgba(24, 31, 50, 1);
    --some-other-color: white;
}

In the code above, my variable names are link-color and some-other-color and the scope is root, meaning the entire web page. While I'm showing colors here, you can specify any valid CSS values, for example, strings like 'center' for text alignment, numeric values, or special CSS keywords (like color names). Once I've defined my variables, I can use them in a CSS rule elsewhere in the file like this:

a {
    color: var(--link-color);
}

The var() syntax directs the CSS parser to use the variable's value — in this case for the text color of a link. Check out the CSS Tricks site I linked to above. You can do math on variables, use them in animations, and a whole lot more.

Putting it all together

By using the media query, my site sets color variable values based on your preference for light or dark mode. My CSS rules use those variable values to set colors for the elements on the site. Here's a snippet of my site's main.css file:

@media (prefers-color-scheme: light) {
    :root {
        --sidebar-bg: rgba(24, 31, 50, 1);
        --sidebar-link: rgba(191, 192, 192, 1);
        --article-fg: rgb(41, 45, 61);
        --article-bg: rgba(228, 228, 228, 1);
        --article-link: rgba(24, 31, 50, 1);
    }
}
@media (prefers-color-scheme: dark) {
    :root {
        --sidebar-bg: rgba(24, 31, 50, 1);
        --sidebar-link: rgba(191, 192, 192, 1);
        --article-bg: rgb(41, 45, 61);
        --article-fg: rgba(191, 192, 192, 1);
        --article-link: rgba(191, 192, 192, 1);
    }
}

And later in that CSS file, here are a couple of the rules I specify:

body, main, article {
    background-color: var(--article-bg);
    color: var(--article-fg);
}

aside.leftbar, div.leftbar {
    background-color: var(--sidebar-bg) !important;
}
aside.leftbar a {
    color: var(--sidebar-link);
}
a {
    color: var(--article-link);
}

By far, the hardest part was figuring out the various classes used by Pelican, the static site generator I use. Oh, and choosing colors.

It's Alive!

I think the coolest part of this technique is that it's applied automatically. If you change your OS preference, your browser will automatically update the page.

There you have it, dark mode, CSS variables, and media queries. Powerful stuff!

Controlling Christmas lights with MQTT

2020-01-02T00:00:00-05:00

I'm one of those Clark Griswold kind of guys that totally over-decorates his house. To make my lights more interesting, I've built my own light animation system which I call PiLit.

Of course, all my code is free and open source. Check it out at https://github.com/skypanther/PiLit.

Overview

PiLit is made up of three components:

Component	Function
PiLit player	The PiLit player is a python script that runs the show animation. It reads from a JSON "script" and sends commands on the network to control the lights.
Nodes	Nodes are the networked hardware components that control the lights.
PiLit GUI	A show generation web app to make creating show sequences relatively easy

The player can run on any computer that supports Python and can connect to the same network as your nodes. I use a Raspberry Pi 3B+ for this. The player sends MQTT messages to the nodes with instructions of what light sequence to play. The nodes are smart devices in that they "know" how to perform various lighting functions. They listen for MQTT messages telling them what function to perform, but they do the actual controlling.

There are a few benefits, and a few downsides to this architecture. On the plus side, much less data must be passed across the network and the nodes and player can use different architectures, programming languages, etc. In other words, the player can be a lot simpler but the nodes must be a bit more complex. Another downside is that to add new lighting functions, both the nodes and player need to be updated.

MQTT

I chose to use MQTT for message transport because there is very little overhead associated with the MQTT protocol. Messages are transmitted very quickly. Plus there are some great libraries for both Python and C/C++.

In PiLit, messages are basically in the form channel/payload where channel is the name of the node that should act on the message and payload is a string describing what animation to play. For example, a leaping arch node subscribed to an arch1 channel would listen for a message like this:

arch1/red:bounce:5:50:true

The message is a colon-delimited string that tells the arch what color to show, what animation sequence to play, and additional timing and control values. See the repo for details on the actual message payloads.

Nodes can subscribe to multiple channels, so a leaping arch might join both the arch1 channel and the all_arches channel. If all my leaping arches have joined all_arches then I can set them all to the same animation with a single MQTT message addressed to that channel.

Nodes

Nodes as I mentioned are smart devices. They are some type of microcontroller or single-board computer running code that I wrote. PiLit currently includes three types of nodes:

Node type	Platform	Language	Lights that are controlled
pixel_node	Arduino/ESP8266	C++	RGB "neopixels"
onoff_node	Arduino/ESP8266	C++	Single relay (e.g. to turn on/off a spotlight)
multi_relay	Raspbery Pi	Python	Multi-channel relay (e.g. a Sainsmart 16-relay board)

For microcontrollers, you could use just about any Arduino-like board. The ESP8266s are great because they offer better specs than most Arduinos, at a lower price, and include WiFi support built in. The downside of most any of these microcontrollers is that they're 3.3V devices and the typical RGB LED is a 5V device. So, you'll need a level shifter to convert the 3.3 into 5V signals. You can generally buy 3.3V relays for on/off nodes, eliminating the need for a level shifter.

The first Christmas light device I ever created was my Raspberry Pi-based megatree controller. Using a cheap 16-channel relay board, I controlled outlets into which the strings of lights on my tree were plugged. A Python script running on the Pi turned the outlets on or off.

I've rewritten and updated that original script to fit the PiLit architecture. But, since I had the hardware, I kept this node as a Python script. There's no reason an ESP8266 couldn't be used instead, though new node control software would be needed.

PiLit GUI

The show-generation app is for now called PiLit GUI. It's a React web app. Basically, I needed to learn React for work so this was a great "real world" app to learn on. (In other words, don't judge the code too harshly. I'm sure I have not written the best React app here.)

This app lets you define one or more channels, each of which corresponds to a node. Then, you define a series of animations for each channel. When done, you export your show to a JSON file. This file is the script you run with the PiLit Player.

Challenges

The system overall works well. The software portions of PiLit have worked out great. But I have had to deal with some physical / hardware challenges.

Keeping water out of the nodes has been a big problem. I put each in some sort of plastic container (an old food container, craft box, etc.). But of course, wires must go in and out of the boxes. Despite my best attempts, water has gotten in causing shorts. One box filled with water, then froze. The ESP8266 in that node did not like being frozen in a block of ice -- it fried. I also discovered that exterior latex caulk dissolves when exposed to constant moisture. Next year I'll use bathtub style caulk instead.

Another challenge has been solid wiring connections. The pixel strips are rather delicate. Wind and ice have broken my connections a few times. Apparently I'm not great at soldering since my connections at the microcontrollers have failed repeatedly too. More than once I've had to drag out my soldering tools and sit in the snow re-soldering connections.

Looking ahead

I designed and wrote PiLit, and built most of my nodes, in my spare time in September and November this year. Over the next year, I plan to refine my physical node builds to better protect from the elements. I also plan to enhance the software components. A few areas I may tackle include:

Improving timing — currently it's impossible to synchronize multiple nodes exactly. So for example, my three leaping arches don't "bounce" in time with each other.
Adding more lighting sequences to the nodes, as well as improving performance.
Updating the GUI show-creation app in various ways, such as offering a show preview mode and refining its UI.
Adding more instructions on building the nodes, as well as on setting up and using PiLit in general.

Finally, I'd really like to add an LED matrix display (to show images, video clips, and text) to my decorations. I'd also like to add an FM transmitter and audio track to my show. I will update PiLit to support those features if I can add those elements.

Multiprocessing in Python

2019-04-15T00:00:00-04:00

Modern CPUs typically feature multiple cores, which in some sense is like having multiple computers at your disposal. By default, your Python code will run on one core. But when performance is critical, you can use multiple cores to run operations simultaneously. In this article, I'll walk through how we used multiprocessing in our FIRST Robotics code to run our scripts across all four cores of our Jetson TX2 board.

Python supports both multithreading and multiprocessing. Judging by the number of StackOverflow posts on the topic, there's a ton of confusion surrounding these terms. There are many articles explaining the differences and nuances of multithreading and multiprocessing. Here's my take:

Multithreading — run portions of your code asynchronously so that your script doesn't have to wait for I/O bound operations; but all your code runs on the same CPU core, and shares the same memory and variable space.
Multiprocessing — run portions of your code simultaneously across multiple cores so that you script doesn't have to wait for CPU bound operations; because your code runs across multiple cores, each process has its own separate memory and variable space.

Use multithreading if your script needs to access data across the network, read a lot of data from disk, or do other I/O bound operations. Multithreading isn't really running code simultaneously. It's just not stopping everything while waiting for the I/O operation to finish.

Use multiprocessing if your script does a lot of computation, such as processing image frames from a webcam stream. You might employ multithreading to do the camera I/O. But multiprocessing would let you do the processing simultaneously with other code your script might perform. Sharing information between processes is more difficult since processes don't share memory or variables.

Multiprocess-bot

While we used both on our 2019 FRC bot, I'm going to focus on multiprocessing. Here are the tasks we programmed our Jetson to do:

One process captured frames from an IP camera, did target identification and field orientation calculations, then wrote the results to NetworkTables so that the data was available to the robot and driver computers.
Another process captured frames from a pair of web cams, received data via a queue, and used that data to create custom overlays atop the video feeds. Then it streamed those processed feeds across the network to the driver computer.
And we had a third process that instantiated and managed those processes.

Starting multi-process scripts

Let's start by seeing how we can start our multiple processes. (You might not want to actually run code like this till I cover how to stop the processes a bit later in this article.) In the main.py file, we import the two custom classes of our bot program, and then call their start() methods:

# import our bot-specific classes
from camstreamer import Camstreamer
from targeting import Targeting

if __name__ == "__main__":
    camstreamer_process = Camstreamer(...)
    target_process = Targeting(...)

    camstreamer_process.start()
    target_process.start()

That part looks pretty standard. The magic that enables multiprocessing is actually put in those custom classes. Let's take a look at the (simplified) code of the targeting.py script:

from multiprocessing import Process

class Targeting(Process):
    def __init__(self, ...):
        super(Targeting, self).__init__()
        # other initialization stuff here

    def run(self):
        # do the work of your class in this method

Some key points here: we import Process and then inherit from it when defining our class. This gives our class the capabilities to run as a distinct process. Then, as the first statement in our class's init method, we call the superclass's init method. For probably no good reason, I'm using the older Python 2.x syntax in that line. With Python 3.x, you can simplify that to super().__init__() (or use the older syntax). For more information on super(), check here and here.

As you saw above, we'll call our class's start() method to actually start the process. But you don't provide a start method in your subclass. Instead, you have to create a method named run(). The inherited start() method will call your run() method. It seems a little weird, but that's how it works.

Pipes and Queues

As I mentioned above, process don't share memory or variables. To communicate between processes, we need to use pipes or queues. Pipes are fast, one-way communication channels between a sender/receiver pair — put a message in at one end and it's immediately available at the other. Multiprocessing queues are two-way channels between multiple senders and receivers. They work like other FIFO queues you might be familiar with — the sender places a message onto the queue, the receiver pulls off the first-available message on the queue.

We used queues to share field / positional data between the processes. The targeting script would calculate its values and put them into the queue. The camera streamer class would pull values off the queue and use them in constructing the overlays. Let's see how that was accomplished.

In the main.py file, we create the queue instance and pass it to each of the processes that will use it. In this way, each process has a reference to the queue in its memory space. (Only thread/process-safe constructs, like Pipes and Queues can be shared between processes like this.)

from multiprocessing import Queue

if __name__ == "__main__":
    targeting_queue = Queue()
    camstreamer_process = Camstreamer(targeting_queue=targeting_queue ...)
    target_process = Targeting(targeting_queue=targeting_queue ...)

To write data to the queue, the targeting.py script just needed to call put() on the queue reference passed to the class, like this:

self.targeting_queue.put(field_data)

Our bot's camera streamer class read data from the queue. In that script, we used the get_nowait() method to read values from the queue. We could have used the get() method. However that function is blocking, meaning your script would pause till data was available on the queue. The get_nowait() method doesn't pause for data to be available, so it is faster. But it will throw an exception if there's nothing on the queue to read. To handle that, we just wrapped it in a try/except block, like this:

try:
    field_data = self.targeting_queue.get_nowait()
    # use the field data to draw the overlays
    # ...
except Empty:
    # exception thrown if there's nothing in the queue to read
    pass

We probably could have used a pipe for this communication, since it was a one-way stream. But, a queue has the advantage of holding multiple values. So, if the sender script gets ahead, messages will still be available to be processed when the reader is available. Pipes pass just a single value. If a new one is written before the previous value is read it will "push" the first value out of the pipe.

Stopping background processes

Once a script is launched as a separate process, it will run till stopped (or you kill it at the operating system level). During the competition, we were fine with the scripts running till we shut down our Jetson board. But, that's not typically how you'd manage your multiprocessing scripts.

Classes that inherit from Process have a terminate() method. When called, it will abruptly halt the process. If you need to close connections or otherwise gracefully stop your script, you'll want to signal it that it's about to be terminated. Once the process finishes its cleanup, then you call terminate().

In our bot program, we implemented this signalling using Pipes. In main.py, we listened for the Escape or "q" key to be pressed. When that happened, we sent a message across each child process's pipe that a shutdown was imminent. The child scripts would then stop their actions and clean up. After a brief delay, main.py would terminate the processes.

Looking again at main.py:

from multiprocessing import Queue, Pipe

stop_pipes = []

if __name__ == "__main__":
    cs_reader, cs_writer = Pipe(duplex=False)
    trgt_reader, trgt_writer = Pipe(duplex=False)
    stop_pipes.append(cs_writer)
    stop_pipes.append(trgt_writer)
    targeting_queue = Queue()
    camstreamer_process = Camstreamer(targeting_queue=targeting_queue, stop_pipe=cs_reader)
    target_process = Targeting(targeting_queue=targeting_queue, stop_pipe=trgt_reader, ...)
    ...

    while True:
        # wait for Esc or q key and then exit
        key = cv2.waitKey(1) & 0xFF
        if key == 27 or key == ord("q"):
            cv2.destroyAllWindows()
            for pipe in stop_pipes:
                pipe.put("stop")
            time.sleep(2)
            camstreamer_process.terminate()
            target_process.terminate()
            exit()

When instantiating a pipe, you get back a tuple whose values represent the read and write ends of the pipe. As shown in the snippet above, we created those pipe instances and passed the read ends to our child processes. In main.py's while loop, we use the write ends of the pipes (which were stored in a list) to put the string "stop" onto the pipe when the user signaled to quit.

In our child processes, all the computation was done inside an endless while True: loop. At the start of each iteration, we'd poll to see if data was available on the pipe. If so, and if it were the string "stop" we'd break out of the loop.

def run(self):
    ...
    while True:
        if self.stop_pipe.poll():
            # try reading from the stop pipe; if it's not empty
            # this block will work, and we'll exit the while
            # loop and terminate the script
            stop = self.stop_pipe.recv()
            if stop == "stop":
                break

Pipes are great for this sort of one-way communication. The main.py control script sent out stop messages, and the child messages listened for those messages.

Robovision on our 2019 FRC bot

2019-04-15T00:00:00-04:00

Now that our 2019 FRC season is over, I'd like to describe how Team 1518 implemented vision processing on our bot. We set some aggressive goals, which we aimed to achieve by using the the robovision library, OpenCV, and Python on a Jetson TX2. The primary goals of this post are to cover the high-level approaches we used and to share what did or didn't work. Of course, I'll link to the code we created.

We set out the following goals for vision on our bot this year:

Identify and isolate the retro-reflective tape targets
Determine our distance to those targets
Determine our angle to the plane of the targets
When aiming reasonably face-on, determine how far left or right of center we were situated relative to the targets

We had a few other needs, such as communicating with the RoboRIO and driver's station, and maintaining good performance while performing multiple tasks. We also decided to control a set of RGB "neopixel" LEDs to use as signals to the driver of the bot's state. And, we planned to use the Jetson to stream a pair of cameras back to the driver's station, with custom overlays.

That's a significant pile-of-priorities and I'm very proud of our sole vision sub-team programmer Emma. Working together, she and I accomplished all but one of those goals on the test field. Unfortunately, we never got to put it into "production." At the Finger Lakes Regional (FLR), our bot was overweight and we had to leave off the Jetson. Between then and Buckeye, we shaved off enough weight to permit the Jetson. But by then, the drive team was comfortable driving the bot manually and didn't want to deal with the new controls. Just the same, our system did work and we learned a lot from implementing it.

Targeting

The core function of our efforts were target identification and measurement. As shown below, a pair of retro-reflective vision tapes were mounted aside each hatch opening. The 2019 Game Manual listed various details of the tapes, for example that they were tipped at inward at 14.5° angles. Our goal was to isolate those tapes and make sure we were looking at a matched pair.

At a high level, we:

Grabbed frames from the camera
Used contour detection, looking for specific colors that represented the tape targets
Used a series of filters to be sure we were looking at a matched pair of tape targets while discarding false matches (e.g. the right tape of one set and the left tape of another)
Measured various characteristics of the targets when we found them
Shared that info with the Rio via NetworkTables and with the Arduino controlling our LEDs via serial-over-USB.

That's all implemented in the targeting.py file in the team's repo. There's a lot going on in this file. I'll go into some of the details below.

Finding and isolating the targets

The robovision library includes a pre-built contour detection method. We just needed to supply a range of colors to look for. We used the standard green LED ring light that comes in the FRC "kit of parts." We used an Axis IP camera with its exposure and brightness settings cranked about as low as they'd go. Through experimentation, we found that we could isolate the reflective tape by looking for colors between HSV (60, 100, 100) and HSV (100, 255, 255).

Looking at the run() function you can see we call robovision's target.get_contours(frame) in each iteration of the loop. Then, we call our own self.process_contours(contours) function. It gives us back info like whether a target is in view.

In that function, we sort the contours left-to-right so that we can methodically step through them in a known order. Then, we pass them through a series of "filters" to eliminate false detections:

Aspect ratio — the tape targets were 2-inches by 5.5-inches, giving an aspect ratio of 0.36. Any contour with an aspect ratio significantly different than this was ignored.
Solidity — a contour that encloses a "non-solid" region (think of something shaped like a letter "C") wasn't likely to be one of the rectangular tape targets. So, we'd ignore it.
Angle — the tape strips were attached at 14.5° angles. Contours that were not oriented at angles close to that were ignored. Robovision provides a convenient target.get_skew_angle(contour) function for finding the contour orientation angle.
And to be a valid target, we need a pair of tapes, one tilted right and one tilted left.

Positional orienteering

Once we had weeded out any false detections, we moved on to measuring and calculating. We used the "triangle similarity" technique, described on the PyImageSearch blog, to measure the bot's distance from the tape targets.

In the "lab," we measured the perceived width in pixels of a 12" piece of reflective tape at various distances. This gave us a ratio of pixels seen in the image to actual distance. At game time, we used robovision's target.get_extreme_points(contour) function to find the x/y coordinates of the corners of the tape targets. Using those coordinates, we counted the pixels between the top-most and bottom-most points. We plugged that pixel count, plus our precalculated ratio, into the triangle formula to determine the bot-to-target distance. This technique was accurate to a fraction of an inch.

We also used those corner coordinates to determine how far left or right the bot was in relation to the midpoint between the tape targets. Our original bot design included an articulated ball/hatch-cover manipulator. We planned to use this offset info to move that arm, alleviating the need for the bot to be centered on the hatch. We were able to calculate the offset within a fraction of an inch.

We were not successful in calculating the bot's angle to the hatch (plane of the target tapes). We tried a few techniques, none of which worked. For example, basic trigonometry tells us that the arc-cosine of the distance between the tapes divided by the distance to the tape equals the approach angle. In practice, we never got a good result. We decided that there were just too few pixels to work with, leading to gross inaccuracies. This is definitely an area we'll explore more over the off-season.

Communicating with the RoboRIO

Per FRC rules, the Rio must control the bot's actions. The parameters calculated by our vision code could be inputs to the Rio's routines. But for that to work, we needed to pass the calculated values in realtime to the Rio.

We considered a couple of options, including streaming data across a custom socket connection. In the end, we settled on a super-simple technique. The wonderful folks at the RobotPy project have created a NetworkTables implementation that worked flawlessly for us. We simply wrote our calculated values to a NetworkTable and the Rio read them from there. (Note: NetworkTables might be too slow for a high-speed, shooting game.)

Powering the Jetson

Our original plan was to power the Jetson from a cellphone powerpack. We found one on Amazon that output 12V and wasn't too heavy. This would have given us a stable power source that wouldn't abruptly shut off when the bot powered down. Unfortunately, the inspectors at Buckeye would not permit us to use that battery because they said it had too high an amperage rating.

After some research, we found that the Jetson TX2 is set up to handle an "uncontrolled" power loss by going through a fast shutdown process. According to the support forums, a power loss shouldn't cause filesystem corruption issues. So, we wired the Jetson directly to the power distribution panel (PDP) and it seemed to work fine. Of course, we didn't use it on the field so your mileage may vary.

Summary

We were intrigued by the Limelight system that some teams used this year. It promised to provide prebuilt routines to do many of the tasks we had to code ourselves this year. Nvidia just released the Jetson Nano, which is much smaller and lighter and runs off a 5V power source. Our TX2 may be retired in favor of one of these other systems. But the techniques we learned this year have given us a deeper understanding of vision processing. We'll put that knowledge to good use next year regardless of which system we use to implement our vision subsystem.

Per FIRST's rules for future re-use, all our code is on our GitHub repo. Feel free to explore it and use it in your projects; there's some cool stuff going on in there. In a future post, I'll go over the parallel processing code that we used to run multiple scripts across the cores of the Jetson TX2.

Introducing Robovision

2019-03-10T00:00:00-05:00

The FIRST robotics competition (FRC) challenges high school students to design and build a robot capable of performing multiple challenging tasks. These annual challenges typically involve computer vision components, such as identifying and using reflective markers to locate targets. High school computer science curriculum rarely covers software engineering topics, let alone advanced topics like computer vision.

To help FRC teams, I have written the robovision python library. This library includes functions useful for the types of vision tasks typically involved in an FRC competition. The goal of this library is to reduce and hide some of the complexity involved with target identification, measuring, field orienteering, etc.

Robovision is open source, licenced under the permissive MIT license, and free for all FRC teams to use. You can install it from PyPI or directly from the GitHub repo.

Some of the functions included in robovision are:

Multi-threaded image acquisition from a web cam, IP cam (i.e. Axis cam), Raspberry Pi camera, or Jetson onboard gstreamer camera
Lens distortion removal based on the camera calibrations created with the provided autocalibrate.py script
Retroreflective target identification, contour finding, and geometry finding functions
Image resizing, equalization, brightness and contrast adjustments, and more
A preprocessor class, which enables you to set up a pipeline of functions that will be applied in series to an image.
Overlay arrows, text, borders, or crosshairs on images
Rolling (moving) average calculations

Robovision complements the RobotPy library. It does not duplicate functionality, nor is it meant to replace that excellent project. Team 1518 uses both robovision and the cscore and networktables components of RobotPy.

Installation

Requirements:

Python 3.5+ (2.x is not supported)
OpenCV 3.4+ (4.x will probably work, but is untested)
Numpy

Robovision is meant to run on a coprocessor (e.g. a Jetson or Raspberry Pi) and hasn't been tested on the RoboRio itself. While it probably works on a Windows computer, it was developed and is tested only on Mac OS and Linux systems.

Installation details are covered in the wiki's Installation and System Setup page.

Example

Here's a sample of how you might use robovision to calculate the distance in inches to a 12" piece of retro-reflective tape held horizontally and face-on to an IP camera:

import cv2
import robovision as rv

source = "http://10.15.18.100/mjpg/video.mjpg"
fl = 100  # Calculated apparent focal length of your camera
cv2.namedWindow('CapturedImage', cv2.WINDOW_NORMAL)

# connect to our streaming video source and start the capture thread
vs = rv.get_video_stream(source)
vs.start()
# instantiate a robovision Target object which does all the work of finding
# and isolating the retro-reflective tape
target = rv.Target()
target.set_color_range(lower=(60, 100, 100), upper=(100, 255, 255))
while True:
    frame = vs.read_frame()
    # get a list of contours around objects within the color range set
    # above; these will be the retro-reflective tape assuming you're using
    # the standard AndyMark green LED light source
    contours = target.get_contours(frame)
    if len(contours) > 1:
        # draw a red border around each of the contours that were found
        cv2.drawContours(frame, contours, 0, (0, 0, 255), 3)
        # assuming you're holding the tape horizontally, you need only
        # the width (in pixels) of the detected object
        _, _, w, _ = target.get_rectangle(for_contour=contours[0])
        # Distance from formula: D’ = (W x F) / P
        d = 12 * fl / w
        print("Distance to tape: {} inches".format(d))
    cv2.imshow('CapturedImage', frame)
    # wait for Esc or q key and then exit
    key = cv2.waitKey(1) & 0xFF
    if key == 27 or key == ord("q"):
        cv2.destroyAllWindows()
        vs.stop()
        break

Examples and documentation

I include a selection of example scripts in the extras folder in the GitHub repo. While not really production ready, these show how to perform selected vision tasks that might be helpful for an FRC challenge. There's also a couple of scripts to help you do lens calibration (used to remove lens distortions, also called field flattening).

Be sure to check out the project's wiki for documentation on the library's classes and functions. You'll need the numpy and OpenCV python packages to use robovision. Some of the example scripts use additional libraries.

FRC Team 1518, Raider Robotics used robovision successfully in its 2019 Deep Space Challenge bot. So, another good source of examples (as well as some "incomplete thoughts") is the team's GitHub repository.

Future

Looking ahead, the library needs even more simplification and abstraction to make it easy for FRC teams to use. Additional documentation and examples are also needed. I have not extensively tested or optimized the library for the Raspberry Pi (we're using a Jetson TX2, which has plenty of horsepower for our vision tasks). I very much welcome pull requests and contributions to the project.

#omgrobots!

Using OpenCV in an iOS app

2019-02-01T00:00:00-05:00

Computer vision is cool tech ... computer vision in an iOS app is even better! In this post, I'll describe how you can integrate and use OpenCV in your Swift-based iOS app. Before we dig in to the process, let's take a look at how the finished integration will work.

As shown in the diagram, you will continue to write most of your app in Swift as you normally would do. The OpenCV framework will be included in your app, inside a wrapper that you'll write. A bridging header will connect your Swift code to the Objective-C / C++ code of OpenCV. (You'll also add a prefix/precompile header to optimize build times.)

Once you've set all that up, you'll be able to make use of OpenCV with Swift code like:

let edged: UIImage = OpenCVWrapper.detectEdges(inRGBImage: your_image)

But we're jumping ahead. There are a few steps to go through to get this all setup. Let's get OpenCV integrated and then I'll go through some usage examples.

Integration overview

At a high-level, you'll need to perform the following steps:

Add the OpenCV framework to your project
Create the wrapper
Create a bridging header
Create a prefix header
Implement the OpenCV functionality you need
Use those methods from Swift

Let's dive in.

Integration steps

First, download and unzip the OpenCV pack:

Go to https://opencv.org/releases.html
Find 3.4.4 (or other suitable version) and click the iOS pack link
Unzip the file to a convenient temporary location

Next, either create or open your Xcode project. You'll want to arrange Xcode and Finder side-by-side and then drag the opencv2.framework bundle (special folder) into the Xcode project tree, at the top level, to add it to your project. Note:

Make sure Copy items if needed is checked
Make sure Create folder references is checked
Make sure Add to targets: your_project is checked

You can close Finder at this point. We'll work in Xcode for the rest of this. Next, we need to create the wrapper around OpenCV. In doing so, we'll also create the bridging header.

Choose File > New > File
Create a Cocoa Touch Class file, naming it OpenCVWrapper (or another name if you prefer)
It should subclass NSOBject
It should be an Objective-C file
When prompted, click the Create Bridging Header button

Open the YourApp-Bridging-Header.h file and add this line:

#import "OpenCVWrapper.h"

That's all you'll need to do inside your bridging header. With this simple file in place, you've created a bridge between Objective-C (where you'll write the actual OpenCV code) and Swift (where you'll use it).

Next, we need to make a couple of changes to the OpenCVWrapper.m file. First, change the name of the OpenCVWrapper.m file to OpenCVWrapper.mm. Doing so informs Xcode to treat it as a C++ rather than C file. Then, inside that file, make sure the imports at the top are in this order. If you don't, you will get weird compile errors later.

#import <opencv2/opencv.hpp>
#import "OpenCVWrapper.h"

Finally, we need to set up the prefix header. Again, there's not much that you'll need to do with this file. It will simply tell Xcode to not recompile the already-compiled OpenCV framework to speed up your builds.

Choose File > New > File
Scroll to near the bottom and choose PCH File
Make sure your app is checked in the Targets list
Click Create

Inside the prefix header, before the closing #endif statement, add:

#ifdef __cplusplus
#include <opencv2/opencv.hpp>
#endif

That is, you're done! Well, you can build the app now and OpenCV is integrated. Of course you've done nothing to implement OpenCV's functions yet. (By the way, you'll get many OpenCV-related build warnings which you can safely ignore. They mostly relate to inline documentation comments being in the wrong format and won't affect your app's execution in any way.)

Implementing an OpenCV function

The basic steps you'll follow to implement an OpenCV function and use it in your app are:

Add the function signature to the OpenCVWrapper.h file
Add the function's implementation code in the OpenCVWrapper.mm file
Call that function in your Swift file

For example, your .h header declares the function and its signature, which might look like this:

#import <Foundation/Foundation.h>

NS_ASSUME_NONNULL_BEGIN

@interface OpenCVWrapper : NSObject

+ (NSString *)openCVVersionString;

@end

NS_ASSUME_NONNULL_END

Your .mm file would have the corresponding function implementation:

#import <opencv2/opencv.hpp>
#import "OpenCVWrapper.h"

@implementation OpenCVWrapper

+ (NSString *)openCVVersionString {
    return [NSString stringWithFormat:@"OpenCV Version %s",  CV_VERSION];
}

@end

Per Objective-C syntax, the + indicates a class method; use - to indicate an instance method. The difference is probably inconsequential since there will only ever be one instance of your OpenCVWrapper class. So, you can probably use + to prefix all of your functions as shown above.

OK, so let's use this powerful new OpenCV capability. In your main ViewController.swift file, update viewDidLoad to call that new function:

override func viewDidLoad() {
    super.viewDidLoad()
    print("\(cv2.openVersionString())")
}

Build to a simulator and watch the Xcode debug console and you should see OpenCV Version 3.4.4 there. Congratulations, you've integrated OpenCV into your iOS app.

A better example

I've created a simple project so that you can explore something a little more meaningful. This app lets you take a picture (so you'll need to run it on a device) and then apply some OpenCV manipulations.

Download the OpenCVTest Xcode project.
Update the project properties with your organization / app-signing profile.
Download and drag the opencv2.framework bundle into the Xcode project tree (it's too big to include in my zip file). I used OpenCV 3.4.4 and any of the 3.x version should work. But I haven't tested with 4.x, so YMMV.
Build the app to a device. The app should be fairly self-explantory.

Make sure to explore the .h and .mm files, as well as the ProcessImageController.swift file in this project. Also check out the UIImage extension. Together, these should give you a good start on writing your own OpenCV functionality.

Closing thoughts

You'll need to write your OpenCV code in Objective-C (that's what's in the .mm file). But as best as I can tell, there is no specific OpenCV-for-iOS documentation. For the most part, any C++ OpenCV sample code you find will work without much modification. From there, I'm sorry to say you'll have to turn to StackOverflow and Google and hope.

I've found OpenCV's functions to be typically much faster than the equivalent Swift implementations. For example, converting an image to grayscale is considerably faster with OpenCV. But test, because some OpenCV operations are slower, sometimes surprisingly so. Blurring an image, for example, is many times faster in Swift than in OpenCV.

But, avoid passing images back-and-forth between Swift and OpenCV. You'll need to change data formats (e.g. from OpenCV Mat to UIImage or CIImage) which can be quite slow and you may encounter memory leaks. Instead, do all the processing you can in one layer or the other and pass only the final result between layers.

Images you capture from an iPhone (and presumably iPad) camera can be appear to be turned upside-down by OpenCV. In fact, it's just a different way of handling the orientation information in the files. Check out the UIImage extension that I've included in the sample project above. As you'll see in that project, you can get a correctly oriented portrait image by simply accessing the normalized property of the image, like this: OpenCVWrapper.convert(toGrayscale: img.normalized!)

Good luck and have fun building great OpenCV vision-enabled iOS apps!

Publishing to PyPI

2019-01-25T00:00:00-05:00

I recently published my first Python package to PyPI. The guides I found on how to do so were mostly out-of-date and confusing. Of course, PyPI is reportedly coming out with new updates soon and my instructions here will soon be outdated. In any case, here's my take on how you can publish your own project.

(Note, the following guide will work for OS X and Linux. You'll need to make some adjustments for Windows, which I apologize but I'm unable to provide guidance for.)

I recommend you start right and use CookieCutter to create the shell of your project. There are a couple of Python package templates that will give you the shell of what you need. (I used the cookiecutter-pypackage-minimal template, but the "ultimate" template might fit your needs better.)

Many guides suggest you reserve your package's name on PyPI right away so that no one else "steals" it. However, PyPI no longer supports package name reservations. The best you can do is search ahead of time to be reasonably sure your chosen name is not already taken.

Local package installation

As you develop, you can test that your package is installable and working by installing it from your local folder. Use pip to install from your development folder:

cd path/to/your/package/folder
pip install -e .

PyPI registration

Once your package is ready, you'll need to create accounts on both pypi.org and test.pypi.org. PyPI says those systems will be linked eventually. But for now, the test and production environments are completely separate. You can use the same information to register for both sites.

The `twine` package uploader

You'll be using twine to upload your package to PyPI. Let's get that set up. First, install it with:

pip install twine

You'll need to create a configuration file in your home directory. Using your favorite code editor, create a file named .pypirc in your home directory with the following contents:

[distutils]
index-servers=
    testpypi
    pypi

[testpypi]
repository = https://test.pypi.org/legacy/
username = your_username
password = your_password

[pypi]
username = your_username
password = your_password

Take note that there's no repository line for the pypi (production) entry. Previous versions of twine required that line but it will cause an error with current versions.

Configure distribution options

You'll need to choose between the wheel and egg packaging formats. In most cases, you'll want to choose wheel. See the linked guide if you think you might need to use the older egg format.

Next, you're going to need to make sure you have a few files in your package directory. If you used one of the CookieCutter templates I linked to above, you'll have these. But, you'll need to make sure they're updated with correct information.

Use the ReStructuredText format for your readme or the info on the PyPI site will be all borked up. That means you'll need a README.rst file in the root directory of your project. Supposedly PyPI supports Markdown but don't believe them. Most code repositories (GitHub, etc.) support ReStructuredText readmes, so you don't need to create multiple files.

Make sure to include a LICENSE and/or LICENSE.md file to announce your package's license.

You'll need to update the setup.cfg file to reflect your package publishing options. First, assuming you're using the wheel format, you'll need to set universal=1 for a package that targets both Python 2.x and Python 3.x or universal=0 if targeting just Python 3.x. Second, make sure the description-file references your readme file's name. Here's my setup.cfg:

[bdist_wheel]
universal=0

[metadata]
description-file=README.rst

Assuming you used CookieCutter, you should have already configured the basics of the setup.py file. It notes various settings and is used by setuptools to create your distribution file. Make sure that the version number, readme file name, license, and other options are correct for your package.

Note that if you have Python files in subdirectories off your main package folder, you'll need to list them in the packages=[] list. See my setup.py file and my source code organization if this doesn't make sense to you.

Create your distribution file

With all that configuration out of the way, you're ready to create your distribution wheel file. Use this command:

python setup.py bdist_wheel

This will package up your file and create a wheel file in the dist directory in your project.

Uploading

You should upload to the test environment first, and install your package from the test environment. That way, you're assured everything is working as expected before you announce your package to the world. Use this command:

twine upload dist/YOUR_PACKAGE_NAME.whl -r testpypi

Use a command like this to install from the test environment:

pip install -i https://test.pypi.org/simple YOUR_PACKAGE_NAME

Once you're sure everything is in order, upload to the production PyPI server:

twine upload dist/YOUR_PACKAGE_NAME.whl

Congratulations! You've published a Python package to PyPI.

Updating your package

Going forward, as you make improvements and additions to your library, you'll need to do just a couple of things to publish an update.

First, update setup.py with the new version number
Update your readme to note any new features, and probably new version number
Use python setup.py bdist_wheel to create a new distribution file
Use twine upload dist/YOUR_PACKAGE_NAME.whl to upload it

Raspberry Pi for IoT

2018-12-14T00:00:00-05:00

I gave a presentation to the Buffalo Python meetup group in December. My talk centered around using the Raspberry Pi as a platform for IoT and embedded development. I want to share my presentation and the resources I mentioned here on my blog.

Update I presented an expanded version of this talk to the RocDev group in January. My updated slides are here as a PDF file. The links below still apply.

View/download as a PDF, PowerPoint, or Keynote.

During my talk, I referred to a bunch of sites and resources. These included:

Getting started with the Pi

The Raspberry Pi home page
OS downloads, including Raspbian and more
Etcher, for copying OS images to an SD card
The Official Raspberry Pi Beginner's Guide looks to be a good starter book for younger RPi tinkerers.

Development tools and docs

Visual Studio Code for the Raspberry Pi
Pi Wheels precompiled python libraries
PiCamera docs

Blogs and tutorial sites

Adafruit offers many great tutorials on the Raspberry Pi, Arduino, and other electronics topics. Their GitHub account has tons of free code, too.
SparkFun has great electronics tutorials, more towards the microcontroller (Arduino) type level than the Pi.
Pololu's blog has some info, but unlike some other vendors, their individual product listings often come with sample code and how-to information.
The PyImageSearch blog is a great resource for OpenCV and computer vision. Adrian, the author, often includes Raspberry Pi specifics for his posts, though most can also be implemented on other platforms. I specifically mentioned his motion detection / tracking post (and its second part)
And while we're on the topic of OpenCV, Satya Mallick's LearnOpenCV.com blog has tons of great computer vision information and he generally posts both Python and C++ code for every example.
And humbly, my blog right here

Stores / resellers

Adafruit - Pi, Arduino, and other electronic components
SparkFun - Generally a bit more Arduino/microcontroller oriented components
Pololu - Robotics, electronics, and other components.
I usually purchase Raspberry Pis from Amazon. Just be sure you're getting the model you want (the 3B+ is the current model) since many vendors are still selling the older boards.

Note: Unlike earlier models, the 3B+ has somewhat stringent power demands. You'll need a 5v power brick that has a 2.4 amps or greater output rating. That old cell phone charger you've got probably won't cut it.

My projects

Disclosure, the code here sucks and is nothing I'd show as part of a job interview. But, it gets the job done.

My PiLit Christmas light controller project
And my someday-I'll-actually-finish-it automated cat feeder

Getting user input with OpenCV trackbars

2018-10-16T00:00:00-04:00

Trackbars, or sliders, are a simple way to get user input when working with OpenCV. In this post, I’ll show you how to add trackbars to an OpenCV window, read their values, and use those values in your Python scripts.

To demonstrate a use for trackbars, we’ll read user input from a pair of trackbars and use that input to tune edge detection parameters in real time. I’ll show you both the common way of reading trackbar values, and a more performant way to structure your code.

Adding trackbars

In order to use trackbars, you have to use a named OpenCV window. We use that name to specify which window to attach the trackbar to.

# first, create your named window
cv2.namedWindow('Window_Name')
# next, create the trackbar
cv2.createTrackbar('Trackbar_label', 'Window_Name', start_value, max_value, callback)
# finally, show something in the window, the trackbar will be attached
cv2.imshow('Window_Name', image)

Let’s take a quick look at the parameters we pass to createTrackbar. The first param is a label that will be shown next to the trackbar. Next is the window name as discussed above. The minimum trackbar value is always 0, thus the third param is setting the starting value. Next comes the max value to which the trackbar can be set. Finally, we pass a reference to a function that will be called each time the trackbar’s value changes.

Edge detection

Edge detection is the process of identifying edges in an image. Edge detection is useful for feature detection and feature extraction, as well as other applications. OpenCV supports a few edge detection algorithms; we’ll be using the Canny edge detection method. In essence, this algorithm looks for a quick change from light to dark pixels (or vice versa) extending across some minimum area in an image to be considered an edge.

A picture is worth a thousand words...

Compare this image to the one at the top of the post. You can see that the edges (white lines) have been drawn in the areas where the itensity changed from light to dark, or dark to light, mostly marking the edges of the lamp.

Canny edge detection is a multi-step process. The image is blurred slightly, a Sobel filter is applied to find and emphasize strong gradients in the image. Then finally, a process of finding edges and filtering out false edges is performed. The Canny algorithm accepts three parameters to tune its operations. These are a minimum and maximum threshold and the aperture size.

The aperture size is used in the Sobel filter, and in my experience seems to have the least effect on Canny results. The min and max threshold values, on the other hand, strongly affect the outcome. Tuning them just right will result in edges being detected; getting them wrong will result in unimportant gradient areas being identified as “edges.”

Tuning with trackbars

With that explanation out of the way, let's get down to code. We’ll use a pair of trackbars to let the user set the values of the min and max thresholds in the Canny edge detection algorithm. You can grab the image I used from Unsplash.

Let's start with the shell of our script:

import argparse
import cv2
import imutils

def main():
   min_val = 200
   max_val = 300
   aperture_size = 3

   ap = argparse.ArgumentParser()
   ap.add_argument('-i', '--image', default='',
                   help='Image to use for edge detection')
   args = vars(ap.parse_args())
   file_name = args['image']
   image = cv2.imread(file_name)
   image = imutils.resize(image, height=480)
   gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

if __name__ == '__main__':
   main()

In the script above, we import our required libraries and begin to build out our main() function. We'll be accepting the name of the image as a command line argument. So, we instantiate the ArgumentParser and specify our one argument.

I'm resizing the image in my demo here, but keep in mind that doing so will reduce the effectiveness of edge detection. In a production app, you will probably want to work with full-size images. Finally, edge detection works best on a grayscale image, so I convert it with cvtColor.

Next, let's add the code to create our named window and trackbars. We'll attach these to the window showing the original version of the image.

   cv2.namedWindow('Original')
   cv2.createTrackbar('Min', 'Original', 0, 800, no_op)
   cv2.createTrackbar('Max', 'Original', 100, 800, no_op)
   cv2.imshow('Original', image)

We're not going to use the callback function, which is called when the user moves the trackbar. For that parameter, we'll pass a no_op function, which we'll add in a minute. First, let's add a loop where we will check for user input.

   while True:
       min_val = int(cv2.getTrackbarPos('Min', 'Original'))
       max_val = int(cv2.getTrackbarPos('Max', 'Original'))
       print('Min: {}'.format(min_val))
       print('Max: {}'.format(max_val))
       edges = cv2.Canny(gray, min_val, max_val, aperture_size)
       cv2.imshow('Edges', imutils.resize(edges, height=480))
       if cv2.waitKey(1) & 0xFF == ord("q"):
           cv2.destroyAllWindows()
           exit()

There are a few things going on in that loop! First, we use getTrackbarPos to get the trackbar position. This function takes two parameters: the label of the trackbar we're reading from and the name of the window it's attached to. For our example script, we need the value as an integer, so we convert it with int().

Here's also where we perform the edge detection. We call the cv2.Canny method passing in the image, min and max values, and the aperture size. This returns the edge-detected version which we show in a new cv2 window.

Let's not forget our no_op function to finish this off:

def no_op(new_val):
   pass

You can grab the entire script from this Gist.

A better trackbar

So, that script works. But, when you run it, you'll notice a couple of problems. The print statements spam your console. That loop is running continuously so that it can read user input. Even with that, the trackbars are somewhat unresponsive. You can't slide them too quickly and the script is slow to reflect your changes.

We can fix those problems by using the callback functions we ignored earlier. The downside to this approach is that you can't control what arguments are passed to the callback; just the new trackbar value is passed. So, we'll need to be a little creative. Here's the full script with explanation following it:

import argparse
import cv2
import imutils
import os

edge_params = {
    'min_val': 200,
    'max_val': 300,
    'aperture_size': 3
}
gray = None

def main():
    global edge_params, gray
    ap = argparse.ArgumentParser()
    ap.add_argument('-i', '--image', default='',
                    help='Image to use for edge detection')
    args = vars(ap.parse_args())
    file_name = args['image']
    if os.path.isfile(file_name) is False:
        print('Cannot open image, quitting...')
        exit()
    image = cv2.imread(file_name)
    image = imutils.resize(image, height=480)
    gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

    cv2.namedWindow('Original')
    cv2.createTrackbar('Min', 'Original', 0, 800, min_change)
    cv2.createTrackbar('Max', 'Original', 100, 800, max_change)
    cv2.imshow('Original', image)
    redraw_edges()

    while True:
        if cv2.waitKey(1) & 0xFF == ord("q"):
            cv2.destroyAllWindows()
            exit()

def min_change(new_val):
    change_params('min_val', new_val)

def max_change(new_val):
    change_params('max_val', new_val)

def change_params(name, value):
    global edge_params
    edge_params[name] = value
    print(edge_params)
    redraw_edges()

def redraw_edges():
    edges = cv2.Canny(gray,
                      edge_params['min_val'],
                      edge_params['max_val'],
                      edge_params['aperture_size'])
    cv2.imshow('Edges', imutils.resize(edges, height=480))

if __name__ == '__main__':
    main()

I've added a couple of global values to store references to our grayscale image and our edge detection parameters. The createTrackbar calls now point to meaningful callback functions. But, since both are doing about the same thing, just on different data, each of those functions simply call the change_params function with an identifier of the data to change. I've moved the edge detection code to its own function too.

This script is also available at the same Gist linked to above. When you run it, you'll notice there's no console output until you move one of the trackbars. The trackbars should be a bit more responsive too. Performance will never be great. You're not writing a native compiled GUI app. But, it's certainly usable to get quick user input.

Summary

So, there you have it, trackbars (sliders) on an OpenCV window. Depending on your operating system, they'll be attached to either the top or bottom of the window (if there's a way to control that, I haven't found it). I don't think there's a limit to how many you can attach, other than the size of your screen. I find them very useful for operations like the edge detection I demoed with here; operations where I need to tune some parameters and don't want to keep updating code and re-running a script.

Multiple cameras with a single Raspberry Pi

2018-09-02T00:00:00-04:00

The Arducam Multi Camera Adapter board is a neat accessory for a Raspberry Pi. With it, you can connect multiple cameras to a single Pi. A single board supports up to four cameras. According to Arducam, you can stack up to four boards for a total of 16 cameras on a single Pi.

I had a chance to use a couple of these boards in a recent project. Our goal was to capture multiple images to form an image mosaic. My setup included a Raspberry Pi 3B+, two of the Arducam multicamera boards, and six cameras. (Only the 5MP v1 Pi cameras, or Arducam's 5MP camera are supported, not the 8MP v2 camera.)

Since Arducam's documentation is sparse, I thought I'd share my experiences with this adapter as well as a Python library I wrote that simplifies accessing each of the cameras.

First, you should know that you cannot capture the images simultaneously. The adapter works by enabling one camera at a time. So, you can take photos one after the other, but there will be a small delay between each. Arducam does not specify how long it takes to switch between cameras. Our code pauses for 0.1 second between photos though we could probably wait less time than that.

Hardware setup

Let's start with the hardware side. The adapter board will use 26 of the GPIO pins of the Pi, leaving you the last 14 pins free for other purposes. You will connect the Pi's camera port (called the CSI port) to the adapter board with a ribbon cable. Then, you'll connect the CSI cable from each of your cameras to the ports on the adapter.

Key hardware setup points:

Connect the Arducam board starting with GPIO pin 1
Connect the Pi's CSI port to the Arducam board, pin side of the ribbon towards the board (blue/tape side up)
Connect camera CSI cables to the Arducam board, pin side of the ribbon towards the board (blue/tape side up)
It does not matter whether you connect the cameras in order (to port A, then B, etc.) though it will matter for your code

Stacking multiple boards

How about stacking multiple boards? You'll need to solder on the 10-pin connector that Arducam supplies to each of the adapters. These are the high-speed MSI bus over which the camera data is transferred between boards. Because of this connector, the bottom adapter didn't sit straight on my Pi3B+ since the connector hits the HDMI connector.

You'll need to set DIP switches on the boards to identify which of the OE (output enable) pins that board will use. For board 1, switches 1 and 5 must be on and the rest off. For board 2, switches 2 and 6 are on. See the Arducam site for the full list.

Pictures on the Amazon listing for the adapter shows the CSI cable going between the bottom board to the Pi. In their emails to me, Arducam suggested connecting the ribbon cable to the top-most board. (It made no difference for me where it was connected.)

Software

With that hardware setup out of the way, let's move onto the software side of things.

Channels and ports

The adapter enables multiple camera support by providing two channels, each with two camera ports. You select which camera to use by enabling one of the channels and one of the camera ports. You do so by setting certain GPIO pins high or low.

Arducam calls these the channel select (CS) and output enable (OE) pins. All of the stacked boards share the same CS pin but have their own OE pins (hence the DIP switches you must set). There's a listing of which pins you must set on the Arducam site, though my library hides that complexity.

Caveat

I must point out that I couldn't get stacked adapters to work. Even after multiple emails back and forth with Arducam, connecting a second adapter would cause my Pi to crash as soon as I tried accessing any camera. So, while my library supports up to 16 cameras, I can only be sure it works with four.

The multicam library

I have not published the library to PyPI (and probably won't). Instead, I've put it on a GitHub Gist. Using it is pretty simple, though it has a couple of dependencies.

RPi.GPIO - installed by default in Raspbian Stretch (full version)
PiCamera - install with pip install picamera

Because my project used OpenCV, by default, my library returns captured photos as an OpenCV "BGR" formatted blob. You can pass the image_format param to the capture method if you want to use another format. Any format supported by PiCamera's capture method will work.

OpenCV is not installed by default, check out PyImageSearch's tutorial for install instructions if you want to use OpenCV on your Pi.

Examples

Using OpenCV, a minimal example follows:

import cv2
from multicam import Multicam

mcam = Multicam()

img = mcam.capture(cam='a')
cv2.imshow('Cam A', img)
cv2.waitKey(0)
mcam.cleanup()

Or, with Matplotlib, you just need to specify the rgb image format:

from matplotlib import pyplot as plt
from multicam import Multicam

mcam = Multicam()

img = mcam.capture(cam='a', image_format='rgb')
plt.figure().canvas.set_window_title('Cam A')
plt.imshow(img)
plt.show()
mcam.cleanup()

Of course, you could do the above with a single camera. Where this library shines is of course accessing the other cameras:

import cv2
from multicam import Multicam

mcam = Multicam()

cv2.imshow('Cam A', mcam.capture(cam='a'))
cv2.imshow('Cam B', mcam.capture(cam='b'))
cv2.imshow('Cam C', mcam.capture(cam='c'))
cv2.imshow('Cam D', mcam.capture(cam='d'))
cv2.waitKey(0)
mcam.cleanup()

There's no need to worry about setting GPIO pins, selecting channels, etc. The library does all that for you. If you have multiple adapters stacked, you can keep going (up to camera 16, or "p")

import cv2
from multicam import Multicam

mcam = Multicam()

# Camera 5  
cv2.imshow('Cam E', mcam.capture(cam='e'))
# Camera 16
cv2.imshow('Cam P', mcam.capture(cam='p'))
cv2.waitKey(0)
mcam.cleanup()

Notes and miscellaneous

Each of the above examples ends with a call to cleanup. This releases the GPIO ports. You should always call this function at the end of your script to avoid errors if you re-run the script (without restarting the Pi).

By default, my library uses "board" port assignments. This means GPIO ports are accessed according to their physical pin number. This use matches the examples on the Arducam site.

However, some libraries, like gpiozero, are limited to using the Broadcom numbering scheme. If you're using such a library, you can tell the multicam library to use "bcm" numbering during initialization:

from multicam import Multicam

mcam = Multicam(gpio_mode='bcm')

Conclusion

If you buy one of these boards, grab a copy of my multicam library to ease working with the cameras. Please comment on that Gist if you find issues. It's MIT licensed so free to use in any of your projects.

Finding the dominant colors of an image

2018-07-24T00:00:00-04:00

The typical way to isolate or find an object in an image is to look for its color. You specify a range of colors, then use OpenCV to identify regions in an image that contain colors within that range. But, even if you know the exact color of your target, lighting, shadows, and your camera's sensor will alter the detected color. So, how do you best determine the color range to use?

In this article, I'll explore some different methods to work with colors. In a future article, we'll use this knowledge, plus selecting a region of interest within an image/video to determine the exact range of colors needed to isolate an object.

Let's say your goal is to isolate the puppy in this image. You'll notice that his fur color varies between an off-white to a golden tan. The foreground is brighter, making his hindquarters darker and less richly colored. The toy he has is only a bit darker red-brown.

An average solution

Your first attempt might be to take an average of the colors to find the midpoint of his range of colors. Then, to make the range you might choose colors a bit darker and lighter than that average. Of course, you wouldn't want to include the grass in the average. Assuming a cropped version of just the puppy, you could use this script to calculate the average color:

#
#  Find the average color in an image
#
import cv2
import numpy as np

img = cv2.imread('puppy_cropped.jpg')
height, width, _ = np.shape(img)

# calculate the average color of each row of our image
avg_color_per_row = np.average(img, axis=0)

# calculate the averages of our rows
avg_colors = np.average(avg_color_per_row, axis=0)

# avg_color is a tuple in BGR order of the average colors
# but as float values
print(f'avg_colors: {avg_colors}')

# so, convert that array to integers
int_averages = np.array(avg_colors, dtype=np.uint8)
print(f'int_averages: {int_averages}')

# create a new image of the same height/width as the original
average_image = np.zeros((height, width, 3), np.uint8)
# and fill its pixels with our average color
average_image[:] = int_averages

# finally, show it side-by-side with the original
cv2.imshow("Avg Color", np.hstack([img, average_image]))
cv2.waitKey(0)

Which would give you this:

Visually, that resulting average tan looks like it would represent the puppy's colors. In practice though, you probably won't find any pixels in the puppy that match that exact shade. Even if you looked at a range of shades lighter to darker than that average, only a few of his pixels would be included. It's even likely you'd get a few pixels of his toy. As bad as the average works with the puppy, the average would totally fail on a multi-colored object, for example the FIRST^® logo.

FIRST®, the FIRST® logo, FIRST® Robotics Competition (formerly also known as FRC®), FIRST® Tech Challenge (formerly also known as FTC®) are trademarks of For Inspiration and Recognition of Science and Technology (FIRST®).

K-means clustering

Let's try a more powerful method. We'll find the most common colors in our image using k-means clustering. A formal definition would go something like "k-means clustering partitions n observations into k clusters". In our case, we have a bunch of pixels (our n) and we want to pull out some number (the k) of colors.

With k-means, you have to specify the number of clusters up-front. There are other techniques that don't have this requirement. In any case, it's not a significant limitation for our needs here.

We'll use the scikit-learn library's sklearn.cluster.kmeans function to do the heavy lifting. (OpenCV itself offers a kmeans() function, but scikit's is a bit more flexible for our needs). The following script is based on a post on the PyImageSearch blog (a great resource!)

#
#  Use k-means clustering to find the most-common colors in an image
#
import cv2
import numpy as np
from sklearn.cluster import KMeans


def make_histogram(cluster):
    """
    Count the number of pixels in each cluster
    :param: KMeans cluster
    :return: numpy histogram
    """
    numLabels = np.arange(0, len(np.unique(cluster.labels_)) + 1)
    hist, _ = np.histogram(cluster.labels_, bins=numLabels)
    hist = hist.astype('float32')
    hist /= hist.sum()
    return hist


def make_bar(height, width, color):
    """
    Create an image of a given color
    :param: height of the image
    :param: width of the image
    :param: BGR pixel values of the color
    :return: tuple of bar, rgb values, and hsv values
    """
    bar = np.zeros((height, width, 3), np.uint8)
    bar[:] = color
    red, green, blue = int(color[2]), int(color[1]), int(color[0])
    hsv_bar = cv2.cvtColor(bar, cv2.COLOR_BGR2HSV)
    hue, sat, val = hsv_bar[0][0]
    return bar, (red, green, blue), (hue, sat, val)


def sort_hsvs(hsv_list):
    """
    Sort the list of HSV values
    :param hsv_list: List of HSV tuples
    :return: List of indexes, sorted by hue, then saturation, then value
    """
    bars_with_indexes = []
    for index, hsv_val in enumerate(hsv_list):
        bars_with_indexes.append((index, hsv_val[0], hsv_val[1], hsv_val[2]))
    bars_with_indexes.sort(key=lambda elem: (elem[1], elem[2], elem[3]))
    return [item[0] for item in bars_with_indexes]


# START HERE
img = cv2.imread('puppy_cropped.jpg')
height, width, _ = np.shape(img)

# reshape the image to be a simple list of RGB pixels
image = img.reshape((height * width, 3))

# we'll pick the 5 most common colors
num_clusters = 5
clusters = KMeans(n_clusters=num_clusters)
clusters.fit(image)

# count the dominant colors and put them in "buckets"
histogram = make_histogram(clusters)
# then sort them, most-common first
combined = zip(histogram, clusters.cluster_centers_)
combined = sorted(combined, key=lambda x: x[0], reverse=True)

# finally, we'll output a graphic showing the colors in order
bars = []
hsv_values = []
for index, rows in enumerate(combined):
    bar, rgb, hsv = make_bar(100, 100, rows[1])
    print(f'Bar {index + 1}')
    print(f'  RGB values: {rgb}')
    print(f'  HSV values: {hsv}')
    hsv_values.append(hsv)
    bars.append(bar)

# sort the bars[] list so that we can show the colored boxes sorted
# by their HSV values -- sort by hue, then saturation
sorted_bar_indexes = sort_hsvs(hsv_values)
sorted_bars = [bars[idx] for idx in sorted_bar_indexes]

cv2.imshow('Sorted by HSV values', np.hstack(sorted_bars))
cv2.imshow(f'{num_clusters} Most Common Colors', np.hstack(bars))
cv2.waitKey(0)

That's a bunch of code I realize. This script does a few things; let's go through it. Skip the functions for now and jump to the "Start Here" part. The script reads in our image and determines its height and width. Then we convert it from a height by width matrix into a list of RBG values for easier processing.

Next, the script examines the image (our cropped puppy) and uses KMeans() to find the five most common colors (lines 59 - 61). The make_histogram() function essentially counts the pixels in each of those "buckets" returning those counts as a value between 0 and 1. We sort those "buckets" into a descending order of most to least common. Finally, starting on line 70, the code outputs some useful info about those dominant colors. We loop through our "buckets" to create images (aka "bars") from those colors and show them in OpenCV windows.

Output

Check out your console: the script outputs the RGB and HSV color values for each of the colored boxes.

> python dominant_color.py 
Bar 1
  RGB values: (131, 114, 99)
  HSV values: (14, 62, 131)
Bar 2
  RGB values: (153, 133, 114)
  HSV values: (15, 65, 153)
Bar 3
  RGB values: (141, 138, 136)
  HSV values: (12, 9, 141)
Bar 4
  RGB values: (165, 156, 148)
  HSV values: (14, 26, 165)
Bar 5
  RGB values: (99, 87, 75)
  HSV values: (15, 62, 99)

Given our cropped puppy image, the script outputs two CV2 windows. The first shows colored boxes for the top 5 dominant colors, listed most to least common left to right. If you move that window out of the way, you'll see a similar one showing those same colors arranged in HSV order (sorted lowest to highest by hue, saturation, then value).

It might be surprising when seen as RGB boxes on your screen, but the dominant colors are fairly close in hue, even in saturation. In fact, hue values range between 12 and 15. (Sidenote: OpenCV represents hue ranges from 0-180, not 0-360 as graphics apps typically do.)

This gives us just what we need to isolate objects in an image or video stream. In a future article, I'll show how to isolate objects based on their color. My plan is to then go on to show how to discern info about the object, such as its actual size, position in the real world, and so forth. But, let's not get ahead of ourselves!

In summary

In this article, I showed how to find an average color of an image. While I didn't go on to prove it with an object isolation script, average colors are typically the best way to represent a target object. Instead, I showed a more powerful technique, the k-means cluster. I showed how to determine the five most common colors. The script I provided output convenient RGB and HSV values that you can use for object isolation.

Source materials and further explorations:

Selecting a region of interest within an image/video (from this site)
PyImageSearch: OpenCV k-means color clustering
StackOverflow: How to find the average colour of an image in python
Adam Spannbauer: Dominant colors in app icons

Ad-hoc objects in Python

2018-07-06T00:00:00-04:00

If you know one programming language, it's natural to look for parallels as you learn a new language. In my case, I've coded in JavaScript for many years while Python is much more recent for me. I regularly find myself thinking "in JavaScript, I'd do ..."

Take object handling. I really like JavaScript's dynamic handling of objects. With JS, there's no need to create a class template (though you can). Just create an ad-hoc object and start assigning it properties and methods. For example:

// JavaScript example
var js_obj = {};

// dynamically add a property
js_obj.foo = 'bar';

// and use it
console.log(js_obj.foo); // bar

// you can even create methods
js_obj.sum = function(x, y) {
    return x + y;
}
// and use them
console.log(js_obj.sum(3, 4)); // 7

What wasn't obvious to me at first was that I could do similar things in Python. I saw the class ClassName() syntax and was fooled into thinking that I had to predefine my class with all its attributes and methods before I could instantiate objects from it. Not perfectly parallel, but in Python, we can do this:

# Python example
class Object(object):
    pass


py_obj = Object()
py_obj.bar = 'baz'

print(py_obj.bar)  # baz

# we can dynamically assign methods with lambdas
py_obj.sum = lambda x, y: x + y

print(py_obj.sum(3, 4))  # 7


# or even full functions
def exponential(num):
    if num < 0:
        return 'undefined'
    exp = num
    start = num - 1
    for step_down in range(start, 0, -1):
        print(step_down)
        exp = exp * step_down
    return exp


py_obj.exp = exponential

print(py_obj.exp(5))  # 120

That's great. But a leg-up that JS has is that objects are "associative arrays" meaning that in addition to the dot-notation, we can use array notation (e.g. Python's dict notation) to access properties of JS objects:

// this works in JS:
js_obj['foo']

# but in Python, this wouldn't work
py_obj['bar']

Why would that matter you ask? If you need to dynamically access properties of an object, say by looping through a list of keys returned by an API, the associative array notation comes in very handy.

Here I present a simple class that gives similar functionality in Python. The Object class will create an object that you can access with either dot or dict notation. It offers support for iteration via items() and could be used by methods expecting a __getitem__ accessor.

class Object(object):
    '''
    Creates an object for simple key/value storage; enables access via
    object or dictionary syntax (i.e. obj.foo or obj['foo']).
    '''

    def __init__(self, **kwargs):
        for arg in kwargs:
            setattr(self, arg, kwargs[arg])

    def __getitem__(self, prop):
        '''
        Enables dict-like access, ie. foo['bar']
        '''
        return self.__getattribute__(prop)

    def __str__(self):
        '''
        String-representation as newline-separated string useful in print()
        '''
        state = [f'{attr}: {val}' for (attr, val) in self.__dict__.items()]
        return '\n'.join(state)

    def items(self):
        '''
        Enables enumeration via foo.items()
        '''
        return self.__dict__.items()

To use it:

generic_object = Object(foo='bar')

print(generic_object.foo)  # bar
print(generic_object['foo'])  # bar

// add attributes dynamically
generic_object.baz = 123
print(generic_object.baz)  # 123

# iterate over its attributes
for key, value in generic_object.items():
    print(key, value)

# even use its convenience string representation
print(generic_object)  # prints multi-line string of key/values

This simple generic object class uses a couple of the special attribute names common to all Python objects. By overriding __getitem__, __str__, and items we give custom functionality to objects created from this class.

Just because you can, doesn't mean you should. Many would argue that ad-hoc objects like this are not "Pythonic." There are other data structures in Python that can serve a similar purpose as generic containers of data:

There's the Dict (dictionary) and for pre-Python 3.7, OrderedDict. The disadvantage to dicts is the way you access their attributes — my_dict['some_attribute']. It's wordy, and to me inelegant compared to the my_object.some_attribute syntax of objects.
Named tuples offer similar functionality, though like "normal" tuples, they are immutable.
And starting with Python 3.7, you have data classes (also see Hackernoon's overview article). If you can use Python 3.7, data classes are superior to my simple class above (such as enforcing type hinting) so use them instead.

As always, use the data structure that best suits your needs and will most clearly communicate to future readers of your code its purpose and requirements. But don't be afraid to explore the ins and outs of the language too.

Path prediction with OpenCV

2018-06-03T00:00:00-04:00

Recently, I spotted an article on HackerNews, Self-driving RC car that uses AI to predict turning angles. The project looked interesting. They applied AI to predict the path a radio-controlled car should take to stay centered over a tape track on the floor. Using a Raspberry Pi and PiCam mounted to the car, their algorithm performed path prediction and controlled the car's servos via the Pi's GPIO pins.

As I read the article and looked through their GitHub project, I realized that much of what they were doing with AI could probably be done nearly as effectively with OpenCV. My solution below won't cover all the functionality of the original project. But, at a basic level, here's what we'll do instead:

Acquire an image and pre-process it.
Isolate the tape from the background by strongly thresholding the image.
Find the largest contour that surrounds the tape track.
Find the minimum size bounding box around that contour.
Finally, calculate the angle of that bounding box to the Y axis

That angle is the needed turning angle to keep the car on track. From there, given the car's speed, you would want to apply some smoothing (turn somewhat less than the angle) to avoid jerky, sudden turns. (I'll note some other limitations below.)

The script is short enough that I'll just include the whole thing here. Comments inline explain the code.

import cv2
import imutils
import glob
import os
import numpy as np

# for demo purposes, we'll use some static JPGs
# of a green tape line
cwd = os.path.dirname(os.path.realpath(__file__))
full_path = os.path.join(cwd, 'images')

for filename in glob.glob(os.path.join(full_path, '*.jpg')):
    image = cv2.imread(filename)
    # https://github.com/jrosebr1/imutils has a handy resize func
    resized = imutils.resize(image, width=480)
    # convert to grayscale
    img = cv2.cvtColor(resized, cv2.COLOR_BGR2GRAY)
    # threshold fairly agressively assuming a bold tape
    # color on a white background
    _, thresh = cv2.threshold(img, 240, 255, cv2.THRESH_BINARY_INV)
    # get the countours
    _, cnts, _ = cv2.findContours(thresh,
                                  cv2.RETR_LIST,
                                  cv2.CHAIN_APPROX_SIMPLE)
    # sort the contours
    cnt = sorted(cnts, key=cv2.contourArea)
    # get the min rotated bounding rect of the largest one
    rect = cv2.minAreaRect(cnt[-1])
    # grab the height, width, and angle of that rect
    height, width = rect[1]
    angle = rect[2]
    # The value of OpenCV's angle depends on the rect's
    # height/width ratio. https://stackoverflow.com/a/24085639/292947
    if width < height:
        angle += 90
    # do something here with the angle to steer the car
    # we'll just print it
    print(angle)
    # and then, for visual purposes of this article, draw
    # the bounding box on and show the resized image
    box = cv2.boxPoints(rect)
    box = np.int0(box)
    cv2.drawContours(resized, [box], 0, (0, 0, 255), 2)
    cv2.imshow("Image", resized)
    cv2.waitKey(0)

Here's a sample of what is output:

You can see the angle output to the console (34.04°) as well as the bounding box around the contour (which outlines the green tape). If you want to try yourself with the images I used, I've put them here: angler_images.zip.

Limitations

As may be obvious, there are a few limitations to this approach. First, as shown above, the algorithm wouldn't permit the car to closely follow the line. If the car turned the 34-degree angle calculated above, it would cut the corner rather than following the bend in the tape.

Perhaps less obvious is that the car could end up driving off center from the tape. Consider the following diagram:

When the car captured this image, it was offset from the tape. The vertical dashed line approximates the center of the image. The solid black line approximates the center of the green tape track, which is ideally where we want the car to drive. Instead, if the car simply turned the roughly 10 degrees calculated by our simple algorithm, it would drive next to the tape track following the solid green line.

Ideally, we'd want to calculate the offset of the tape and adjust our angle. In the simple case shown in the above diagram, simply driving straight forward for a moment or two would resolve the offset. But, if we were well off the tape track, we might even need to turn left instead of the calculated right.

More formally, we would want to adjust our calculated bounding box angle by approximately the angle formed by the tape offset and the height of the image. We'd find the tape offset (the base of our triangle) by finding the center point of the bottom of the bounding box and subtracting that from the center of the image. Then, the adjustment angle would be the arctan(tape_offset ÷ image_height).

The final limitation to consider, and something faced by the original project, is the processing speed of the Raspberry Pi. The faster you drive the car, the faster the images would need to be processed. At some point, the car will drive faster than the Pi can calculate. Your car would drive enough between frames that the tape would no longer be in the captured image.

Summary

Obviously, I haven't put this algorithm to the real test. Maybe someday I'll dig my RC-10 out of the garage and try it for real. Just the same, I still believe that OpenCV and a little trigonometry would be enough to let a toy car automatically follow a tape track, no artificial intelligence needed.

Handling mouse events in OpenCV

2018-05-21T00:00:00-04:00

For a project I'm working on, I need to select a region of interest in a video frame and extract info about that region. Fortunately, that's a pretty simple task to do in OpenCV. In this article, I'll show you how to detect and react to mouse actions, such as clicking and moving. Then, as an example of what to do with that sort of functionality, we'll see how to crop an image to the dimensions selected.

You'll need Python and OpenCV installed. I'm using Python 3.6 (Anaconda) and OpenCV 3.4 myself. Open your favorite text editor and create a file named crop_image.py and enter the following code:

import argparse
import cv2

coords = []
drawing = False

def main():

def get_image(source):

def click_and_crop(event, x, y, flag, image):

if __name__ == "__main__":
    main()

That gives us the shell of the file we'll fill in throughout the rest of this article. We'll need the argparse and cv2 (OpenCV) libraries, as well as a couple of global variables. We'll have a couple of functions plus main() and the code to call main() when we run the script.

Let's fill in the rest of main():

def main():
    # construct the argument parser and parse the arguments
    ap = argparse.ArgumentParser()
    ap.add_argument("-i", "--image", help="Path to the image")
    args = vars(ap.parse_args())
    image_source = args["image"] if args["image"] else 0
    # now get our image from either the file or built-in webcam
    image = get_image(image_source)
    if image is not None:
        # show the captured image in a window
        cv2.namedWindow('CapturedImage', cv2.WINDOW_NORMAL)
        cv2.imshow('CapturedImage', image)
        # specify the callback function to be called when the user
        # clicks/drags in the 'CapturedImage' window
        cv2.setMouseCallback('CapturedImage', click_and_crop, image)
        while True:
            # wait for Esc or q key and then exit
            key = cv2.waitKey(1) & 0xFF
            if key == 27 or key == ord("q"):
                print('Image cropped at coordinates: {}'.format(coords))
                cv2.destroyAllWindows()
                break

That's a good chunk of code. Let's walk through what it's doing. Using argparse, the script will look for and parse command-line arguments. This script will accept a single argument (either -i or --image) to specify an image file to load. We'll store the value in an image_source variable on line 12. If no image is specified, we'll store 0 in the variable -- that will tell the script to use the default webcam on the system as the image source.

One line 14, we call our get_image() function (which we've yet to write) to load our image. If we successfully load the image, the statements in the if block will run. line 17 createa a named cv2.imshow window and the next line shows the image in that window.

Line 21 is perhaps the key line here. With setMouseCallback() we specify which window to listen for mouse events on ('CapturedImage'), what function to call each time there's a mouse event (a click, drag, move, etc.), as well as an optional third parameter. The cv2.setMouseCallback() function lets us pass whatever value we want in that third parameter. In the docs and other examples, you'll typically see it named generically as param for this reason. We'll use it to pass the image to that function.

Finally, we want the image window to remain open while the rest of the script operates. So, we use a while loop. In each iteration of the loop, we use the cv.waitKey() function to listen for a key press. If the key pressed is the escape key (value 27) or the letter q (the script would actually receive the ordinal value of the character 'q'), then we destroy all the cv2 windows and break out of the loop, ending the script.

There's the main logic of our script -- get an image, show it, then watch for mouse events and call a function when they happen. We've got a couple of other functions to write. Let's tackle get_image() first.

def get_image(source):
    # open the camera, grab a frame, and release the camera
    cam = cv2.VideoCapture(source)
    image_captured, image = cam.read()
    cam.release()
    if (image_captured):
        return image
    return None

This function is pretty straightforward. We'll use cv2.VideoCapture() to open whatever source it's passed (which will be either a filename or 0 for the webcam). It will read a single frame, then return the captured image.

The bigger function is click_and_crop() which is our event handler. Let's see this one in stages.

def click_and_crop(event, x, y, flag, image):
    """
    Callback function, called by OpenCV when the user interacts
    with the window using the mouse. This function will be called
    repeatedly as the user interacts.
    """
    # get access to the two global variables we'll need
    global coords, drawing

Here we see the function signature. When called, the function will be passed five arguments. The first is the event, such as the left button being pressed down (cv2.EVENT_LBUTTONDOWN). Next comes the x/y coordinates of the event. I could not find documentation on the flag parameter, other than the docs saying it was an integer. In testing, it appears to identify the event type and modifier keys. For example on my Mac, while moving the mouse, flag is 0. Holding down the Control key while moving the mouse, it was 8. Holding down Shift and moving was 16. Since this may vary by platform, test before using this parameter.

The last param, image, corresponds to whatever third parameter we passed in the cv2.setMouseCallback() call. In our case, our call looks like cv2.setMouseCallback('CapturedImage', click_and_crop, image) where we're passing the captured image in that parameter.

Next, we're going to handle three different mouse events associated with cropping an image. First, the user will click the mouse down at the top-left corner. Then, they'll drag down to the bottom right. And third, they'll release the mouse button. I'll cover these three separately. As you examine this code, remember, this function will be called repeatedly, every time there's a mouse event (such as moving the mouse over the image).

    if event == cv2.EVENT_LBUTTONDOWN:
        # user has clicked the mouse's left button
        drawing = True
        # save those starting coordinates
        coords = [(x, y)]

When the user clicks the button down, we'll set our global drawing variable to True. We'll use this for tracking state in the next sections. Then, we store the x/y coordinates of the mouse down event in our global coords list.

    elif event == cv2.EVENT_MOUSEMOVE:
        # user is moving the mouse within the window
        if drawing is True:
            # if we're in drawing mode, we'll draw a green rectangle
            # from the starting x,y coords to our current coords
            clone = image.copy()
            cv2.rectangle(clone, coords[0], (x, y), (0, 255, 0), 2)
            cv2.imshow('CapturedImage', clone)

The user will move their mouse all over the image, even when they're not trying to crop it. We want this section of the code to run only if they've clicked down the mouse button. For that reason, we check the drawing variable set in the mouse down portion of the function. When that's true, we'll create a clone of our original image. Next, we'll draw a rectangle on the clone with the top-left corner at coords[0] (where the user clicked the mouse button down) and the current x/y coordinates. We'll draw a green outline (remember, OpenCV specifies the colors in blue-green-red order) with a width of 2. Finally, we show that image in our CapturedImage window.

    elif event == cv2.EVENT_LBUTTONUP:
        # user has released the mouse button, leave drawing mode
        # and crop the photo
        drawing = False
        # save our ending coordinates
        coords.append((x, y))
        if len(coords) == 2:
            # calculate the four corners of our region of interest
            ty, by, tx, bx = coords[0][1], coords[1][1], coords[0][0], coords[1][0]
            # crop the image using array slicing
            roi = image[ty:by, tx:bx]
            height, width = roi.shape[:2]
            if width > 0 and height > 0:
                # make sure roi has height/width to prevent imshow error
                # and show the cropped image in a new window
                cv2.namedWindow("ROI", cv2.WINDOW_NORMAL)
                cv2.imshow("ROI", roi)

Finally, we handle the case when the user releases the mouse button. This is the bottom-right corner of the cropping rectangle. We set drawing back to False so we stop processing mouse-move events. We append the new x/y coordinates to our global list. Assuming something didn't go wrong, that list will have two members.

On line 70, we crop the image. OpenCV images are represented as Numpy arrays of pixel values. We're using array slicing, in perhaps a non-inuitive way. Let's step back to line 68, where we grab the y values from the top-left and bottom-right corners (so ty and by), and then the x values from the two corners (tx and bx) out of our global coords list.

We're ready to crop the image. OpenCV images are represented as Numpy arrays of pixel values. Since the image is just an array, we can use array slicing to select a portion of it -- in other words, to crop it. On line 68, we grab the y values from the top-left and bottom-right corners (so ty and by), and then the x values from the two corners (tx and bx) out of our global coords list. Then on line 70, we crop by slicing the image array to those ty/by, tx/bx values and storing the "region of interest" in the roi variable.

If the user were to select a very small slice, such that either the width or height were treated as zero, the imshow() function would throw an error. To avoid that, we'll calcualte the height and width of the cropped image by grabbing the first two members of the roi's shape. We do our zero-test and then show the roi image in a new OpenCV window.

And that's it! If you haven't so far, try it out.

python3 crop_image.py

Click, drag, and release on the "CapturedImage" window and you'll get a new window with the cropped portion of your original. It may show up behind other windows you have open. (That's an annoying OpenCV bug on some platforms.)

The whole shebang

Finally, let's close by seeing the entire script. Hopefully you've typed in the code as we went and didn't just copy & paste it. But this here will let you check your work against my original.

import argparse
import cv2

coords = []
drawing = False

def main():
    # construct the argument parser and parse the arguments
    ap = argparse.ArgumentParser()
    ap.add_argument("-i", "--image", help="Path to the image")
    args = vars(ap.parse_args())
    image_source = args["image"] if args["image"] else 0
    # now get our image from either the file or built-in webcam
    image = get_image(image_source)
    if image is not None:
        # show the captured image in a window
        cv2.namedWindow('CapturedImage', cv2.WINDOW_NORMAL)
        cv2.imshow('CapturedImage', image)
        # specify the callback function to be called when the user
        # clicks/drags in the 'CapturedImage' window
        cv2.setMouseCallback('CapturedImage', click_and_crop, image)
        while True:
            # wait for Esc or q key and then exit
            key = cv2.waitKey(1) & 0xFF
            if key == 27 or key == ord("q"):
                print('Image cropped at coordinates: {}'.format(coords))
                cv2.destroyAllWindows()
                break

def get_image(source):
    # open the camera, grab a frame, and release the camera
    cam = cv2.VideoCapture(source)
    image_captured, image = cam.read()
    cam.release()
    if (image_captured):
        return image
    return None

def click_and_crop(event, x, y, flag, image):
    """
    Callback function, called by OpenCV when the user interacts
    with the window using the mouse. This function will be called
    repeatedly as the user interacts.
    """
    # get access to a couple of global variables we'll need
    global coords, drawing
    if event == cv2.EVENT_LBUTTONDOWN:
        # user has clicked the mouse's left button
        drawing = True
        # save those starting coordinates
        coords = [(x, y)]
    elif event == cv2.EVENT_MOUSEMOVE:
        # user is moving the mouse within the window
        if drawing is True:
            # if we're in drawing mode, we'll draw a green rectangle
            # from the starting x,y coords to our current coords
            clone = image.copy()
            cv2.rectangle(clone, coords[0], (x, y), (0, 255, 0), 2)
            cv2.imshow('CapturedImage', clone)
    elif event == cv2.EVENT_LBUTTONUP:
        # user has released the mouse button, leave drawing mode
        # and crop the photo
        drawing = False
        # save our ending coordinates
        coords.append((x, y))
        if len(coords) == 2:
            # calculate the four corners of our region of interest
            ty, by, tx, bx = coords[0][1], coords[1][1], coords[0][0], coords[1][0]
            # crop the image using array slicing
            roi = image[ty:by, tx:bx]
            height, width = roi.shape[:2]
            if width > 0 and height > 0:
                # make sure roi has height/width to prevent imshow error
                # and show the cropped image in a new window
                cv2.namedWindow("ROI", cv2.WINDOW_NORMAL)
                cv2.imshow("ROI", roi)

if __name__ == "__main__":
    main()

In this article, you saw how to detect and react to mouse actions, such as clicking and moving. Then, as an example of what to do with that sort of functionality, I showed you how to crop an image to the dimensions selected. In a future article, I will show you how to extract information from that region of interest, such as the predominant color.

Reading email with Python

2018-05-10T00:00:00-04:00

For a recent project at work, I needed to read and parse email messages with a python script. I found the documentation confusing, and most of the samples on various blogs and StackOverflow posts to be old and not fully compatible with python 3.x. So, here is an adaptation of my solution.

Further below in this post is my actual class for interacting with an IMAP email account, logging in and out, accessing the messages in a folder, and even deleting messages. But let's start with a simple script that uses the class.

Using the class

As I'm sure you know, it's a bad idea to code passwords into a file. Instead, you should load them at runtime, either from a command line argument (e.g. with the argparse library) or from the system environment. The script I wrote for work was going to be part of a Django project deployed in a Docker container, so for my needs an environment variable was the best choice.

Before you can use the script below, you'll have to add a mailpwd environment variable (or modify the code to use argparse). On Linux/OS X, use export mailpwd=password and on Windows, er, sorry, I don't know.

from .ImapClient import ImapClient


def main():
    """
    You will need to store your email password in your environment, e.g.
    export mailpwd=password
    """
    imap = ImapClient(recipient='you@gmail.com')
    imap.login()
    # retrieve messages from a given sender
    messages = imap.get_messages(sender='a_friend@another_isp.com')
    # Do something with the messages
    print("Messages in my inbox:")
    for msg in messages:
        # msg is a dict of {'num': num, 'body': body}
        print(msg['body'])
        # you could delete them after viewing
        # imap.delete_message(msg['num'])
    # when done, you should log out
    imap.logout()


if __name__ == "__main__":
    main()

Some peculiarities of the class

For my work needs, I needed to access a GSuite (Gmail for business) email account. So, the class is a bit specific to Gmail. While it will work with other providers, you may need to tweak the delete_message method since it is specific to Gmail's trash-vs-deleted folder locations.

Another specific need I had was to monitor emails from a specific sender. Thus, as shown on line 12 above, you must pass in a sender's email address when calling get_message(). The script would need some rewriting to accommodate other needs. But hopefully what's below will get you started.

The class

The class itself is considerably longer and more involved than the script that uses it. The following should be in a file named ImapClient.py in the same directory as the script that uses it. Look the code over. I'll explain some its points after the code.

import email
import email.header
import imaplib
import os
import sys


class ImapClient:
    imap = None

    def __init__(self,
                 recipient,
                 server='imap.gmail.com',
                 use_ssl=True,
                 move_to_trash=True):
        # check for required param
        if not recipient:
            raise ValueError('You must provide a recipient email address')
        self.recipient = recipient
        self.use_ssl = use_ssl
        self.move_to_trash = move_to_trash
        self.recipient_folder = 'INBOX'
        # instantiate our IMAP client object
        if self.use_ssl:
            self.imap = imaplib.IMAP4_SSL(server)
        else:
            self.imap = imaplib.IMAP4(server)

    def login(self):
        try:
            rv, data = self.imap.login(self.recipient, os.getenv('mailpwd', ''))
        except (imaplib.IMAP4_SSL.error, imaplib.IMAP4.error) as err:
            print('LOGIN FAILED!')
            print(err)
            sys.exit(1)

    def logout(self):
        self.imap.close()
        self.imap.logout()

    def select_folder(self, folder):
        """
        Select the IMAP folder to read messages from. By default
        the class will read from the INBOX folder
        """
        self.recipient_folder = folder

    def get_messages(self, sender, subject=''):
        """
        Scans for email messages from the given sender and optionally
        with the given subject

        :param sender Email address of sender of messages you're searching for
        :param subject (Partial) subject line to scan for
        :return List of dicts of {'num': num, 'body': body}
        """
        if not sender:
            raise ValueError('You must provide a sender email address')

        # select the folder, by default INBOX
        resp, _ = self.imap.select(self.recipient_folder)
        if resp != 'OK':
            print(f"ERROR: Unable to open the {self.recipient_folder} folder")
            sys.exit(1)

        messages = []

        mbox_response, msgnums = self.imap.search(None, 'FROM', sender)
        if mbox_response == 'OK':
            for num in msgnums[0].split():
                retval, rawmsg = self.imap.fetch(num, '(RFC822)')
                if retval != 'OK':
                    print('ERROR getting message', num)
                    continue
                msg = email.message_from_bytes(rawmsg[0][1])
                msg_subject = msg["Subject"]
                if subject in msg_subject:
                    body = ""
                    if msg.is_multipart():
                        for part in msg.walk():
                            type = part.get_content_type()
                            disp = str(part.get('Content-Disposition'))
                            # look for plain text parts, but skip attachments
                            if type == 'text/plain' and 'attachment' not in disp:
                                charset = part.get_content_charset()
                                # decode the base64 unicode bytestring into plain text
                                body = part.get_payload(decode=True).decode(encoding=charset, errors="ignore")
                                # if we've found the plain/text part, stop looping thru the parts
                                break
                    else:
                        # not multipart - i.e. plain text, no attachments
                        charset = msg.get_content_charset()
                        body = msg.get_payload(decode=True).decode(encoding=charset, errors="ignore")
                    messages.append({'num': num, 'body': body})
        return messages

    def delete_message(self, msg_id):
        if not msg_id:
            return
        if self.move_to_trash:
            # move to Trash folder
            self.imap.store(msg_id, '+X-GM-LABELS', '\\Trash')
            self.imap.expunge()
        else:
            self.imap.store(msg_id, '+FLAGS', '\\Deleted')
            self.imap.expunge()

Class details

The login() and logout() methods should be fairly self-explanatory. The select_folder() method is optional. By default, the class will read from the INBOX folder but you could change that with this method.

As you can see in the __init__ method, you must specify a recipient address. This is the email account you're logging into. You can override other defaults, such as the server to log into, whether to connect over SSL, and so forth.

Let's dig into the get_messages() method since it's the meat of what the class is offering. After checking params, it attempts to select the folder to search. Of the two values returned, we care only about the first one resp which will be the string 'OK' if we succeed. Next we search for messages from the sender (line 68). Again, two values are returned, but this time we want to use both.

As before, the first value, mbox_response is an OK/not okay string. The second value is a list, the first member of which is the list of message numbers that match our search criteria. Every IMAP message in the folder is identified by an integer ID. To fetch the actual message, we'll use the imap.fetch() method, passing to it the message ID number, and the portion of the content we want to retrieve. If you've looked at other IMAP examples, you'll see folks using many other content selectors. Here, we're using '(RFC822)' to basically grab the entire message.

Assuming that was retrieved from the server successfully, on line 75 we convert the raw representation of the message into a mail object we can further inspect. Then, we grab the subject on line 76. My original needs called for looking for messages with a given subject. It's optional; if you don't pass in a subject param to the function you'll retrieve all the messages from the sender.

Starting on line 78, we extract the text from the message. The full IMAP specification (RFC 822) is quite complex. But for our needs now, we can ignore many of those details. Messages can be single or multipart. A single part message would be a very simple plain text email message, the type generated by email clients of the days before pretty formatted emails. (Such messages are still sent, though more typically these days by scripts/generators rather than by users with some mail app.) Multipart messages represent their contents in multiple formats, such as plain text and HTML formatted. The typical Gmail, Outlook, webmail generated email message would be a multipart message containing probably both a plain text and HTML formatted part.

On line 79, we determine the message type and on line 80, we begin walking the parts looking for the plain text part. We do so in two ways. First, with part.get_content_type() we determine whether it is plain/text or not. But, a text attachment to a message would have that content type too. So, on line 84 we check for that by looking at the content disposition, which will include the string 'attachment' if this part is an attachment.

Finally (and we do this for both the multipart and single part sections) we need to convert the raw byte representation of the part into plain text. We determine its original character set (e.g. ISO-8859-1, UTF-8, etc.) using part.get_content_charset() and then convert to that representation on line 87 (or 92).

Each message that makes it through this filtering and converting is added as a dict to the messages list. We return both the message ID and its text. And then finally we return that list to the caller.

Deleting messages

Deleting a message from an IMAP server involves two steps: copying it to the trash folder and calling expunge() to remove it from its original folder. With Gmail, there are two "deleted" folders: Trash and Deleted. The first of these is like the Recycle Bin / Trash folder on your computer. Messages in Trash are easily recoverable. Depending on your mail provider, messages moved to Deleted could be fully removed. Google never deletes anything, even when you empty the trash. You can always do a search to find messages in the Deleted folder (but you'll need to know a subject or sender or some other criterion to find it).

The delete_message() method requires a message ID and will then move that message to the Trash (default behavior) or delete the message based on the value of move_to_trash parameter you supplied when instantiating the ImapClient object.

Summary

While the class is a bit involved, it really only scratches the surface of IMAP message complexity. Still, this class makes for a fairly simple means to grab messages from an email inbox and read their plaintext representations. I hope it helps clarify how IMAP works and gives you a good starting point for your email needs.

Streaming birdcam with a Raspberry Pi and picamera

2018-05-04T00:00:00-04:00

My darling wife got me a Raspberry Pi 3B+ and a NOIR Pi camera for my birthday so that I could set up a streaming camera to monitor our nesting bluebirds. Here's how I did it.

My first attempt was to use the Motion program as described by Hackernoon. It worked, but the resulting video stream was very slow and jerky. I'd say I was getting only a few frames a second at best. I knew I could do better.

Picamera

The better solution turned out to be picamera. This is a Python module for using the Raspberry Pi camera. The author has not only written a fantastic library, but he's also provided lots of great recipes (sample scripts) showing how to use it.

His pistreaming recipe looked to be perfect for my needs. Raspbian Stretch comes with Python installed. So I jumped right into the pistreamer install steps described on the pistreaming GitHub site.

The setup was quite simple and within just a few minutes I had the script running. It streamed much faster than Motion. I just needed to get this streaming across the web and I would be good to go.

Houston, we have a problem

I hit a little snag at this point. While I could access the stream just fine from the Pi itself, I could not do so from another computer on my local network. I found that I could access the pistreaming page if I connected the Pi to my network over Ethernet. It had to be something with my wifi connection.

Eventually I discovered that my Asus router or the Tomato firmware I run on it was blocking Wireless Multicast Routing, meaning that it was not allowing RTP/RTSP streaming between wifi nodes. Once I enabled that option, I could access the stream from my laptop.

Streaming to the internet

Streaming to the internet involved a couple of changes. First, I had to enable Port Forwarding in my router's admin tool:

Note: Make sure you use the IP addresses of your Pi, not those in my screenshot. You'll also notice that I set a custom port for the stream (8888 rather than the default 8082). Make sure you use the port your copy of pistreaming is set up to use.

According to a closed issue on the pistreaming repo, I needed to modify the script's index.html file. That file includes a hard-coded LAN address for the stream's address. It needs to be modified to use your public IP, which is the one your ISP gives you. You can do a Google search for "what's my IP" to find that address.

Then, update index.html, changing the var client line near the bottom like this:

// change the var client line (at the bottom) to be:
var client = new WebSocket('ws://your_external_ip_address:${WS_PORT}/');

Make sure you have the quotes right on that. There are few quotes in there to start and you end up with two single quotes around everything between the parentheses.

Streaming on startup

I wanted the streaming server to start up automatically when the Pi booted up. To do this, I modified /etc/rc.local to run the server.py script. Keep in mind that scripts run by system processes like this don't have access to the same environment as you would have at a terminal prompt. When auto-running scripts like this, you have to use full (absolute) paths to python and your script.

Here are my steps for this:

First, I found the path to python with which python3
Then, I opened the file in nano (sudo nano /etc/rc.local)
At the end of the file, right before the exit 0 line, I added:
/path/to/python3 /home/pi/pistreaming/server.py &
Note that the & at the end runs the script in the background so that it will keep running once rc.local exits.
Then, Ctrl + o to save and Ctrl + x to exit nano.

I had one more change to make, again related to absolute paths. I had to modify server.py so it could find the index.html and jsmpg.js files when run by rc.local. The relative paths it used didn't work until I updated the class StreamingHttpServer class definition follows:

class StreamingHttpServer(HTTPServer):
    def __init(self):
        # I added this line
        cwd = os.path.dirname(os.path.realpath(__file__))
        super(StreamingHttpServer, self).__init__(
              ('', HTTP_PORT), StreamingHttpHandler)
        # then modified this next line
        with io.open(os.path.join(cwd, 'index.html'), 'r') as f:
            self.index_template = f.read()
        # this one too
        with io.open(os.path.join(cwd, 'jsmpg.js'), 'r') as f:
            self.jsmpg_content = f.read()

And with that, I was done. A quick reboot of the Pi and a check and sure enough it was streaming away.

Lighting the way

The inside of a birdbox is pretty dark, even during the day. From what I could find in the literature, birds see wavelengths between roughly 300 to 700 nm. That would mean a 940 nm IR LED would be outside their vision range but it would be visible to my NOIR (no infrared filter) Pi camera.

I used a 100 ohm resister and wired up the LED to pins 6 (ground) and 12 (Broadcom pin 18). I chose this pin because it's the PWM pin, which means if the LED is too bright, I can dim it by implementing pulse width modulation in a script rather than having to solder in a bigger resister.

Python on a Raspberry Pi comes with the RPi.GPIO library for accessing the pins. I happen to be using Berryconda (the Anaconda distribution for the Pi) which doesn't come with the library. But, it was a simple install with pip install RPi.GPIO. Here's my dead-simple script for turning on the LED.

import RPi.GPIO as GPIO

ledPin = 18  # Pi pin 12 is Broadcom pin 18

GPIO.setmode(GPIO.BCM) # Use Broadcom pin-numbering mode
GPIO.setup(ledPin, GPIO.OUT) # LED pin set as output
GPIO.output(ledPin, GPIO.HIGH)

I saved that as led.py in my home directory. I updated rc.local to also run this script after starting the streaming server.

Focusing the camera

Finally, I had to adjust the focus of the camera. A bluebird box is roughly 9 inches tall and the nests are usually a few inches of packed grass and sticks. By default, the Pi camera is not in focus that close. But, you can change that.

To avoid breaking the camera lens assembly right off the board, you'll want to hold the square frame of the camera lens with one set of pliers. Then, use a pair of needle nose pliers to turn the inner lens to focus. I focused while watching the live stream until it was in focus at roughly 6 or 7 inches away.

Update 1

Sadly, the nest box that had four bluebird eggs is now empty. Something (a snake, raccoon, house sparrow, grackle, etc.) raided the box and took the eggs. I found a partial shell about 30 feet from the box, so I suspect another bird. As sad as it is, nest raiding is unfortunately common.

As luck would have it, we have a nest with eggs in the next box over. We think it is a Carolina Wren nest based on the shape, building materials, and egg coloration. Unfortunately, Carolina Wrens make deep nests such that there's no way to mount the camera inside the box like I'd do with bluebirds.

I went with a rather high-tech housing for the Pi, camera, and power supply. Everything fit inside an empty plastic nut container. I Gorilla-taped it to a metal garden stake. Ugly, but it works.

And aimed it at the house. We can watch the wrens come and go, which is definitely not as cool as seeing inside the box.

Update 2

The damn sparrows got the wrens too. Fortunately, a bluebird pair nested in another nearby box on our property. So far, they're safe and happily watching their eggs. Here's a brief look inside (recorded, not live).

Love them !

Acquiring images with OpenCV

2018-04-16T00:00:00-04:00

To manipulate an image, the first thing you must do is open it. OpenCV lets you open images and videos from files and cameras, both locally attached and on the network. Let's check it out.

Still images

We'll start with the simple case of still images. This is probably best illustrated with some sample code:

import cv2

# read an image from the current folder
image = cv2.imread('cat.jpg')

# or from another folder
image = cv2.imread('images/cat.jpg')

# then we'll just show it in an OpenCV window
cv2.imshow('Photo', image)

In the above example, the image variable is a binary object representing the RGB data of your file. Keep in mind that internally, OpenCV represents these "backwards" as BGR images. That will matter if you use a different library, such as matplotlib, in your script which expects RGB image data.

OpenCV supports the most common image file formats, including JPG, PNG, WebP, TIF, and more.

Errors and exception handling

The imread method does not raise a proper exception in the case the file doesn't exist. So, you can't wrap your code in a try/except block to handle errors. Instead, you'll need to check before calling imread with Python's usual file-handling functions, for example:

from os import path

if path.isfile('cat.jpg'):
    image = cv2.imread('cat.jpg')
    ...

However, just because a file exists and can be opened by imread does not mean that you can successfully read image data from it. In such cases, the function will return None which you should test for:

image = cv2.imread('not_an_image.txt')
if image is None:
print('Could not read image data')

Note: OpenCV determines the file type by its contents, not its extension. So, if you renamed cat.jpg to cat.txt, the imread() function would still be able to read it as a JPG file.

Video files and cameras

OpenCV lets you read video data from files and cameras. What's better, you use the same technique no matter the video source. First, you get a reference, or handle to the video source. Then, you read from that source frame by frame.

Strictly speaking, you use the cv2.VideoCapture.open() method to get a reference to your video source. However, OpenCV offers a shorter equivalent in the cv2.VideoCapture() method.

VideoCapture(source[, options])

Where source is one of:

an integer, representing which locally attached webcam to read from. A value of 0 represents your built-in webcam if available.
a string, representing a path to a local video file
a string, representing the URL to a streaming network camera, such as an IP camera

Typically, you don't need to provide any options for the method. These flags would be used to specify the format of the data stream and other specifics. As with still images, OpenCV can typically figure out the video parameters automatically by examining the input data.

Here is a very typical loop used to read frames from a video source. In this simple example, we just show each frame in an OpenCV window. Within each loop, the script checks for a key-press and if the letter "q" was typed, we break out of the loop, which in this example would release the video source and end the program:

camera = cv2.VideoCapture(0)
while True:
    # loop continuously reading frame-by-frame
    success, frame = camera.read()
    if success:
        # a frame was successfully read
        # we'll just show it in a window
        cv2.imshow("Live Video", frame)

        # if the 'q' key is pressed, stop the loop
        if cv2.waitKey(1) & 0xFF == ord("q"):
            camera.release()
            break
    else:
        # frame wasn't read, handle that problem, for example
        camera.release()
        break

Errors and exceptions

As with reading still images, OpenCV does not offer proper exception handling when reading from a video source. If you specify a source that doesn't exist, you'll get an error like the following

out device of bound (0-0): 1 opencv: camera failed to properly initialize!

(To create the preceding error, I used camera = cv2.VideoCapture(1) and since my laptop has just one camera, camera 1 doesn't exist.)

It's important to release the video source properly when you're done by calling the release() method. Failing to do so can leave your camera unusable. Below is an example of the error you get attempting to use the built-in camera on a Mac that was not released properly before.

OpenCV: error in [AVCaptureDeviceInput initWithDevice:error:]
OpenCV: Cannot Use FaceTime HD Camera (Built-in)
OpenCV: camera failed to properly initialize!

If you end up in this state, you may have to restart your computer to recover. On the Mac, you can use the following command rather than restarting:

sudo killall VDCAssistant

Summary

There you have it, the ins-and-outs of aquiring image data to use with OpenCV.

OpenCV Basics

2018-04-15T00:00:00-04:00

OpenCV is a library of computer vision functions available for C++, Java, Python, and more on Windows, Linux, OS X, Android, and iOS. The OpenCV project began way back in 1999 and has been continually expanded and improved since then.

This article and the others on this site focus on OpenCV-Python. As the OpenCV site puts it, "OpenCV-Python is a library of Python bindings" for OpenCV. What they mean by that is the core functionality of OpenCV is performed by compiled C++ code (so its fast) yet is programmed from Python (which means it's easier to use).

What can you do with OpenCV?

Here's a short list of what's possible with OpenCV:

Read an image from a file, a USB web cam, an IP cam, and more
Convert an image to another color space: RGB, HSV, LAB, YCrCb, grayscale, etc.
Blur or sharpen an image
Add noise to or remove noise from (denoise) an image
Resize, rotate, warp, and adjust perspective of an image
Extract portions of an image
Find the edges of objects in the image
Get the outlines (contours) of objects in the image and find the dimensions, area, and more about those contours

Of course, you can do all the above operations on a video source too, with both files and video streams.

Installing OpenCV

For Python environments, installing OpenCV is pretty simple. You'll need to choose between installing just OpenCV or the contributed modules as well. The "contrib" modules are extra functions provided by the community. Popular and well-written contrib modules sometimes get pulled into the core OpenCV distribution. However, not all of them are error free or fully optimized.

To install the core OpenCV Python version:

pip install opencv-python

To install the version containing the contrib modules:

pip install opencv-contrib-python

(The above commands will work just fine if you're using the Anaconda Python distribution, which I recommend you use.)

Mac users can also install OpenCV using Homebrew and Linux users can sometimes use their package manager (e.g. apt-get). You can even compile OpenCV from its source code version. You would do this to get the newest version or to get a version tailored to your specific computer.

Learning to use OpenCV

The OpenCV docs are not particularly beginner-friendly. Fortunately, there are many great resources for learning OpenCV. A few sites I recommend you check out are:

And of course, check back here too. In future articles on this site, I will cover how you can use OpenCV to perform some interesting and fun operations on images and video.

Build OpenCV and install from source

2018-03-27T00:00:00-04:00

(Originally posted at skypanther.com)

I wanted to set up the latest OpenCV version on my Mac. I found various instructions, but few that applied specifically to my setup — Mac OS X High Sierra running Anaconda. What follows are my steps to compile and install the latest OpenCV version onto my Mac.

Prerequisites

You’ll need a few things first…if you don’t have these, set them up now.

Xcode and its command-line tools
A working Python 3.x environment (hint: use Anaconda)
Homebrew

Once you have those installed, you’re ready to start. You’ll need to install some pre-requisite packages. Each of the following is a separate command:

brew install cmake pkg-config
brew install jpeg libpng libtiff openexr
brew install eigen tbb

Next, you’ll need the OpenCV source code. Clone OpenCV and OpenCV_contrib:

cd ~
git clone https://github.com/opencv/opencv
git clone https://github.com/opencv/opencv_contrib
cd ~/opencv
mkdir build
cd build

Next you’ll configure make with the following command. All of this is one big command, split over multiple lines, and yes it does end with two periods. You can just copy it all and paste in your terminal.

cmake -D CMAKE_BUILD_TYPE=RELEASE \
    -D CMAKE_INSTALL_PREFIX=$(python3 -c "import sys; print(sys.prefix)") \
    -D OPENCV_EXTRA_MODULES_PATH=~/opencv_contrib/modules \
    -D PYTHON3_INCLUDE_DIR=$(python3 -c "from distutils.sysconfig import get_python_inc; print(get_python_inc())") \
    -D PYTHON_PACKAGES_PATH=$(python3 -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())") \
    -D PYTHON3_EXECUTABLE=$(which python3) \
    -D BUILD_opencv_python2=OFF \
    -D BUILD_opencv_python3=ON \
    -D INSTALL_PYTHON_EXAMPLES=OFF \
    -D INSTALL_C_EXAMPLES=OFF \
    -D BUILD_TESTS=OFF \
    -D BUILD_PERF_TESTS=OFF \
    -D WITH_CUDA=OFF \
    -D WITH_FFMPEG=ON \
    -D ENABLE_PRECOMPILED_HEADERS=OFF \
    -D BUILD_EXAMPLES=OFF ..

That command will take a while...after which you can do the actual build:

make -j4

That will take even longer; depending on the speed of your system it could take an hour or longer. Once it’s all done, you’re ready to actually install the compiled project:

sudo make install

That will copy all the files to the right spot on your system. At this point, you’re pretty much done. I found a few guides that suggested that you need to rename (or copy or link) one of the .so files that was copied to your python lib folder.

Rather than digging for the location of that folder, what the first command below does is ask python to find that folder for you and pass it to the cd command:

cd $(python3 -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())")
cp cv2.cpython-36m-darwin.so cv2.so

Finally, you’re done and ready test your work:

$ python

Python 3.6.3 |Anaconda, Inc.| (default, Oct  6 2017, 12:04:38) 
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> cv2.__version__
'3.4.1-dev'

That’s it, OpenCV is installed!