Designing embedded systems with web frameworks

I have a prediction.

We're about to see a shift in embedded systems development. Around 2008 or 2009, embedded microprocessors like the one in your cellphone, reached a threshold where they can perform as decent webservers without special tuning. Even with a slow database query or some inefficient templates, they've got the speed to handle real web serving. This means that suddenly the most efficient development pattern for embedded systems is not the proprietary hardware and software tools that have dominated the industry for the last 30 years, but the open source web development tools that have arisen in the last decade.

The transition will be painfully slow, and of course there are some niches where specialized hardware and real-time control will preclude the use of generic web tools-- motion control springs to mind. But I think that the combination of cheap hardware and modern web frameworks will crush the industrial controller market, like digital cameras did to film cameras.

First, some background on tiny computers

There are millions of tiny computers used for monitoring and control systems around the world. Let's break them into two categories: small (microcontrollers like the Arduino or a PIC development board, which runs $10-500) and large (embedded controllers like National Instruments hardware, which cost $500-5000).

To use the small ones, you write code from scratch, perhaps with some pre-written libraries to talk to certain peripherals and a bootloader to run your code on power-up. The vast majority cannot be connected to the internet without substantial effort, and when connected to the internet, they aren't powerful enough to work like most servers on the internet. For example, a webserver built on a small microcontroller would be overwhelmed by the background noise on the internet, i.e. traffic from bots and viruses. This kind of system is perfect if you want to log temperature in your basement, or turn on a light whenever the garage door is open. They're no good for running an inventory management system in a warehouse with 20 workers.

Below: an Arduino

The large ones come with an operating system, like Windows CE, Linux, or VxWorks. Most of the devices are reworked versions of hardware from the pre-internet days that have had Ethernet ports added to them. They can handle real internet traffic, but they usually use proprietary software to do it. They're shaped like a long, skinny shoebox.

Below: a larger controller

The change is that something equivalent to the iPhone, which uses a 600 MHz ARM processor, can store years of data and serve it up to anyone with a web browser in seconds with hardware cost of a few hundred dollars. Industrial controllers lose on cost of both hardware and programming time, and performance. The hardware has to be expensive to support the R&D costs, which are borne entirely by each manufacturer. There's just no way that even a large industrial control company, which might have a dozen dedicated programmers at best, can compete with the thousands of developers working on open source web software worldwide.

How web software development has changed

In the 90's, software development for the internet meant either writing server software or designing static web pages. Starting around 2000 (give or take a few years), websites started incorporating dynamic data, which was stored in databases. Around 2005, a new kind of web software gained popularity-- the web framework-- with the release of Ruby on Rails.

With old-style web development, a web programmer would write code that inserted data into a database, more code that updated the database with new data, and more code that retrieved and sorted the data for presentation in a web page. With a web framework, the programmer writes out a template for how the data should be presented on a webpage, and the framework figures out what to request from the database. Web frameworks can't scale to the level of a big website like Amazon, but for low traffic systems, they work fine and reduce the programming time needed dramatically.

Most of the time, what people want to do with microcontrollers is log some data from sensors and maybe trigger some actuators in response. After they log the data, people want to analyze the data, make graphs with it, and then do it again, maybe with a different sensor. This matches well with the typical database-backed website. The only substantial additions are code to interact with hardware-- read sensors and trigger actuators. I think this is the least developed part of the systems I expect we'll see in the next few years.

Proof of concept

It's definitely possible that I'm just some blowhard on the internet. I mean, I'm at least some blowhard on the internet, but I might still be right in predicting this change. To that end, I've done some testing to see whether I'm going in the right direction.

Using an off-the-shelf microcontroller kit that cost $339 plus shipping, I installed a Python web framework called Django and wrote code to make it act like a thermostat to replace the one in our house. It took about 2.5 weekends to write the code, which is much faster than I could write a similar application for something like an Arduino with an Ethernet carrier board, and this was my first try. I had played around with Django a few times previously, but this was my largest effort by far.

Below: the proof of concept

The hardware interface is just a cron job that runs once per minute. It queries a temperature sensor on the PCB using a short C program called by a Python script, which also logs the data to a SQLite database. Retrieval of a webpage that queries the database for the last 300 datapoints (5 hours worth) and generates a chart of the data using Javascript takes around 1.5 seconds. That's with a processor running at 250 MHz (relatively slow) and the database stored on an SD card. Most of the time is spent converting Python datetimes to Javascript timestamps.

I could have written a similar application even faster using a GUI tool like LabView, but that requires specialized hardware that cost 3-10 times more-- either a dedicated PC with a USB device for sensors, or a industrial controller with a sensor module. With Django, I got an administrative interface, allowing different users and groups different levels of access, for free. If I want a central repository of users with LabView, I'm looking at another $800 for the "Datalogging and Supervisory Control Module." If I want to talk to a SQL database, I'll need the "Database Connectivity Toolkit" for another $1000. This is on top of the $1500 I would have paid for LabView already, plus the hardware.

Embedded control systems running web services are still an immature technology at best, but I think they'll grow up quickly in the next few years.

Why are you writing this?

I've been thinking about this change for a year or two now. I'm working on developing something like the Arduino, but a little heavier duty, so it can run a web framework, but with the hardware drivers pre-integrated. Send me an email at brandon at pingswept.org or post a comment if you want to know when it exists for real.

Picking a Cortex A8 processor for an embedded Linux board in 2010

ARM has several different processor families in production at present. The newest releases are the Cortex processors, which come in three series-- A, R, and M. (See what they did there?) The M processors are the weakest and cheapest, below $10 even in low quantities. The R processors, intended for real-time applications like anti-lock brakes in cars, are the next step up. The top of the heap are the Cortex A series. So far, the A8 and A9 have been released, and the A5 is scheduled for release in 2011. The A8 is the processor at the heart of some recent smartphones, like the Iphone 3GS, the Nexus One and the Droid, for example.

Of the A8 and A9, the A8 is the cheaper, slower one, running in the 500 MHz to 1 GHz range; the A9 can have multiple cores and run up to 2 GHz. To me, the A8 crosses the threshold where we can build embedded systems that connect to the internet, have decent performance without requiring tuning to make applications run fast, and have a price in the $100-200 range. There are certainly many processors that can handle internet traffic in that price zone, like every decent consumer-grade Ethernet switch built in the last 10 years, but those are machines optimized for niche tasks. What's new is that we're finally getting enough clearance above the minimum requirements that cheap, general-purpose systems can function on the internet.

So if you want to make an embedded device that uses a Cortex A8, what chips can you buy? ARM is an unusual company in that they don't produce chips themselves; they license their designs to manufacturers in return for royalties on each chip sold. Right now, ARM lists 7 public licensees of the A8 design; in addition to those listed, Alchip and Ziilabs are claiming to have licensed the design. Several of the licensees, such as Ziilabs and Broadcom, are targeting niche multimedia applications and will likely only sell to large manufacturers of stuff like DVD players and video cameras. Of the remaining companies, two have released general-purpose A8 processors: Texas Instruments and Freescale.

Freescale has released the i.MX5x series, with two subfamilies: i.MX50 and i.MX51. They cost $30-40 in low quantities but the sole distributor listed (Avnet) reports a lead time of 26 weeks for all parts.

TI has two series of A8 processors: OMAP3 and the not-quite-released-yet Sitara AM35xx series. The OMAP3 series has been around since 2008, and there are several embedded Linux boards (Gumstix Overo, Beagleboard based around the OMAP35xx series, though none have a Ethernet port in their stock configuration. The first Sitara, probably the AM3517, will likely be the cheapest of the lot, but assuming we want to limit ourselves to chips that we can actually order, that leaves us with four choices (before we consider packaging): OMAP3503, OMAP3515, OMAP3525, and OMAP3530.

The two higher-end OMAP35xx chips, the OMAP3525 and OMAP3530, include a TMS320 DSP onboard, which is not worth the cost in a general purpose tool. This leaves us with the OMAP3503 or OMAP3515. The major difference between the two is the PowerVR SGX graphics accelerator in the OMAP3515, which, like the DSP, isn't worth the cost. Until the AM3517 or AM3505 hit the distributors, I think the OMAP3503 is the winner. We'd have to choose between the three different packages the chip comes in, but Digikey only has the CBB package (a 515-pin ball-grid array, distinguished from the CBC package by the pin spacing of 0.40 mm rather than 0.50 mm).

In the words of Captain Stillman, "Load and fire the weapon, soldier!"

Buildroot on the Gumstix Verdex XL6P

Hello internet,

Should you attempt to compile the Buildroot toolchain so you can build a new filesystem image for your Gumstix Verdex board, you will find that as of mid-2009, files.gumstix.com no longer hosts several of the files that you will need. However, you can find them elsewhere on the web.

A handy shortcut is to replace the files.gumstix.com URL in gumstix-buildroot/toolchain/getter_script.sh with a different URL where most of the files appear: www.daimi.au.dk/~spider/gumstix/gumstix-buildroot/dl/. Thanks are due to the wise and honorable Martin Mogensen at the University of Aarhus for hosting the files. You'll also need this file in your gumstix-buildroot/dl folder: xmlsoft.org/sources/old/libxml2-sources-2.6.29.tar.gz

Lastly, you'll want to add to the file gumstix-buildroot/build_arm_nofpu/linux-2.6.21gum/scripts/mod/sumversion.c the line:

#include <limits.h>

or you'll get the error "error: ‘PATH_MAX’ undeclared (first use in this function)."

Fixed version of gumstix-buildroot/toolchain/getter_script.sh:

#!/bin/bash
wget -nd --passive-ftp $@ || (
echo Retrying from gumstix alternate site...
index=$#-1
# Copy all params into an array
for (( i=0; $?==0; i++ ));do a[$i]=$1; shift; done
# Chop all but filename from last param and prepend out URL
a[$index]=${a[index]/*\//http:\/\/www.daimi.au.dk/~spider/gumstix/gumstix-buildroot/dl/}
# Now wget that from out server
wget -nd ${a[@]}
)

Just a few other notes, since I finally got the Ethernet port working with Gumstix SVN r1642:

/etc/network/interfaces

auto lo
iface lo inet loopback

iface usb0 inet dhcp

auto eth0
iface eth0 inet dhcp

/etc/modules, with correct Ethernet driver selected (smc91x, not smc911x)

# MMC support -- comment the next two lines to enable using CF
mmc_block
pxamci

# Compact Flash support -- Must load smc91x or smc911x first!!
smc91x
#smc911x
pcmcia

# Load USB host driver
ohci-hcd

Output of lsmod

Module                  Size  Used by
ipv6                  248416  10
af_packet              16872  2
proc_gpio               9412  0
gumstix_bluetooth       1408  0
ohci_hcd               19620  0
usbcore               113340  2 ohci_hcd
pxa2xx_cs               3336  1
pxa2xx_core            10368  1 pxa2xx_cs
pcmcia                 25064  0
pcmcia_core            30576  2 pxa2xx_core,pcmcia
firmware_class          7520  1 pcmcia
smc91x                 16104  0
mii                     4736  1 smc91x
gumstix_smc91x          2816  1 smc91x
pxamci                  6240  0
mmc_block               6568  0
mmc_core               22100  2 pxamci,mmc_block
unix                   22292  18

Installing Xubuntu on a G3 iMac

I work in a school where we have lots of old iMacs that are barely usable under OS 9.2. The newest browser we can get for OS 9 is Netscape 7.0, which crashes a lot (repeatably on Gmail, for example).

Xubuntu is a derivative of Ubuntu Linux designed for low-end machines. It uses the XFCE desktop. I've just finished installing Xubuntu on a 400 Mhz iMac. It went reasonably smoothly, and now that I know the few tricks listed below, doing it again should be quite easy. The steps:

1. Put the Breezy Badger for PowerPC install disk in the CD drive.

2. Reboot.

3. At the first prompt, type "server" and hit return. This will install everything in normal Ubuntu install except the GNOME desktop.

4. Install the Xubuntu desktop and the GNU display manager using apt-get:

sudo apt-get install xubuntu-desktop gdm

1. To get the graphical interface working, I had to tweak /etc/X11/xorg.conf a little. The first change was to alter the HorizSync and VertRefresh settings. I also switched from the fbdev driver to the ati driver. The snippet below shows the original settings commented out and the new settings added. (Link to full xorg.conf.) > Section "Device" > Identifier "Generic Video Card" > # Driver "fbdev" > Driver "ati" > Option "UseFBDev" "true" > EndSectionSection "Monitor" > Identifier "Generic Monitor" > Option "DPMS" > # HorizSync 28-51 > # VertRefresh 43-60 > HorizSync 60-60 > VertRefresh 43-117 > EndSection

2. The last tweak was to add a printer to CUPS manually. To enable the web administration for CUPS, I added a root password: > sudo -s

passwd

In /etc/cups/cupsd.conf, I changed RunAsUser to No, so that CUPS would run as root, and not switch to run as the user cupsys, as I believe this is what disables the web interface: > RunAsUser No Then restart CUPS: > /etc/init.d/cupsys restart Here is what ended up in /etc/cups/printers.conf: (Yes, I live in a farmhouse, and I work on Sunday nights) > # Printer configuration file for CUPS v1.1.23 > # Written by cupsd on Sun 08 Jan 2006 07:34:40 PM EST

Info Farmhouse DeviceURI socket://192.168.1.131 State Idle Accepting Yes JobSheets none none QuotaPeriod 0 PageLimit 0 KLimit 0

Overall, Xubuntu is working really well-- much better than OS 9.2. It's only got 64 MB of RAM, but Firefox runs surprisingly well. I plan on maxing out the memory when I get the chance.

The Xubuntu people have been planning on releasing a CD version of Xubuntu coincident with the release of the Dapper Drake in April. I found the XFCE file manager, xffm, to be a little squirrely, and I couldn't get it to connect to our file server through Samba; maybe that will work in the next release, or maybe I will have figured out how to configure Samba. The Dapper release of Xubuntu will likely be based on XFCE 4.4, which will allegedly include the first release of Thunar, XFCE's new file manager.

Hmm. If Thunar is good, I might switch to Xubuntu entirely. So far, XFCE seems like a fast version of GNOME to me, and I spend most of my time in Firefox and a terminal window anyway.