Cheerlights on a Raspberry Pi Pico using MQTT-SN over CAN bus

At first glance, an innocent little project for the Festive Season - Cheerlights on a Pico. Big Deal?!

But look round the other end…

Oooh! That looks rather like a CAN bus interface!

The title of this blog is rather a lot to swallow in one go - you probably recognise most of the words in it, but have you ever seen them all together in the same sentence? No.

Let’s break it down, starting with the easy parts.

Cheerlights

Cheerlights is a project started by Hans Scharler in 2011, to “spread a bit of cheer, through lights”. You tweet @cheerlights then a colour (e.g. red) and all the Cheerlights devices in the world turn red! Simple but very effective.

Needless to say, there is an MQTT feed for Cheerlights (as well as an HTTP one, if you insist), at mqtt.cheerlights.com. Topic hex gives you the hex web colour version of the current Cheerlights colour (e.g. #ff0000 for red). This makes it really easy to create a Cheerlights device from an Arduino-type device, connected to the Internet (e.g. an ESP8266) and a Neopixel RGB LED.

Raspberry-Pi Pico

The Pico is a low-cost microcontroller device using the RP2040 CPU chip designed by Raspberry Pi. It’s quite a lot like other Arduino-like devices, in that it has no operating system and you build your application to run as the firmware on the device. This makes it excellent for IoT applications, and amusingly is sold on reels, so is the first computer you can buy by the metre!

The Pi Pico W adds 802.11 WiFi, and thus is similar to ESP8266 / ESP32 type devices. The Pico and Pico W have 2MB of flash ROM, and 264K of RAM. This gives enough room (amazingly) to run MicroPython, and leaves plenty of space for all your program files and libraries. There are libraries for all the usual things you need - MQTT, Neopixels… um.. well, what else is there in life?! Grudgingly, and against my long-stated intentions, I have had to learn Python for this project.

MQTT-SN

MQTT-SN was designed (I was the co-inventor) a long time ago to be an MQTT-like messaging protocol for networks and devices that didn’t support TCP/IP. Hard to imagine there are such things when for so little money you can get something as powerful as an ESP8266 or a Pico W, but there are: Zigbee, LoRa, NB-IoT, and CAN bus. MQTT-SN only needs a UDP-like layer to run on; that is, if packets arrive, they arrive intact. The rest of the magic is handled by the protocol. MQTT-SN is designed to be semantically compatible with MQTT, making it easy to bridge from an MQTT-SN network (non-TCP/IP) to an MQTT network (TCP/IP) through a gateway device. The MQTT-SN specification (PDF) explains the function of the gateway, and the interactions a client device has with it in order to publish and subscribe upstream to the MQTT network. MQTT-SN is currently in the process of being standardised by OASIS. The Technical Committee is very ably led by Ian Craggs and Simon Johnson. Simon has collaborated with me on this project.

Two of the most interesting features of MQTT-SN, compared to MQTT, apart from its intense frugality of bytes on the wire/air, are:

• There is an additional Quality of Service (QoS) for publishing messages. It’s “even less than QoS 0”, and is therefore known as QoS -1. This was originally designed for very primitive sensor devices which only have a transmit capability and no receiver, so they are unable to establish a session with the gateway. QoS -1 enables “drive-by publishing” where you throw a message towards a gateway in the hope that it might be received, but with no way of finding out if it was, or not.
• To keep messages short, topics are either 2 bytes long, or, use a 16-bit identifier, which is either pre-agreed with the gateway, or is registered before use. If I want to publish to Andy/greenhouse/temperature, I register that topic with the gateway. It returns a topic identifier (e.g. 3) and from then on I publish to topic 3, and the gateway translates that into its original long-form topic before sending it out over MQTT. And the same for subscriptions. This all helps keep the messages very small.

CAN bus

You’ve probably heard of CAN bus (Controller Area Network) - it’s the networking technology used in cars. It runs over a twisted-pair of wires, and is the way all the hundreds of sensors, actuators and computers in your (modern) car talk to each other. Because things like ABS, airbags, Engine Control Units (ECUs) and other really important things use it, it’s well locked-down, all the messages are well-defined for a particular make/model of vehicle, and it’s all fast, low latency, and synchronous.

It’s hard to draw analogies between CAN bus and other networking technologies you might know more about. It’s a multi-drop serial protocol, uses an “Arbitration ID” which combines what we might think of as a “topic” with the priority, and each “frame” has up to 8 bytes of data payload.

The most exciting thing (for me, at least) is that CAN bus is inherently a multicast publish/subscribe network. Each message “published” onto the bus is (effectively) simultaneously received by all the other devices on the bus, and they can filter out the types of messages they are interested in by using “filters” in the CAN controller chip so they only see the messages that are relevant for them. Rather like subscriptions in the MQTT world.

The other interesting twist is that the marine instrumentation networking standard NMEA-2000 is, in fact, under the covers, CAN bus! Also larger vehicles like trucks use a standard called J1939, which is a set of pre-defined message types running on CAN bus. So there’s a lot of it about.

I am fortunate to count Ken Tindell amongst my friends - he is a noted expert in the CAN bus world, and has patiently introduced me to the CAN world, advised me of things to avoid, suggested things I should do, pointed out little secret bits dotted around the frames, that I have been able to use to convey extra information. Ken is the CTO of Canis Automotive Labs specialising in security products for the automotive industry. My gratitude to Ken, for all his help on this project, is enormous!

Bringing together the Pi Pico and CAN bus, Canis Labs has produced a CAN bus interface board with a Pi Pico or Pico W as the microcontroller board, called the CANPico. Using their CANPico MicroPython library, one can write CAN bus applications in Python.

MQTT-SN over CAN bus

I had been thinking for a while about whether we could somehow marry the pub-sub-ness of CAN bus with that of MQTT. I mentioned it as a vague idea in passing to my buddy Rob Poor. He replied:

Whoa - YES! If someone in your ambit could make MQTT a core part of the CAN bus standard, that would be a Very Useful (And Important) contribution. Seriously.

That, coupled with some discussions with the engineers at MarineAI, who are the awesome folks who develop the control systems and the AI Captain technology for autonomous vessels like Mayflower Autonomous Ship which I have been privileged to work on for the past three years. With MQTT as the internal “enterprise messaging system” inside Mayflower, it makes a lot of sense to think about “end-to-end MQTT” all the way from sensors, through the brain of the ship, out to actuators.

There were some important design considerations, though, and it took a while for all the thoughts to “gel” in my mind.

Ken guided me through the minefield of Arbitration IDs: we ideally want a protocol which can live alongside other CAN bus traffic on an existing network. We decided on a format which is “J1939 compatible”. That is, if all the devices on your CAN bus are talking J1939-compliant messages, then MQTT-SN/CAN will not cause conflicts.

Topics and Arbitration IDs

It was tempting to recreate the MQTT-SN semantics of connecting to a gateway node and then directing all publishes to the gateway, and receiving all your messages from up-stream MQTT subscriptions, directed by the gateway to the nodes on the CAN bus which had subscribed to them.

But I wanted to capitalise on the inherent pub/sub nature of CAN bus and enable clients to establish a form of subscription by using filters on the Arbitration IDs. Another “oddity” (feature?) of CAN bus is that two stations can’t simultaneously transmit different frames with the same Arbitration ID. This would effectively mean that two clients couldn’t simultaneously publish to the same topic with different payloads. This meant I had to include the “station ID” in the Arbitration ID, and each station on the CAN bus must have a different Station ID. This is rather like the MQTT requirement that Client IDs must be unique on any given broker.

This gave me a schema for the way MQTT-SN/CAN would use the 29-bit (extended) Arbitration ID space of the CAN bus frame. As mentioned, the nice thing about this approach is that you can do local MQTT-SN pub/sub on the CAN bus. One station publishes at QoS -1 (remember: drive-by publishing mode), and other stations can subscribe to local publications using a connectionless subscribe function - CANsub(topic) - which creates an appropriate filter which is deployed to the CAN controller on the board. Thereafter, the client receives messages that were published on the requested topic.

A wildcard subscribe is also valid here: CANsub() creates a filter which puts “don’t care” into the topic bytes of the Arbitration ID, whilst matching the bits that identify it as MQTT-SN/CAN.

Multi-frame messages

Although frugal in its use of bytes on the wire, MQTT-SN has several bytes of header information before we get to the data payload. For a QoS -1 publish, after I’d squeezed what I could into the Arbitration ID, there were still 3 bytes that had to go into the CAN frame data field, which remember is only 8 bytes long. That left us with 5 bytes for our payload. Just enough room to send “Andy!” as my very first MQTT-SN/CAN message.

But the novelty of that soon wore off and I realised I needed a way to segment MQTT-SN messages so they could span multiple CAN frames. Ken proposed a scheme using a “stolen” 2 bits in the “DLC” field of the frame, involving a “start message” and an “end message” bit. This rather elegant solution proved easy to implement, and I was soon sending “Peter Piper picked a peck of pickled peppers” messages around the place!

Gateway

Although the semantics of MQTT-SN are designed to make bridging to and from MQTT quite easy, the role of the gateway is actually quite complicated, as it has to maintain a fair bit of state on behalf of the CAN-side clients. I didn’t really fancy implementing all that up front, preferring my approach of incremental development adding functionality. I already had MQTT-SN pub/sub working on the CAN bus side, and all I really wanted was a way to bridge those messages bidirectionally to MQTT, effectively bridging the two networks together.

There are several great articles around on implementing an MQTT client on the Pico W. This one, by the awesome @biglesp gave me what I wanted. It uses the simple but elegant umqtt.simple MQTT library for MicroPython.

I glued my CANsub() local subscriber to an MQTT publisher on the Pico W and soon had an MQTT-SN/CAN to MQTT gateway forwarding messages published from the CAN bus up to a broker. Amazing!

To get the other way working, I had to add the MQTT-SN/CAN publisher into the mix. After a bit of head-scratching I worked out how that was going to work, using an MQTT subscription to a specified topic or topic tree on the remote broker, then mapping topics back into pre-defined topic IDs, or two character short topic names before publishing them onto the CAN bus. A client wishing to receive any of those messages could use CANsub() to subscribe locally to messages on the CAN bus. Yay!

Cheerlights on a Pi Pico using MQTT-SN over CAN bus

And so, finally, we get to the point of this blog: on the Pico W we have an MQTT connection to mqtt.cheerlights.com, a subscription to the hex topic, mapping that back to a topic ID (8, as it happens), and publishing it onto the CAN bus as an MQTT-SN message.

Another node on the CAN bus uses CANsub(8) to subscribe locally to the topic ID assigned to the hex topic, receives the #ff0000 type messages, converts to R, G, B 0-255 values and uses this Neopixel library to turn the Cheerlight that colour!

@cheerlights red

Of course, there’s more than one interface to Cheerlights, but they all end up as MQTT messages going to Cheerlights devices, so we could ask Alexa to change the colour…

This explains the lengthy path that a command to my Amazon Echo takes in order to turn my CAN-connected Cheerlight device red. My voice is processed in Amazon’s cloud, recognised as a command to the Cheerlights “skill”, which sends a tweet to Twitter (@cheerlights red). The tweet is picked up by the Cheerlights infrastructure, parsed for colour names, converted to hex colour values and published to the Cheerlights MQTT broker. The Pico W subscribes to the hex topic on the Cheerlights broker, converts the MQTT message to MQTT-SN and sends it out over CAN bus to local subscribers, one of which is my Cheerlights display device (which amply explains why it takes 6 seconds!).

So What?

I’m really excited about the possibilities for this technology. Being able to use MQTT-style pub/sub messaging from a sensor on a CAN bus, to local subscribers, and through a gateway to an MQTT broker infrastructure presents all sorts of possibilities. And the reverse route also means we could do CAN-to-CAN messaging across the Internet.

Clearly security is a big concern here, so it’s very good that I’m working closely with Ken Tindell, who reminds me that I need to pay attention to the security implications at every opportunity. I have already designed-in an accommodation for his (ingenious) encrypted CAN technology.

I believe MQTT-SN/CAN is compatible with NMEA-2000, as that is based on J1939, and we’re compatible with that, so that opens up opportunities in the marine sector.

What’s next?

I want to explore the client-gateway lifecycle next. I have a cunning plan to use reserved topic $$ for messages to the gateway, and $X for messages from the gateway to station X (e.g. $4 from gateway to station 4). I haven’t thought this all the way through yet, nor written any code (as you can probably tell), so I reserve the right to change my mind. But I think it gives a nice “service-based” access scheme for the gateway administration functions. I’ll report back when I’ve tried it!

Thanks

Vast quantities of thanks to the awesome Ken Tindell for his patient explanations, fascinating discussions, lively debates, the CANPico boards, the CANPico libraries, and inspiring this whole project, really! And if you have a need for CAN consulting - Ken and Canis Automotive Labs are where you should go.

Thanks also to Simon Johnson, co-chair of the MQTT-SN standardisation Technical Committee for instantly getting why anyone would want to do such a thing, and providing a rather wizzy MQTT-SN gateway instance at HiveMQ.

Oh, and thanks to Hans Scharler for inventing Cheerlights, and Eben Upton for Raspberry Pi :)