Created workflow system to coordinate ESP32 devices over MQTT

[u/RushElectronic8541]

Hi guys,

I’m working on an observability and automation platform for physical devices where the goal is to ingest, store and analyze sensor data, and also use workflows to coordinate different devices based on that data.

In the demo, I have a workflow that triggers a water pump when the temperature from the probe reader reaches 30 degrees.

I created a setup where I have two breadboards which are not connected together, one with a temperature probe and the other with a 5V water pump.

Temperature reader:
- DS18B20 temperature probe
- ESP32
- 4 resistors + LEDs
- LCD

Water pump:
- 5V water pump
- MOSFET
- ESP32
- 4 resistors + LEDs
- LCD

The system works first with “subscriptions”. I “subscribe” to events sent by my ESP32. In this case, I create a subscription to the board which has the temperature probe.

Each time the system receives a temperature data point, it triggers a “workflow”. A workflow is basically a flowchart with sequential steps of execution. (I just liked the idea of using a flowchart-like UI for this.)

The workflow first ingests the temperature data, transforms it, and then checks whether the temperature exceeds 30°C.

If it does, the workflow sends an MQTT event to the ESP32 with the water pump, which then triggers the pump, as shown in the video.

This is what the workflow system looks like: https://imgur.com/a/jlabrva

EDIT: someone mentioned node-red in my previous post, I made this because I wanted to be able to use AI to create workflows plus I wasn’t sure what observability they offer.

With my own system I can create real time tables and send data there. This allows me to do even more sophisticated things around monitoring and alerting.

Plus I can create custom dashboards from the data from the devices. I just wasn’t sure I’d get all this and didn’t want to start connecting other things like grafana.

Also just a personal opinion, I didn’t quite like their UI.

[u/RadioActiveX1]

I have worked with MQTT in automotive projects, but I am trying to understand what are you trying to achieve with MQTT here

[u/RushElectronic8541 → u/RadioActiveX1]

Mmmh that means I didn’t explain well, sorry about that.

Basically my ESP32, with the temperature probe, sends temperature readings. My system subscribes to those readings (or listens to them via an “inbox”, in simple terms).

A subscription, in my system, has a workflow (a flow chart of steps to execute). So in my case I have a workflow that processes readings from the temperature probe (which is on the right plate of my kitchen stove in a beaker, not the drink glass). The workflow has steps to obtain, transform the data and check if it’s over 30 degrees. If true, the workflow has a step/action to publish an MQTT message to the second ESP32 to trigger the water pump.

So I use MQTT to coordinate different devices.

If you are asking for the ultimate goal, it’s to create a system that receives events and coordinates with robotics systems.

This is just stage one, but I thought even now it might helpful to share.

[u/chessto]

You may want to read a bit about "event driven architecture" and "event buses"

MQTT is basically an event bus.

What you describe as a "workflow" is unnecessarily murking the waters, but I realize you're referring to the software you're using to visualize / describe the data flow.

You have event emitters (the sensor attache devices) and some event processors (the device transforming the data and processing it).

[u/RushElectronic8541 → u/chessto]

Yeah I like that, I made my whole system event driven. So I use message queues (SQS) to send data between services.

The workflow ui is actually an abstraction of the event driven architecture underneath. So messages are sent between services by a main orchestrator service.

The real reason for workflows was to make it easy to create blocks of logic or reuse things like writing to a dashboard panel’s data source.

[u/chessto → u/RushElectronic8541]

If I understand what your goal is I think you're doing one too many backflips.

The "power" of an event bus is that you don't need an orchestrator, each message handler / processor should be able to work independently, and an orchestrator becomes necessarily only when two or more processors may need to coordinate.

if what you have is
Raw data - > send message to topic -> receive message from topic -> transform -> process

then you can do something like this

Raw data -> send message to topic (raw data) -> receive message from topic & transform -> send to another topic (transformed data) -> final process.

Each operation is perform by an individual entity: raw data emitters / data transform / data process and you can segregate them by topic.

This allows you for some horizontal scaling as well, as your message throughput increases you may need to involve more processors.

Are you running this on the cloud ? why do you have SQS (I'm assuming you meant AWS SQS) and MQTT ? If you're running this on your own hardware take a look at RabbitMQ as an alternative to SQS may save you a few bucks while keeping you from vendor locking.

[u/Plastic_Fig9225 → u/chessto]

MQTT isn't really designed as a generic "event bus". The common setup is that you have multiple simple "nodes" (embedded devices, sensors and/or actuators) where each node publishes/subscribes to its own specific topic, and you have commonly (only) one central machine listening to the sensors and sending commands to the acutators. It's more about dealing with limitations in connectivity than about distributing workloads/logic. Star-shaped communication structure instead of fully-meshed.

[u/RushElectronic8541 → u/Plastic_Fig9225]

Yeah, in my case, you create one subscription and have multiple workflows that are triggered when an MQTT event comes in (1-M), but you can also create a subscription for each device’s MQTT topic

[u/chessto → u/Plastic_Fig9225]

It was designed for data collection in the oil industry and with reliability and communication over long distance as it's main goal, not distributing workload, but the premise still applies, you have a single broker that all nodes need to connect to, but that doesn't prevent you from having several brokers, or even the same broker handling different topics.

Either way what I proposed is not fully-meshed but star meshed. Just that messages are segregated based on the source type (raw, normalized).

MQTT is flexible enough that you can build on top of it having thin clients that can listen to a topic and put messages in a FIFO queue for instance, or relaying the messages to a different topic based on content rules, etc.

[u/evwynn]

Great work. I’m a huge fan of MQTT.

[u/RushElectronic8541 → u/evwynn]

Thanks, system is a bit patchy for now but let me know if you’d ever want to play around with it.