Amzair: A 10-year warranty, thanks to IoT

Amzair is a high-end heat pump manufacturer in France, near Brest city. They build devices to last at least 10 years, with stainless steel everywhere, always seeking repairability to extend their product life. Buying such a heat pump for your home is expensive, but its durability is what makes it worth it!

But as with any semi-industrial system, you need to good maintenance and sometimes adjust the servo assisting parameters during the lifetime. A long warranty time is only possible with good system monitoring. One year ago, Amzair started their "Serenity" project: a 10 years warranty for their product.

Amzair, connecting heat pumps

The heat pump electronics are driven by an industrial controller. The requirements could be expressed as 4-line specifications:

• Read 500 variables from the controller, with a 1s resolution. Store them locally for 7 days.
• Extract 60 important metrics and store them in the cloud for at least 2 years.
• Be able to modify in real-time some of the 500 variables.
• GDPR compliant

Storing time series is Warp 10's job. So, Amzair asked us to develop the solution. It means connecting an industrial controller to the most advanced time series database. It means developing hardware. As the SenX team electronic guy, that's my job!

Here is the challenge:

On one side, a nice HTTPS entry point to push my data.
On the other side, the Carel board provides an opto-isolated RS485 Modbus link.
Between, all kinds of home networks.
Around: non-reliable internet access, non-reliable power supply. Home is not a data center.

The hardware must:

• Connect to RS485
• Connect to the Internet
• Have long-time support (LTS)
• Withstand 24V AC power supply
• Manage its own battery for safe power cycles
• Withstand 10 years of read-write cycles on the NAND memory

Open Hardware

On the embedded computer market, I found only one LTS board that can manage a battery. I mean with a real battery controller, such as your laptop or phone: The PINE64 LTS board. The 24V AC to 5V DC could be done on a custom HAT (Hardware Attached on Top). The RS485 transceiver could also be implemented on the custom HAT.

A few LEDs on the side are always welcome to help the maintenance guys. One for power, one for internet, one for Modbus link.
Note the AA 650mAh Li-ion battery on top.

The storage challenge

If you have spare time, google "sd card failures raspberry". Yes, SD card is a problem!

When you write one single byte on an SD card, the controller writes data in a new physical block, updates the block lookup table, and erases the block. NAND flash has an erase life cycle. Depending on the technology, a block could be written 1'000 times (TLC) to 50'000 times (SLC). Here is a nice chart from Transcend:

A standard SSD will provide a nice report of its lifetime, using SMART technology:

sudo smartctl -a /dev/nvme0 | grep -A5 -i "available"
(out)Available Spare:                    100%
(out)Available Spare Threshold:          50%
(out)Percentage Used:                    6%
(out)Data Units Read:                    7 237 952 [3,70 TB]
(out)Data Units Written:                 13 244 390 [6,78 TB]
(out)Host Read Commands:                 130 147 472
(out)Host Write Commands:                228 259 301

My SSD has a few spare blocks (known as over-provisioning) and clearly indicates that I used 6% of the NAND flash memory.

Warp 10 uses a LevelDB backend. Log rotation means compaction of the database. When you compact, you rewrite lots of files. I played with ext4 commit cycles, with tmpfs, with noatime option, but I quickly face the reality: I cannot know what the SD card controller does exactly behind. I do need SMART for the SD card to be sure to comply with the 10-year lifetime requirement!

Storage solution

There is very few capable SD Card on the market. After tests and discussions with Amzair, we decided to use the Transcend Industrial grade SD card. The Transcend support team kindly answered my questions and sent me the source code to be able to read the controller data.

Of course, this is not exactly the SMART protocol of a SSD, but the output is crystal clear:

Total Erase Count     0x0000BF26
Average Erase Count   0x00000016
Minimum Erase Count   0x00000000
Maximum Erase Count   0x00000041

It means that the Flash blocks were erased at most 0x41=65 times. The SuperMLC used does support 30000 erase cycles. After one month of testing, I did a prudent extrapolation, and my conclusion was that the storage will last more than 20 years.

Warp 10 inside

The PINE64LTS has 2 GB of RAM. A lot more than what you need to run Warp 10! I modified the /bin/warp10-standalone.sh to set up the java heap to a reasonable size:

WARP10_HEAP=256m
WARP10_HEAP_MAX=512m

The Modbus is managed by a Vert.x application. Each time a new data block is available, the Vert.x application calls the local Warp 10 instance exec endpoint. A WarpScript decodes the data, buffer data, and writes it locally. The event-driven architecture of Vert.x is perfect for polling/wait tasks.

Warp 10 internal scheduler launches periodically all the required WarpScripts to analyze failure, check internet status, and push data to our Warp 10 SaaS instance. There is also a downlink through Warp 10 SaaS: Amzair can write commands in the SaaS that are only accessible by one heat pump.

Warp 10 and Vert.x happily talk together locally through HTTP APIs. WEBCALL WarpScript function makes this easy and asynchronous.

Security and customers isolation

In the build process, Amzair program a set of unique read/write token on each system. The read token allows the system to read encrypted commands from the database. The write token allows the system to write data to our SaaS instance. I already discuss this point in a previous article: every system is isolated at the database level.

Reverse engineering is possible

Security is done at the cloud level. If a clever customer reverse-engineers its heat pump, he will find two tokens. Great! He can read and erase its data. Isn't it exactly the right GDPR grants him?

Empower IoT data

In a next step, Amzair will do advanced analytics on data. WarpScript will not restrict them to basic operators… If Amzair engineers want to spot an abnormal ramp-up of the compressor output pressure using pattern detection, they can do it. On every system. On 2 years of history. This is a dream tool for engineers… They can spot problems in a large range of existing systems, improve the command-control algorithms, test, and provide new services to customers easily.

Generating any kind of dashboard with Discovery is very easy. Please read this article.

Conclusion

In Warp 10 world, Amzair is a "small" data producer ("middle" starts at 100k datapoint/s.). But data rate does not matter, Amzair is the first customer to use the full power of the Warp 10 token system to build a perfectly GDPR-compliant system. We are proud to host their customer's data!

Read more about why and how Amzair uses Warp 10 ☟

Pierre Papin

Electronics engineer, fond of computer science, embedded solution developer.