A 10 year warranty, thanks to IoT

Use Case Warp 10 Amzair

Amzair is a high end heat pump manufacturer in France, near Brest city. They build devices to last at least 10 years, with stainless steel everywhere, always seeking for repairability to extend their product life. Buying such a heat pump for your home is expensive, but its durability is what makes it worth it!

Amzair Heat Pump

But as any semi industrial system, you need to a good maintenance and sometimes adjust the servo assisting parameters during the lifetime. A long warranty time is only possible with a good system monitoring. One year ago, Amzair started their “Serenity” project: a 10 years warranty for their product.

Amzair, connecting heat pumps

The heat pump electronics is driven by an industrial controller. The requirements could be expressed as a 4-line specifications:

  • Read 500 variables from the controller, with a 1s resolution. Store them locally for 7 days.
  • Extract 60 important metrics and store them in the cloud for at least 2 years.
  • Be able to modify in real time some of the 500 variables.
  • GDPR compliant

Storing time series is Warp 10™’s job. So, Amzair asked us to develop the solution. It means connecting an industrial controller to the most advanced time series database. It means develop hardware. As the SenX team electronic guy, that’s my job!

Here is the challenge:

Challenge accepted.

On one side, a nice HTTPS entry point to push my data.
On the other side, the Carel board provide a opto isolated RS485 modbus link.
Between, all kind of home network.
Around: non reliable internet access, non reliable power supply. Home is not a datacenter.

The hardware must:

  • Connect to RS485
  • Connect to Internet
  • Have a long time support (LTS)
  • Withstand 24V AC power supply
  • Manage its own battery for safe power cycles
  • Withstand 10 years of read write cycles on the NAND memory

Open Hardware

On the embedded computer market, I found only one LTS board which can manage a battery. I mean with a real battery controller, such as your laptop or phone: The PINE64 LTS board. The 24V AC to 5V DC could be done on a custom HAT (Hardware Attached on Top). The RS485 transceiver could also be implemented on the custom HAT.

PINE64 with a custom RS485 HAT + power supply

A few LEDs on the side are always welcome to help the maintenance guys. One for power, one for internet, one for Modbus link.
Note the AA 650mAh Li-ion battery on top.

The storage challenge

If you have spare time, google “sd card failures raspberry”. Yes, SD card is a problem!

When you write one single byte on a SD card, the controller writes data in a new physical block, updates the block lookup table and erase the block. NAND flash has an erase life cycle. Depending on the technology, a block could be written 1000 times (TLC) to 50’000 times (SLC). Here is a nice chart from Transcend:

(source : Transcend website)

A standard SSD will provide a nice report of its lifetime, using SMART technology:

% sudo smartctl -a /dev/nvme0 | grep -A5 -i "available"             
Available Spare:                    100%
Available Spare Threshold:          50%
Percentage Used:                    6%
Data Units Read:                    7 237 952 [3,70 TB]
Data Units Written:                 13 244 390 [6,78 TB]
Host Read Commands:                 130 147 472
Host Write Commands:                228 259 301

My SSD has a few spare blocks (known as over-provisioning) and clearly indicates that I used 6% of the NAND flash memory.

Warp 10™ uses a LevelDB backend. Log rotation means compaction of the database. When you compact, you rewrite lots of file. I played with ext4 commit cycles, with tmpfs, with noatime option, but I quickly face the reality: I cannot know what the SD card controller does exactly behind. I do need SMART for SD card to be sure to comply to the 10 year life time requirement!

Storage solution

There is very few capable SD Card on the market. After tests and discussions with Amzair, we decided to use Transcend Industrial grade SD card. Transcend support team kindly answered my questions and sent me the source code to be able to read the controller data.

These SD Card do have a SMART feedback

Of course, this is not exactly the SMART protocol of a SSD, but the output is crystal clear:

Total Erase Count     0x0000BF26
Average Erase Count   0x00000016
Minimum Erase Count   0x00000000
Maximum Erase Count   0x00000041

It means that the Flash blocks were erased at most 0x41=65 times. The SuperMLC used does support 30000 erase cycles. After one month of test, I did a prudent extrapolation, and my conclusion was that the storage will last more than 20 years.

Warp 10™ inside

The PINE64LTS has 2 GB of RAM. A lot more than what you need to run Warp 10™! I modified the /bin/warp10-standalone.sh to setup the java heap to a reasonable size:

WARP10_HEAP=256m
WARP10_HEAP_MAX=512m

The modbus is managed by a VertX application. Each time a new data block is available, the VertX application call the local Warp 10 instance exec endpoint. A WarpScript™ decode the data, buffer data, and write it locally. The event driven architecture of VertX is perfect for polling/wait tasks.

Warp 10™ internal scheduler launches periodically all the required WarpScripts to analyze failure, check internet status, and push data to our Warp 10™ SAAS instance. There is also a downlink through Warp 10™ SAAS: Amzair can write commands in the SAAS that are only accessible by one heat pump.

Warp 10 ™and VertX happily talk together locally through HTTP APIs. WEBCALL WarpScript function makes this easy and asynchronous.

Security and customers isolation

In the build process, Amzair program a set of unique read/write token on each system. The read token allow the system to read encrypted commands from the database. The write token allow the system to write data on our SAAS instance. I already discuss this point in a previous article: every system is isolated at the database level.

Warp 10™ with a good data model, GDPR ready
Amzair took advantage of the Warp 10 token system

Reverse engineering is possible

Security is done at the cloud level. If a clever customer reverse-engineers its heat pump, he will find two tokens. Great! He can read and erase its data. Isn’t it exactly the right GDPR grants him?

Empower IoT data

In a next step, Amzair will do advanced analytics on data. WarpScript will not restrict them to basic operators… If Amzair engineers want to spot an abnormal ramp up of the compressor output pressure using pattern detection, they can do it. On every systems. On 2 years of history. This is a dream tool for engineers… They can spot problems on a large range of existing systems, improve command-control algorithm, test and provide new services to customers easily.

Generating any kind of dashboard with WarpView is easy: on my desk, there is an Amzair controller + a few temperature sensor. You can see the temperature in the room reaches nearly 30°C in the beginning of February.

50 d @amzair/read_room_temp [ SWAP bucketizer.mean 0 5 m 0 ] BUCKETIZE 0 GET 10.0 /

Conclusion

In Warp 10™ world, Amzair is a “small” data producer (“middle” starts at 100k datapoint/s.). But data rate does not matter, Amzair is the first customer to use the full power of Warp 10™ token system to build a perfectly GDPR compliant system. We are proud to host their customers data!


Share