Amzair is a high end heat pump manufacturer in France, near Brest city. They build devices to last at least 10 years, with stainless steel everywhere, always seeking for repairability to extend their product life. Buying such a heat pump for your home is expensive, but its durability is what makes it worth it!
But as any semi industrial system, you need to a good maintenance and sometimes adjust the servo assisting parameters during the lifetime. A long warranty time is only possible with a good system monitoring. One year ago, Amzair started their "Serenity" project: a 10 years warranty for their product.
Amzair, connecting heat pumps
The heat pump electronics is driven by an industrial controller. The requirements could be expressed as a 4-line specifications:
- Read 500 variables from the controller, with a 1s resolution. Store them locally for 7 days.
- Extract 60 important metrics and store them in the cloud for at least 2 years.
- Be able to modify in real time some of the 500 variables.
- GDPR compliant
Storing time series is Warp 10™'s job. So, Amzair asked us to develop the solution. It means connecting an industrial controller to the most advanced time series database. It means develop hardware. As the SenX team electronic guy, that's my job!
Here is the challenge:
On one side, a nice HTTPS entry point to push my data.
On the other side, the Carel board provide a opto isolated RS485 modbus link.
Between, all kind of home network.
Around: non reliable internet access, non reliable power supply. Home is not a datacenter.
The hardware must:
- Connect to RS485
- Connect to Internet
- Have a long time support (LTS)
- Withstand 24V AC power supply
- Manage its own battery for safe power cycles
- Withstand 10 years of read write cycles on the NAND memory
On the embedded computer market, I found only one LTS board which can manage a battery. I mean with a real battery controller, such as your laptop or phone: The PINE64 LTS board. The 24V AC to 5V DC could be done on a custom HAT (Hardware Attached on Top). The RS485 transceiver could also be implemented on the custom HAT.
A few LEDs on the side are always welcome to help the maintenance guys. One for power, one for internet, one for Modbus link.
Note the AA 650mAh Li-ion battery on top.
The storage challenge
If you have spare time, google "sd card failures raspberry". Yes, SD card is a problem!
When you write one single byte on a SD card, the controller writes data in a new physical block, updates the block lookup table and erase the block. NAND flash has an erase life cycle. Depending on the technology, a block could be written 1000 times (TLC) to 50'000 times (SLC). Here is a nice chart from Transcend:
A standard SSD will provide a nice report of its lifetime, using SMART technology:
% sudo smartctl -a /dev/nvme0 | grep -A5 -i "available" Available Spare: 100% Available Spare Threshold: 50% Percentage Used: 6% Data Units Read: 7 237 952 [3,70 TB] Data Units Written: 13 244 390 [6,78 TB] Host Read Commands: 130 147 472 Host Write Commands: 228 259 301
My SSD has a few spare blocks (known as over-provisioning) and clearly indicates that I used 6% of the NAND flash memory.
Warp 10™ uses a LevelDB backend. Log rotation means compaction of the database. When you compact, you rewrite lots of file. I played with ext4 commit cycles, with tmpfs, with noatime option, but I quickly face the reality: I cannot know what the SD card controller does exactly behind. I do need SMART for SD card to be sure to comply to the 10 year life time requirement!
There is very few capable SD Card on the market. After tests and discussions with Amzair, we decided to use Transcend Industrial grade SD card. Transcend support team kindly answered my questions and sent me the source code to be able to read the controller data.
Of course, this is not exactly the SMART protocol of a SSD, but the output is crystal clear:
Total Erase Count 0x0000BF26 Average Erase Count 0x00000016 Minimum Erase Count 0x00000000 Maximum Erase Count 0x00000041
It means that the Flash blocks were erased at most 0x41=65 times. The SuperMLC used does support 30000 erase cycles. After one month of test, I did a prudent extrapolation, and my conclusion was that the storage will last more than 20 years.
Warp 10™ inside
The PINE64LTS has 2 GB of RAM. A lot more than what you need to run Warp 10™! I modified the
/bin/warp10-standalone.sh to setup the java heap to a reasonable size:
The modbus is managed by a VertX application. Each time a new data block is available, the VertX application call the local Warp 10 instance exec endpoint. A WarpScript™ decode the data, buffer data, and write it locally. The event driven architecture of VertX is perfect for polling/wait tasks.
Warp 10™ internal scheduler launches periodically all the required WarpScripts to analyze failure, check internet status, and push data to our Warp 10™ SAAS instance. There is also a downlink through Warp 10™ SAAS: Amzair can write commands in the SAAS that are only accessible by one heat pump.
Warp 10 ™and VertX happily talk together locally through HTTP APIs. WEBCALL WarpScript function makes this easy and asynchronous.
Security and customers isolation
In the build process, Amzair program a set of unique read/write token on each system. The read token allow the system to read encrypted commands from the database. The write token allow the system to write data on our SAAS instance. I already discuss this point in a previous article: every system is isolated at the database level.
Reverse engineering is possible
Security is done at the cloud level. If a clever customer reverse-engineers its heat pump, he will find two tokens. Great! He can read and erase its data. Isn't it exactly the right GDPR grants him?
Empower IoT data
In a next step, Amzair will do advanced analytics on data. WarpScript will not restrict them to basic operators... If Amzair engineers want to spot an abnormal ramp up of the compressor output pressure using pattern detection, they can do it. On every systems. On 2 years of history. This is a dream tool for engineers... They can spot problems on a large range of existing systems, improve command-control algorithm, test and provide new services to customers easily.
Generating any kind of dashboard with WarpView is easy: on my desk, there is an Amzair controller + a few temperature sensor. You can see the temperature in the room reaches nearly 30°C in the beginning of February.
In Warp 10™ world, Amzair is a "small" data producer ("middle" starts at 100k datapoint/s.). But data rate does not matter, Amzair is the first customer to use the full power of Warp 10™ token system to build a perfectly GDPR compliant system. We are proud to host their customers data!
Ingesting 300k datapoints per second in the Warp 10™ timeseries database on a Raspberry Pi is possible with Warp 10 2.1 new ingestion format. We also benched the PINE64 board!
The token mechanism included in Warp 10 allow to simply generate one token per customer to extract or delete a single customer data safely. Perfect for GDPR without data leak risks.
Electronics engineer, fond of computer science, embedded solution developer.