Introduction
The SiteController is actually intended to send acquired sensor data and other information to cloud as fast as possible. But if an upload of the data is not possible or not intended, for instance by the configuration of the SiteController, the collected sensor data are stored locally until a connection to the cloud is established. The USB data export procedure is supported by the visualization using one LED at the azeti-machina NEXCOM NIFE103.
Neither a keypad nor a monitor are required.
SiteController is regularly uploading data to the azeti Cloud if it has an active cloud connection. You'll only see data if there was a period of offline time.
This document describes how to configure the SiteController to force the local storage of data and how to retrieve the collected data offline to a prepared USB flash drive. Furthermore is added, how these data on the USB flash drive could be used.
Prerequisites
Configuration of the SiteController
SiteController.cfg
The following code block example shows settings in the SiteController configuration file, which is located in /opt/azeti/SiteController/config/SiteController.cfg There are also other entries in the configuration file, but they are not related to the topic which is described in this document.
The comment lines, starting by #, explain the settings.
[ExternalBroker] # left empty to guarantee the offline storage host= port=1883 # not required with no connection user_id= password= # an organizationShortName is still required organizationShortName=my_org tls_enable=False [data_store] # this password must be equal to the password at the USB flash drive L2_export_password=P@s$w0rd! # required for the USB data export export_status_representation=sensor export_status_sensor_id=usb_export_state # -1 means no limitation by the table based size limits L2_max_imgdata=-1 L2_max_sys_msg=-1 L2_max_events=-1 L2_max_HD=-1 # limitations to the offline storage L2_max_age_days=183 L2_max_total_size_MiB=1024 L2_time_of_day_to_cleanup=02:00 # Optimization to have permanently more data in L2. # !!! Following 2 lines should be deleted if the SiteController is going to operate online !!! max_items_L1 = 500 L1_max_sys_msg = 50 [SiteController.conf] # required for the USB export start_usb_exporter=Yes [remote_exec_calls] # required for the export process visualization using the LED led_on=/usr/local/bin/led.on led_off=/usr/local/bin/led.off
Following configuration lines in the SiteController.cfg are optional:
# Optimization to have permanently more data in L2. # !!! Following 2 lines should be deleted if the SiteController is going to operate online !!! max_items_L1 = 500 L1_max_sys_msg = 50
With these settings new data are moved much earlier to the L2 database file and available for USB storage. More latest data are available to get copied. On the other hand these settings are not sufficient for the online mode, when the SiteController is connected to cloud. In this case the L2 database should be accessed as less as possible. With more L1 cache space the data can be better combined and uploaded.
If the lines are not present in the SiteController.cfg, the default is used (15000,5000).
Site template
The control of the LED for the visualization of the USB export process is completely realized by configuration of the SiteController. For this purpose, the site template of the SiteController should (also) inherit a component template like the following one:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<component_template config_version="1.0" schema_version="1.2.8">
<description>
NIFE103 USB export monitor by LED
</description>
<class>multi-sensor</class>
<vendor>azeti</vendor>
<version>2</version>
<sensors>
<sensor sensor_id="cloud_connected">
<sensor_class>unknown</sensor_class>
</sensor>
<sensor sensor_id="blink_indicator">
<description>Indicates the current state of the blinker rule</description>
<sensor_class>boolean</sensor_class>
<state_evaluation_expressions>
<state_evaluation_expression>
<expression>value</expression>
<true>ON</true>
<false>OFF</false>
</state_evaluation_expression>
</state_evaluation_expressions>
</sensor>
<sensor sensor_id="usb_export_state">
<description>Shows the current state of the USB exporter</description>
<sensor_class>code</sensor_class>
<state_evaluation_expressions>
<state_evaluation_expression >
<expression>value == 'error'</expression>
<true severity="200">%value%</true>
<false severity="100">%value%</false>
</state_evaluation_expression>
</state_evaluation_expressions>
</sensor>
</sensors>
<devices>
<device device_id="VirtualSensorProvider">
<simulator/>
</device>
</devices>
<actions>
<action action_id="LED_control" device_id="VirtualSensorProvider">
<commands>
<command command_id="led_on">launch_exec_job('led_on')</command>
<command command_id="led_off">launch_exec_job('led_off')</command>
</commands>
</action>
<action action_id="blinker_switch_1" device_id="VirtualSensorProvider">
<commands>
<command command_id="on">publish_calibrated_result('blinker_switch_1', True)</command>
<command command_id="off" exec_on_startup="true">publish_calibrated_result('blinker_switch_1', False)</command>
</commands>
</action>
<action action_id="blinker_switch_2" device_id="VirtualSensorProvider">
<commands>
<command command_id="on">publish_calibrated_result('blinker_switch_2', True)</command>
<command command_id="off" exec_on_startup="true">publish_calibrated_result('blinker_switch_2', False)</command>
</commands>
</action>
<action action_id="blink_indicator" device_id="VirtualSensorProvider">
<commands>
<command command_id="indicator_on" fallback_command_id="off" fallback_time_ms="500">publish_calibrated_result('blink_indicator', True)</command>
<command command_id="indicator_off" fallback_command_id="on" fallback_time_ms="500">publish_calibrated_result('blink_indicator', False)</command>
<command command_id="on">publish_calibrated_result('blinker', True)</command>
<command command_id="off">publish_calibrated_result('blinker', False)</command>
</commands>
</action>
</actions>
<ac_rules>
<rule rule_id="blinker_1">
<triggers>
<trigger sensor_id="blinker_switch_1" trigger_topic="calibrated_result" value_name="blink_switch" value_type="boolean"/>
<trigger sensor_id="blinker" trigger_topic="calibrated_result" value_name="blink_state" value_type="boolean"/>
</triggers>
<conditions>
<condition expr="blink_switch and blink_state">
<true>
<action_list>
<action action_id="blink_indicator" command="indicator_on"/>
<action action_id="LED_control" command="led_on"/>
</action_list>
</true>
</condition>
<condition expr="blink_switch and not blink_state">
<true>
<action_list>
<action action_id="blink_indicator" command="indicator_off"/>
<action action_id="LED_control" command="led_off"/>
</action_list>
</true>
</condition>
</conditions>
</rule>
<rule rule_id="blinker_2">
<triggers>
<trigger sensor_id="blinker_switch_2" trigger_topic="calibrated_result" value_name="blink_switch" value_type="boolean"/>
<trigger sensor_id="blinker" trigger_topic="calibrated_result" value_name="blink_state" value_type="boolean"/>
</triggers>
<timers>
<timer delay="1" timer_id="T_on"/>
<timer delay="1" timer_id="T_off"/>
</timers>
<conditions>
<condition expr="blink_switch">
<false>
<kill_timers>
<timer_id>T_on</timer_id>
<timer_id>T_off</timer_id>
</kill_timers>
</false>
</condition>
<condition expr="blink_switch and blink_state">
<true>
<action_list>
<action action_id="LED_control" command="led_on"/>
</action_list>
<result_list>
<result value_type="boolean" sensor_id="blink_indicator" result_value="True"/>
<result value_type="boolean" sensor_id="blinker" result_value="False" timer="T_on"/>
</result_list>
</true>
</condition>
<condition expr="blink_switch and not blink_state">
<true>
<action_list>
<action action_id="LED_control" command="led_off"/>
</action_list>
<result_list>
<result value_type="boolean" sensor_id="blink_indicator" result_value="False"/>
<result value_type="boolean" sensor_id="blinker" result_value="True" timer="T_off"/>
</result_list>
</true>
</condition>
</conditions>
</rule>
<rule rule_id="show_usb_export_state">
<triggers>
<trigger sensor_id="usb_export_state" trigger_topic="calibrated_result" value_name="usb_state" value_type="string"/>
</triggers>
<conditions>
<condition expr="usb_state == 'normal_operation'">
<true>
<action_list>
<action action_id="LED_control" command="led_on"/>
</action_list>
<result_list>
<result result_value="False" sensor_id="blinker_switch_1" value_type="boolean"/>
<result result_value="False" sensor_id="blinker_switch_2" value_type="boolean"/>
<result result_value="True" sensor_id="blink_indicator" value_type="boolean"/>
</result_list>
</true>
</condition>
<condition expr="usb_state == 'in_progress'">
<true>
<result_list>
<result result_value="True" sensor_id="blinker_switch_2" value_type="boolean"/>
</result_list>
</true>
</condition>
<condition expr="usb_state == 'success'">
<true>
<result_list>
<result result_value="True" sensor_id="blinker_switch_1" value_type="boolean"/>
</result_list>
</true>
</condition>
<condition expr="usb_state == 'error'">
<true>
<action_list>
<action action_id="LED_control" command="led_off"/>
</action_list>
<result_list>
<result result_value="False" sensor_id="blinker_switch_1" value_type="boolean"/>
<result result_value="False" sensor_id="blinker_switch_2" value_type="boolean"/>
<result result_value="False" sensor_id="blink_indicator" value_type="boolean"/>
</result_list>
</true>
</condition>
</conditions>
</rule>
</ac_rules>
</component_template>
Preparation for the data restoration
It is important to have the complete site template of the SiteController, from which the offline data are retrieved, available for the data restoration procedure, described below. This site template is also exported to the USB flash drive . But the SiteController was once configured, probably by cloud, and thus this site template could be downloaded from that place, where the SiteController was configured.
Preparation of a USB flash drive
The USB flash drive, on which to store the collected data, must have a single formatted partition containing a file named "Password.txt" in the root folder. The file "Password.txt" needs to contain a clear text password, which must match the one preconfigured in the SiteController.cfg.
The USB flash drive may already contain other files, even older export files, because they get a unique name. Of course, the USB flash drive should have enough free space.
Export procedure
There are 4 states regarding the USB data export.
| USB export state | short description | LED indicator |
|---|---|---|
| normal_operation | To indicate that the SiteController is running in regular operation and there's no USB export in progress. | LED is permanently glowing |
| in_progress | The USB export is in progress. Do not remove the USB drive in that state! | LED flashes every second |
| error | There was an error while exporting the Database to the USB drive. | LED is permanently off |
| success | The export of the database has finished successfully. You may remove the USB drive now. | LED flashes every half a second |
The following diagram shows two possible flows of operation:
The LED to observe is the 'GPO1', the most left one in the LED bar
.JPG?version=1&modificationDate=1543339248821&cacheVersion=1&api=v2&width=533&height=400)
Steps to do the data export
- Insert the prepared USB flash drive, the LED should permanently glow (normal_operation)
- It takes one or two seconds, to see the next state of the USB data export, either 'error' (LED permanently off) or 'in_progress' (LED flashes every second)
- In case of 'error', remove the USB flash drive
- In case of 'in progress', wait until the shows the 'success' state (LED flashes every half a second)
- In case of 'success', remove the USB flash drive
After a successful export operation the USB flash drive contains a file named like this: data_store-NIFE103_test-20181127T174239+0000.db with the meaning data_store_<Serial ID of the SiteController>-<datetime+timezone>.db.
The USB flash drive contains also the sensor configuration of the SiteController in a file named like this: sensor_config-NIFE103_test-20181127T174239+0000.xml with the meaning sensor_config-<Serial ID of the SiteController>-<datetime+timezone>.xml.
In case of the error state
If it is possible to open a shell on the device, it would be possible to crosscheck the file /opt/azeti/SiteController/log/usb_exporter.log to find the cause why the export failed. If there is no possibility to open a shell, you could crosscheck the USB flash drive regarding free space, partitions, password.
Usage of the exported data
The database file could be copied into the /opt/azeti/SiteController/persistent folder of another regular (stopped) SiteController installation of same version as the data was gathered with. Make sure there are no other files in that persistent folder and the SiteController.cfg is properly configured to have access to the cloud before starting the SiteController.
Of course, as the data_store.db file on the USB drive is just a sqlite3 database file, it could easily reviewed or processed directly with an appropriate sqlite client, too.
Procedure to restore the formerly exported data to cloud
The following steps describe how to get the exported data to the cloud, but only these data (not mixed with other data), separated in a new site.
Preparation of the cloud
- Verify in the Control Panel at
/Management / Templates / Site Templatesif there is a site template checked as default. - If so, open the drop-down menu at the right side and press the Delete Default
Preparation of a SiteController
The SiteController might be an azeti-machina or any other SiteController installation.
SSH access is required to open a shell and to copy the database file. In the following documentation all actions with the shell assume the execution with root permission by carrying out the sudo su after opening the shell, which could be done once and at this point.
The SiteController should have a tested connection to cloud with appropriate [data_store] settings in the /opt/azeti/SiteController/config/SiteController.cfg.
# There are more settings in this file, here are setting relevant for the upload [General] # It is recommended here to use a new serial ID, to avoid the presence of not related (former) data serial=a_unique_serial_id [ExternalBroker] host= the.cloud.broker port=8883 user_id=valid@user.id password=LetMeIn organizationShortName=my_org tls_enable=True tls_version=PROTOCOL_TLSv1_2 [data_store] L2_max_imgdata=-1 L2_max_sys_msg=-1 L2_max_events=-1 L2_max_HD=-1 L2_max_age_days=9999 L2_max_total_size_MiB=9999 L2_time_of_day_to_cleanup=02:00
Follow these steps after the cloud connection is established:
Stop the SiteController execution
/opt/azeti/SiteController/run_SiteController.py stop
Empty the running site template
rm /opt/azeti/SiteController/config/sensor_config.xml
Stop the mosquitto MQTT broker (To avoid any remaining data which does not belong to the data of the exported database)
/opt/azeti/SiteController/run_SiteController.py stop_mosquitto
Remove the mosquitto database, as well as the SiteController data store database
rm /var/lib/mosquitto/mosquitto.db rm /opt/azeti/SiteController/persistent/*
- For the same benefit change the Serial (
serial=<my serial>) of the SiteController to a new one in theSiteController.cfg. - After carrying out a
/opt/azeti/SiteController/run_SiteController.py startthe new site should appear in cloud with a blank Admin and Life table:
After this verification stop the SiteController execution again by
/opt/azeti/SiteController/run_SiteController.py stop
Make sure, the site template, which the SiteController used where the data to restore come from, is available.
Database restoration steps
Below steps are only intended for experts and require an active azeti Cloud Subscription. Please contact consulting@azeti.net before you continue.
With this prepared environment (SiteController suite is still not running) This can be done by several means, like using the Windows WinSCP. In case of trouble regarding permissions you could copy it first to your home directory at the SiteController device and then with su permissions to /opt/azeti/SiteController/persistent .
- Copy (SFTP transfer, e.g. WinSCP or Filezilla) the database, retrieved with the USB flash drive, e.g.
data_store-NIFE103_test-20181127T174239+0000.db,to/opt/azeti/SiteController/persistent Rename this database to data_store.db
mv /opt/azeti/SiteController/persistent/data_store-NIFE103_test-20181123T103643+0000.db data_store.db .
Copy the site template that was originally used within the SiteController that you restore, e.g.
sensor_config-NIFE103_test-20181127T174239+0000.xml, into/opt/azeti/SiteController/configand rename it tosensor_config.xmlIt is recommended to use the original serial ID in
/opt/azeti/SiteController/config/SiteController.cfgnow, which can be found within the name of the exported database file.Start the upload of the data by starting the SiteController with
/opt/azeti/SiteController/run_SiteController.py startThe upload of the historical data can take quite some time, depending on the used network connection, but at least 15 minutes for most scenarios. You'll now be able to see the data in your azeti Cloud organization under the SiteController that was used for restore.