This feature is end of life starting with Site Controller version 3.0.

Introduction

The SiteController is actually intended to send acquired sensor data and other information to cloud as fast as possible. But if an upload of the data is not possible or not intended, for instance by the configuration of the SiteController, the collected sensor data are stored locally until a connection to the cloud is established. The USB data export procedure is supported by the visualization using one LED at the azeti-machina NEXCOM NIFE103.

Neither a keypad nor a monitor are required.

SiteController is regularly uploading data to the azeti Cloud if it has an active cloud connection. You'll only see data if there was a period of offline time.

This document describes how to configure the SiteController to force the local storage of data and how to retrieve the collected data offline to a prepared USB flash drive. Furthermore is added, how these data on the USB flash drive could be used.

On this page:

Related pages:

- Page:
  
  Scheduling, Timing and Polling Intervals

Prerequisites

Configuration of the SiteController

SiteController.cfg

The following code block example shows settings in the SiteController configuration file, which is located in /opt/azeti/SiteController/config/SiteController.cfg There are also other entries in the configuration file, but they are not related to the topic which is described in this document.

The comment lines, starting by #, explain the settings.

Offline data storage and USB export relevant settings in the SiteController.cfg

[ExternalBroker]
# left empty to guarantee the offline storage
host=
port=1883
# not required with no connection
user_id=
password=
# an organizationShortName is still required
organizationShortName=my_org
tls_enable=False
 
[data_store]
# this password must be equal to the password at the USB flash drive
L2_export_password=P@s$w0rd!
# required for the USB data export
export_status_representation=sensor
export_status_sensor_id=usb_export_state
# -1 means no limitation by the table based size limits
L2_max_imgdata=-1
L2_max_sys_msg=-1
L2_max_events=-1
L2_max_HD=-1
# limitations to the offline storage
L2_max_age_days=183
L2_max_total_size_MiB=1024
L2_time_of_day_to_cleanup=02:00
# Optimization to have permanently more data in L2.
# !!! Following 2 lines should be deleted if the SiteController is going to operate online !!!
max_items_L1 = 500
L1_max_sys_msg = 50
 
[SiteController.conf]
# required for the USB export
start_usb_exporter=Yes

[remote_exec_calls]
# required for the export process visualization using the LED
led_on=/usr/local/bin/led.on
led_off=/usr/local/bin/led.off

Following configuration lines in the SiteController.cfg are optional:

# Optimization to have permanently more data in L2. 
# !!! Following 2 lines should be deleted if the SiteController is going to operate online !!! 
max_items_L1 = 500
L1_max_sys_msg = 50

With these settings new data are moved much earlier to the L2 database file and available for USB storage. More latest data are available to get copied. On the other hand these settings are not sufficient for the online mode, when the SiteController is connected to cloud. In this case the L2 database should be accessed as less as possible. With more L1 cache space the data can be better combined and uploaded.

If the lines are not present in the SiteController.cfg, the default is used (15000,5000).

Site template

The control of the LED for the visualization of the USB export process is completely realized by configuration of the SiteController. For this purpose, the site template of the SiteController should (also) inherit a component template like the following one:

component template NIFE103 USB export monitor by LED

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<component_template config_version="1.0" schema_version="1.2.8">
  <description>
  NIFE103 USB export monitor by LED
  </description>
  <class>multi-sensor</class>
  <vendor>azeti</vendor>
  <version>2</version>
  <sensors>
    <sensor sensor_id="cloud_connected">
      <sensor_class>unknown</sensor_class>
    </sensor>
    <sensor sensor_id="blink_indicator">   
      <description>Indicates the current state of the blinker rule</description>
      <sensor_class>boolean</sensor_class>
      <state_evaluation_expressions>
        <state_evaluation_expression>
          <expression>value</expression>
            <true>ON</true>
            <false>OFF</false>
        </state_evaluation_expression>
      </state_evaluation_expressions>
    </sensor>
    <sensor sensor_id="usb_export_state">
      <description>Shows the current state of the USB exporter</description>
      <sensor_class>code</sensor_class>
      <state_evaluation_expressions>
        <state_evaluation_expression >
          <expression>value == 'error'</expression>
          <true severity="200">%value%</true>
          <false severity="100">%value%</false>
        </state_evaluation_expression>
      </state_evaluation_expressions>
    </sensor>
  </sensors>
  <devices>
    <device device_id="VirtualSensorProvider">
      <simulator/>
    </device>
  </devices>
  <actions>
    <action action_id="LED_control" device_id="VirtualSensorProvider">
      <commands>
        <command command_id="led_on">launch_exec_job('led_on')</command>
        <command command_id="led_off">launch_exec_job('led_off')</command>
      </commands>
    </action>
    <action action_id="blinker_switch_1" device_id="VirtualSensorProvider">
      <commands>
        <command command_id="on">publish_calibrated_result('blinker_switch_1', True)</command>
        <command command_id="off" exec_on_startup="true">publish_calibrated_result('blinker_switch_1', False)</command>
      </commands>
    </action>
    <action action_id="blinker_switch_2" device_id="VirtualSensorProvider">
      <commands>
        <command command_id="on">publish_calibrated_result('blinker_switch_2', True)</command>
        <command command_id="off" exec_on_startup="true">publish_calibrated_result('blinker_switch_2', False)</command>
      </commands>
    </action>
    <action action_id="blink_indicator" device_id="VirtualSensorProvider">
      <commands>
        <command command_id="indicator_on" fallback_command_id="off" fallback_time_ms="500">publish_calibrated_result('blink_indicator', True)</command>
        <command command_id="indicator_off" fallback_command_id="on" fallback_time_ms="500">publish_calibrated_result('blink_indicator', False)</command>
        <command command_id="on">publish_calibrated_result('blinker', True)</command>
        <command command_id="off">publish_calibrated_result('blinker', False)</command>
      </commands>
    </action>
  </actions>
  <ac_rules>
    <rule rule_id="blinker_1">
      <triggers>
        <trigger sensor_id="blinker_switch_1" trigger_topic="calibrated_result" value_name="blink_switch" value_type="boolean"/>
        <trigger sensor_id="blinker" trigger_topic="calibrated_result" value_name="blink_state" value_type="boolean"/>
      </triggers>
      <conditions>  
        <condition expr="blink_switch and blink_state">
          <true>
            <action_list>
                <action action_id="blink_indicator" command="indicator_on"/>
                <action action_id="LED_control" command="led_on"/>
            </action_list>
          </true>
        </condition>
        <condition expr="blink_switch and not blink_state">
          <true>
            <action_list>
              <action action_id="blink_indicator" command="indicator_off"/>
              <action action_id="LED_control" command="led_off"/>
            </action_list>
          </true>
        </condition>
      </conditions>
    </rule>
    <rule rule_id="blinker_2">
      <triggers>
        <trigger sensor_id="blinker_switch_2" trigger_topic="calibrated_result" value_name="blink_switch" value_type="boolean"/>
        <trigger sensor_id="blinker" trigger_topic="calibrated_result" value_name="blink_state" value_type="boolean"/>
      </triggers>
      <timers>
        <timer delay="1" timer_id="T_on"/>
        <timer delay="1" timer_id="T_off"/>
      </timers>
      <conditions>
        <condition expr="blink_switch">
          <false>
            <kill_timers>
              <timer_id>T_on</timer_id>
              <timer_id>T_off</timer_id>
            </kill_timers>
          </false>
        </condition>
        <condition expr="blink_switch and blink_state">
          <true>
            <action_list>
              <action action_id="LED_control" command="led_on"/>
            </action_list>
            <result_list>
              <result value_type="boolean" sensor_id="blink_indicator" result_value="True"/>
              <result value_type="boolean" sensor_id="blinker" result_value="False" timer="T_on"/>
            </result_list>
          </true>
        </condition>
        <condition expr="blink_switch and not blink_state">
          <true>
            <action_list>
              <action action_id="LED_control" command="led_off"/>
            </action_list>
            <result_list>
              <result value_type="boolean" sensor_id="blink_indicator" result_value="False"/>
              <result value_type="boolean" sensor_id="blinker" result_value="True" timer="T_off"/>
            </result_list>
          </true>
        </condition>      
      </conditions>
    </rule>
    <rule rule_id="show_usb_export_state">
      <triggers>
        <trigger sensor_id="usb_export_state" trigger_topic="calibrated_result" value_name="usb_state" value_type="string"/>
      </triggers>
      <conditions>
        <condition expr="usb_state == 'normal_operation'">
          <true>
            <action_list>
              <action action_id="LED_control" command="led_on"/>
            </action_list>
            <result_list>
              <result result_value="False" sensor_id="blinker_switch_1" value_type="boolean"/>
              <result result_value="False" sensor_id="blinker_switch_2" value_type="boolean"/>
              <result result_value="True" sensor_id="blink_indicator" value_type="boolean"/>
            </result_list>
          </true>
        </condition>
        <condition expr="usb_state == 'in_progress'">
          <true>
            <result_list>
              <result result_value="True" sensor_id="blinker_switch_2" value_type="boolean"/>
            </result_list>
          </true>
        </condition>
        <condition expr="usb_state == 'success'">
          <true>
            <result_list>
              <result result_value="True" sensor_id="blinker_switch_1" value_type="boolean"/>
            </result_list>
          </true>
        </condition>
        <condition expr="usb_state == 'error'">
          <true>
            <action_list>
              <action action_id="LED_control" command="led_off"/>
            </action_list>
            <result_list>
              <result result_value="False" sensor_id="blinker_switch_1" value_type="boolean"/>
              <result result_value="False" sensor_id="blinker_switch_2" value_type="boolean"/>
              <result result_value="False" sensor_id="blink_indicator" value_type="boolean"/>
            </result_list>
          </true>
        </condition>
      </conditions>
    </rule>
  </ac_rules>
</component_template>

Preparation for the data restoration

It is important to have the complete site template of the SiteController, from which the offline data are retrieved, available for the data restoration procedure, described below. This site template is also exported to the USB flash drive . But the SiteController was once configured, probably by cloud, and thus this site template could be downloaded from that place, where the SiteController was configured.

Preparation of a USB flash drive

The USB flash drive, on which to store the collected data, must have a single formatted partition containing a file named "Password.txt" in the root folder. The file "Password.txt" needs to contain a clear text password, which must match the one preconfigured in the SiteController.cfg.

The USB flash drive may already contain other files, even older export files, because they get a unique name. Of course, the USB flash drive should have enough free space.

Export procedure

There are 4 states regarding the USB data export.

USB export state	short description	LED indicator
normal_operation	To indicate that the SiteController is running in regular operation and there's no USB export in progress.	LED is permanently glowing
in_progress	The USB export is in progress. Do not remove the USB drive in that state!	LED flashes every second
error	There was an error while exporting the Database to the USB drive.	LED is permanently off
success	The export of the database has finished successfully. You may remove the USB drive now.	LED flashes every half a second

The following diagram shows two possible flows of operation:

The LED to observe is the 'GPO1', the most left one in the LED bar

Steps to do the data export

Insert the prepared USB flash drive, the LED should permanently glow (normal_operation)
It takes one or two seconds, to see the next state of the USB data export, either 'error' (LED permanently off) or 'in_progress' (LED flashes every second)
In case of 'error', remove the USB flash drive
In case of 'in progress', wait until the shows the 'success' state (LED flashes every half a second)
In case of 'success', remove the USB flash drive

After a successful export operation the USB flash drive contains a file named like this: data_store-NIFE103_test-20181127T174239+0000.db with the meaning data_store_<Serial ID of the SiteController>-<datetime+timezone>.db.

The USB flash drive contains also the sensor configuration of the SiteController in a file named like this: sensor_config-NIFE103_test-20181127T174239+0000.xml with the meaning sensor_config-<Serial ID of the SiteController>-<datetime+timezone>.xml.

In case of the error state

If it is possible to open a shell on the device, it would be possible to crosscheck the file /opt/azeti/SiteController/log/usb_exporter.log to find the cause why the export failed. If there is no possibility to open a shell, you could crosscheck the USB flash drive regarding free space, partitions, password.

Usage of the exported data

The database file could be copied into the /opt/azeti/SiteController/persistent folder of another regular (stopped) SiteController installation of same version as the data was gathered with. Make sure there are no other files in that persistent folder and the SiteController.cfg is properly configured to have access to the cloud before starting the SiteController.

Of course, as the data_store.db file on the USB drive is just a sqlite3 database file, it could easily reviewed or processed directly with an appropriate sqlite client, too.

Procedure to restore the formerly exported data to cloud

The following steps describe how to get the exported data to the cloud, but only these data (not mixed with other data), separated in a new site.

Preparation of the cloud

Verify in the Control Panel at / Management / Templates / Site Templates if there is a site template checked as default.
If so, open the drop-down menu at the right side and press the Delete Default

Preparation of a SiteController

The SiteController might be an azeti-machina or any other SiteController installation.

SSH access is required to open a shell and to copy the database file. In the following documentation all actions with the shell assume the execution with root permission by carrying out the sudo su after opening the shell, which could be done once and at this point.

The SiteController should have a tested connection to cloud with appropriate [data_store] settings in the /opt/azeti/SiteController/config/SiteController.cfg.

Example SiteController.cfg for restoring data

# There are more settings in this file, here are setting relevant for the upload 
[General]
# It is recommended here to use a new serial ID, to avoid the presence of not related (former) data
serial=a_unique_serial_id

[ExternalBroker]
host= the.cloud.broker
port=8883
user_id=valid@user.id
password=LetMeIn
organizationShortName=my_org
tls_enable=True
tls_version=PROTOCOL_TLSv1_2

[data_store]
L2_max_imgdata=-1
L2_max_sys_msg=-1
L2_max_events=-1
L2_max_HD=-1
L2_max_age_days=9999
L2_max_total_size_MiB=9999
L2_time_of_day_to_cleanup=02:00

Follow these steps after the cloud connection is established:

Stop the SiteController execution

/opt/azeti/SiteController/run_SiteController.py stop

Empty the running site template

rm /opt/azeti/SiteController/config/sensor_config.xml

Stop the mosquitto MQTT broker (To avoid any remaining data which does not belong to the data of the exported database)
```
/opt/azeti/SiteController/run_SiteController.py stop_mosquitto
```
Remove the mosquitto database, as well as the SiteController data store database
```
rm /var/lib/mosquitto/mosquitto.db
rm /opt/azeti/SiteController/persistent/*
```
For the same benefit change the Serial (serial=<my serial>) of the SiteController to a new one in the SiteController.cfg.
After carrying out a /opt/azeti/SiteController/run_SiteController.py start the new site should appear in cloud with a blank Admin and Life table:
After this verification stop the SiteController execution again by
```
 /opt/azeti/SiteController/run_SiteController.py stop
```

Make sure, the site template, which the SiteController used where the data to restore come from, is available.

Database restoration steps

Below steps are only intended for experts and require an active azeti Cloud Subscription. Please contact consulting@azeti.net before you continue.

With this prepared environment (SiteController suite is still not running) This can be done by several means, like using the Windows WinSCP. In case of trouble regarding permissions you could copy it first to your home directory at the SiteController device and then with su permissions to /opt/azeti/SiteController/persistent .

Copy (SFTP transfer, e.g. WinSCP or Filezilla) the database, retrieved with the USB flash drive, e.g. data_store-NIFE103_test-20181127T174239+0000.db, to /opt/azeti/SiteController/persistent

Rename this database to data_store.db

mv /opt/azeti/SiteController/persistent/data_store-NIFE103_test-20181123T103643+0000.db data_store.db  .

Copy the site template that was originally used within the SiteController that you restore, e.g. sensor_config-NIFE103_test-20181127T174239+0000.xml, into /opt/azeti/SiteController/config and rename it to sensor_config.xml

It is recommended to use the original serial ID in /opt/azeti/SiteController/config/SiteController.cfg now, which can be found within the name of the exported database file.
Start the upload of the data by starting the SiteController with /opt/azeti/SiteController/run_SiteController.py start
The upload of the historical data can take quite some time, depending on the used network connection, but at least 15 minutes for most scenarios. You'll now be able to see the data in your azeti Cloud organization under the SiteController that was used for restore.