SiteController 3.0.0 Release Notes

What’s new

• Hello World from Python 3!
  SiteController is now running under Python 3.8+, with some compatibility still in place for Python 2.7

• Tani PlcEngine upgraded to 2.4.6, this brings a lot of new features, bug-fixes and licensing options

• Now we provide an official SiteController docker image based on Ubuntu 20 for x86_64 architecture

• Several libraries were upgraded

• SiteController's watchdog now can kill and restart processes that were detected to be hanging after a configurable amount of time

• When upgrading, the SiteController installer now creates a backup of PlcEngine settings, too

• automation_controller now can set a severity to a generated event

• automation_controller now supports skipping rules with uninitialized states

• We streamlined several modules, removing unused or outdated ones

Breaking Changes

Several parts of the SiteController and 3^rd party libraries were upgraded, the following list contains an overview of the breaking changes. Please consult the /wiki/spaces/SIT/pages/3051225116 or contact azeti's Consulting team for more details.

• external_checker module is now running on Python 3, this means all the external scripts running as a python module need to be migrated to Python 3 as well

• Due to the upgrade of PlcEngine from version 1.x to 2.x, a license reactivation will be required, so should you have PLCs or OPC UA devices in your configurations, please contact azeti Consulting on how to do that.

• Due to the double Python environment, the virtual environment’s directory of the SiteController changed from /opt/azeti/SiteController/lib/ to /opt/azeti/SiteController/lib/python2/ and /opt/azeti/SiteController/lib/python3/

• Python 3.8 or higher is now mandatory, this means it is required that the gateway machine is running Ubuntu 20+ or Debian 11, or may need to have Python 3.8+ installed manually

• The following modules were removed, contact the azeti Consulting team if you need a replacement:

  • core

    • usb_exporter (used to export the local database to USB pen drive upon sticking it into the port and a check for a password stored on the media is valid)

  • VirtualSensor

    • access_control (used to conveniently implement a physical access control – opening doors via pinpad or keycard reader, supporting alert conditions and dead man button scenarios)

    • tank_simulator (used for demonstration and internal self monitoring purposes mainly)

    • fill_quantity (convenient volume calculation routines for tanks in different shapes and configurations)

    • flow_rate (convenient calculations for change in volume of a tank with theft detection)

  • Data Acquisition

    • camera image acquisition support

    • GPS TCP/Serial support

    • BLE scanner support

    • SimonsVoss SmartIntego support

Bug-fixes

• Fixed specific case leading to an unhandled exception in automation_controller

• Added error handling for some edge cases where PlcEngine interface is not properly initialized

• Fixed a condition leading the SiteController launcher to get stuck

• Fixed error of a base class leading to an AttributeError: 'Client' object has no attribute 'connack_string' during an unclean MQTT disconnection

• Added proper handling of a missing hd entry in calibrated_results being processed by history_analyzer

• Fixed RuntimeError: dictionary changed size during iteration exception sometimes occurring in raw_results_demux

• Implemented handling for unknown boolean values in SiteController.cfg

• When executing scripts as modules, external_checker now calls the method run() once instead of twice per iteration only

• Improved error handling on data_store, previously causing problems when the SSL encryption was enabled

• Fixed OSError: [Errno 99] Cannot assign requested address in launcher, when restarting a module in a dockerized mosquitto setup

• Fixed some cases where watchdog was unable to restart data_store

SiteController version 2.1.0 - Minor Release

Important notes:

What's new

• Data retention feature

Bug fixes

• Big sensor configurations cause the default mosquitto in Ubuntu 16.04 and 18.04 to silently drop messages
• Entry on logs show - "don't know this tob type"
• Wrong launcher path in sitecontroller-mosquitto docker image entrypoint
• Database is archived when data retention recovery scripts takes too long
• data_store may delete unsent retained results during startup
• raw_results_demux hangs after receiving a config update
• data_store does not save the severity of an event in the data_store.db
• After each SiteController restart ancient results are published to the cloud again, and again, …
• "ValueError: unmatched '{' in format" in automation_controller
• TypeError: unbound method from_json() must be called with calibrated_result instance as first argument
• Upload strategy "on_change" does not filter all same values
• UnicodeDecodeError: 'ascii' codec can't decode byte 0xb4 in position 65: ordinal not in range(128) in data_store.log
• AttributeError: 'module' object has no attribute '_strptime' in external_checker.log
• GPS: UnicodeDecodeError: 'ascii' codec can't decode byte 0x83 in position 0: ordinal not in range(128) -- Logged from file nmea_decoder.py, line 74
• Watchdog does not revive the process after a crashed data_store
• Docker image entrypoint is pointing to old path
• Ensure external_checker handle properly unicode exceptions
• TypeError: decoding Unicode is not supported in CalibResultsEvaluator
• During startup the Watchdog logs a lot of messages with warning state about modules that are "not expected" and "not running"
• Workaround to have interval-based upload
• Fixes to Dockerfile
• Refactor Docker setup

SiteController version 2.0.0 - Major Release

What's new

• Include the latest version of Tani PLC Engine to SC 2.0 Major Release
• SC repo restructure: migration script to convert old SiteController.cfg config files to new one
• SC repo restructure: gps_tcp_device needs to be updated
• SC repo restructure: update_src job -- Remove component
• SC repo restructure: setup must inspect sensor_config.xml for compatibility before update
• Send running sensor_config.xml to a topic in the cloud via additional script and by data_store

Bug fixes

• Last will then hanging data_store after active monitoring timeout
• Setup fails with: "ImportError: No module named pyxb"
• Workaround: Updating SC to version 1.9.5 breaks upload to cloud
• Regresssion: Enable SiteController setup to run completely parametrized via command line
• Regression: data_store sometimes does not send the disconnected event to external broker if it is shut down
• AttributeError: 'checker_sensor_gateway' object has no attribute '_job' in external_checker.log
• SiteController does not start automatically after a reboot of a UniPi or debian system
• AttributeError: 'NoneType' object has no attribute 'recv' in Modbus with IoT-Kit config
• TypeError: to_json() takes exactly 1 argument (2 given) in data_store when live monitoring turns off
• Symlink run_SiteController.py is not properly created after an upgrade
• SC stops sending same values after some time for upload strategy as interval
• Test of core.test_jobs_module_command.TestCreateExternalConnection shows exception and passes
• Traceback when trying to run PLC checks
• Watchdog still hangs if SC is supplied with a specific sensor_config.xml in some cases
• AttributeError: 'RawResultsDeMux' object has no attribute '_processResult'
• modbus: AttributeError: 'module' object has no attribute 'mqtt_handler'
• HistoryAnalyser.py seems not to be renamed correctly somewhere
• REST client crashes on start and print traceback on console
• Bug when trying to update the SiteController
• Investigate: data_store: RLock is locked by <super: <class 'RLockLogger'>, <RLockLogger object>>
• Traceback to console in the Watchdog
• PLC Setup fails on Rockwell CPU scan
• No sensor config update immediatly after a tsap/cpu scan
• Some files in the installer are world writable, especially in v.2.0 all scripts files are
• Traceback when trying to install Fixed version SC 2.0.0
• Introduction of keywords "-MAJOR", "-MINOR" and "-PATCH" in version names screws up the whole version numbering
• Already transferred hd data (last field = 3) may get stuck in the L2 persistent database
• SC repo restructure: Don't process invalid sensor_config.xml
• SC repo restructure: Fix issues found in tests of SiteController after SC restructure
• Watchdog sometimes hangs and the launcher is unable to detect it is still in the process list so it does not even try to kill it

SiteController version 1.8 - Minor Release

What's new

• Extended logging mechanism for PLC modules
• Prodos polling and logfile enhancement
• Trigger action with the da_rest_client module
• SiteController configuration management from the cloud
• Rockwell SPS implementation
• RAMDisk Feature

Bug Fixes

• Topic reported are now working according to the Watchdog
• Traceback when publishing certain configuration combinations
• Configuration update from cloud
• Mismatch of topics
• Deadlock of Modbus
• PLC symbol scan too big for Ramdisk
• Rev-info with wrong information
• Sysid tool missing in installer
• TrapDaemon - Unicode issue
• Memory leak in AzetiInterface
• Backward compatibility with systemid generator
• Logging level set to info by default
• Jobprocessor Traceback when executing a command
• Smaller Unittest fixes

SiteController version 1.6 - Codename Aspirator

What's new

• New job to handle requests for module/<module>/command
• The build scripts now integrate PlcEngine and the Azeti Interface software on the SC installer
• Refactor of the icmp
• The Azeti interface is now in compliance with the cloud configuration
• Automation Script that installs the OS + Azeti engine + Azeti config tool + Azeti driver + SiteController Aspirator on a Nexcom NIFE103
• Complely integrated our structure to the new Aspirator changes
• We now have performance metrics for the Azei driver module
• Smoke tests are now defined
• The System is now able to work with proxy authentication
• Integration tests where vastly aplyed in the major supported plc plataforms
• We created a configuration to visualize the usb data export using the one LED of the NIFE103
• We now support the state representation "calibrated result" and "configuration dump"
• We put in place remote scripts for common maintenance tasks (Reboot, Starting Services)
• Smoke tests for the Aspirator release
• All the main PLC and OPC-UA features were tested
• There are no more sys_id in internal mqtt topics
• Demux now accept partial/incomplete raw_results
• We now support UTF-8 characters for Aspirator and all Azeti related modules (config_provider)
• The now are able to handle large files
• The default TLS version is now v1.2
• Paho-mqtt version is now 1.4.0

Bug Fixes

• Date and time are now following ISO8601 format
• All the modules now have UTF-8 support
• Fixed recurring misconfiguration sys messages
• Lost connection to external broker is now fixed
• AzetiInterface crashes has been fixed
• Error on big datapoints has been fixed
• No more error messages during plcengine installation
• No more data_store crashes when selecting a different time zone than UTC
• No more hanging state thread on data_store
• PlcEngine interface software is not publishing raw_results with "garbage" after the json message anymore
• OPC-UA scan for symbols is now working
• IDs of the MQTT topics are now encoded before using them in topic names
• Now all the datapoints on a Symbol file are imported
• Found the source of the the message "error: bad char in struct format"
• Fix the problem that made the data_store stop communicating with the external broker
• %value% is now translated to the state
• ICMP module is now not generating tracebacks on screen anymore
• Fix the problem that amde the data_store stop publishing to the cloud
• AzetiInterface now publishes debug messages even if compiled in non-debug mode
• Fixed the data flow from the Prodos web server to the Sitecontroller offline storage