Distinction between this Watch Dog and the other Watch Dog

In order to distinguish the watch dog documented here from the one implemented in Python being a part of the SiteController for years by now, the function range of the two watch dogs are briefly recapitulated here.

The Watch Dog of the SiteController

The Python watch dog is monitoring the SiteController modules that are supposed to run according to its sensor configuration (a.k.a. Site template). That watch dog will send a special request telegram in a regular interval to the modules and expects an answer from each of the modules stating the internal health state of each module by themselves. The watch dog waits for those answers. Should an answer time out, the watch dog checks if that modules has a process in the operating system process table.

If the process is missing, the watch dog declares the module as crashed and immediately tries to start that process anew. If by contrast the watch dog finds a process however, it assumes that process to be hanging and reports this to the cloud server. But it will not try to automatically heal the situation as sometimes a process just needs a bit of extra time for completing the processing of a bigger set of data.

A Hardware Watch Dog

The type of watch dog this document is about however is supposed to check the entire system for its health state. It uses a special watch dog hardware timer that needs to be reset (kicked) in regular intervals. If that watch dog times out the system will be rebooted immediately.

That could bring a system back up in case it crashed entirely, without any intervention of an operator required.

The Hardware Watch Dog of a NIFE103

The NIFE103 comes with a chip (namely NCT7904D) that allows (next to monitoring other critical system sensor parameters) to use a hardware based watch dog.

A (very rough) documentation of the NIFE103's Watch Dog functionality could be found in [1] and a documentation of the specific hardware monitoring chip in [2].

In short, the watch dog needs to be configured and kicked via SMBus ioctls. An attempt to do so via a Python script directly was unsuccessfull so the actual calls needed to be implemented and compiled in C.

Requirements

• The binaries provided by this package requires to run on a Nexcom NIFE103 hardware that has a NCT7904D chip build in which answers on i2c-7 bus on address 2d_hex.

  Do not attempt to run these binaries on any other hardware as they may otherwise permanently damage your system.

• The binaries require the i2c support either compiled into the kernel or as a module.
• As of time of writing of this README, the nct7904 kernel module needs to be blacklisted in modprobe configuration.
• The kernel module i2c_i801 must NOT be blacklisted in modprobe configuration.
• The binaries have to run as root user (either natively, via sudo or by running suid root).

Installing NIFE103 Watch Dog Support

1. If not already provided by your system's installed OS image, extract the files from the archive into your ${SITECONTROLLER_HOME}/scripts folder. This is usually located at /opt/azeti/SiteController/scripts:

   mkdir -p /opt/azeti/SiteController/scripts
   tar -xvzf NIFE103-control.tar.gz -C /opt/azeti/SiteController/scripts

2. make sure the binaries have correct ownership and permissions:

   chown root:root /opt/azeti/SiteController/scripts/NIFE103-wdt* \
     /opt/azeti/SiteController/scripts/watchdog.sh
   chmod 0700 /opt/azeti/SiteController/scripts/NIFE103-wdt* \
     /opt/azeti/SiteController/scripts/watchdog.sh

3. Double check nct7904 is blacklisted:

   grep "nct7904" /etc/modprobe.d/blacklist*
   /etc/modprobe.d/blacklist.conf:blacklist nct7904

   If it is not blacklisted, just add a blacklist nct7904 entry to /etc/modprobe.d/blacklist.conf or remove the # in the beginning of the line, if that entry was just commented out.

4. Double check i2c_i801 is NOT blacklisted (note the #):

   grep "i2c_i801" /etc/modprobe.d/blacklist*
   /etc/modprobe.d/blacklist.conf:#blacklist i2c_i801

   If it is blacklisted, just add a # at the beginning of that line.

5. modify the SiteController.cfg file a.k.a. Site configuration so that in section [remote_exec_calls] the following entries are present:

   [remote_exec_calls]
   watchdog_start = /opt/azeti/SiteController/scripts/watchdog.sh start
   watchdog_stop = /opt/azeti/SiteController/scripts/watchdog.sh stop
   watchdog_kick = /opt/azeti/SiteController/scripts/watchdog.sh kick

6. upload the watchdog-NIFE103.template.xml file as a new component template to the cloud (if it's not already available there, that is)

7. add this component template to your Site template in the usual way.

The Files included in the Tarball

Using NIFE103 Watch Dog Support

The SiteController is now set up to be monitoring the system state. All it remains to do is a restart of the SiteController. Once restarted the hardware based watch dog is initialized and started, the timer in the automation rule set will kick the watchdog_kick action every 120 seconds which in turn will call the necessary io controls to reset the timer.

Should the watch dog run into a time-out after 10 minutes without any call to these ioctl, the system is rebooted.

Restrictions

• By using these binaries you can no longer use the lm-sensors package to monitor the NIFE103 hardware sensors because otherwise the kernel module locks the access to the SMBus.
• Does not yet cooperate with watchdogd
• as the watch dog is using the same ioctl as the indicator LED on the front panel of the NIFE103 the two features may interfere with each other (still needs to be tested).

References

• [1]: http://files.nexcom.com/Driver/NIFE103/User_Manual_NIFE103_170928.pdf (local copy)

• [2]: https://www.nuvoton.com/resource-files/NCT7904D_Datasheet_V1.44.pdf (local copy)

Next Steps