Overview

Shutdown design

When your UPS batteries get low, the operating system needs to be brought down cleanly. Also, the UPS load should be turned off so that all devices that are attached to it are forcibly rebooted.

Here are the steps that occur when a critical power event happens:

  1. The UPS goes on battery
  2. The UPS reaches low battery (a "critical" UPS), that is to say upsc displays:

    ups.status: OB LB

    The exact behavior depends on the specific device, and is related to:

    • battery.charge and battery.charge.low
    • battery.runtime and battery.runtime.low


  3. The upsmon master notices and sets "FSD" - the "forced shutdown" flag to tell all slave systems that it will soon power down the load.

    (If you have no slaves, skip to step 6)

  4. upsmon slave systems see "FSD" and:

    • generate a NOTIFY_SHUTDOWN event
    • wait FINALDELAY seconds - typically 5
    • call their SHUTDOWNCMD
    • disconnect from upsd


  5. The upsmon master system waits up to HOSTSYNC seconds (typically 15) for the slaves to disconnect from upsd. If any are connected after this time, upsmon stops waiting and proceeds with the shutdown process.
  6. The upsmon master:

    • generates a NOTIFY_SHUTDOWN event
    • waits FINALDELAY seconds - typically 5
    • creates the POWERDOWNFLAG file - usually /etc/killpower
    • calls the SHUTDOWNCMD


  7. On most systems, init takes over, kills your processes, syncs and unmounts some filesystems, and remounts some read-only.
  8. init then runs your shutdown script. This checks for the POWERDOWNFLAG, finds it, and tells the UPS driver(s) to power off the load.
  9. The system loses power.
  10. Time passes. The power returns, and the UPS switches back on.
  11. All systems reboot and go back to work.


Proposed Architecture

For this setup, we will be setting up our UPS master on a raspberry pi with slaves setup on a QNAP NAS and a Ubuntu Server.



Master Setup on Raspberry Pi


Connect UPS using USP cable

Installation

$ sudo apt-get install nut

Configuring the UPS/UPSD


$ vi /etc/nut/ups.conf


Add the following config

[qnapups]
  driver = usbhid-ups
    port = auto
    desc = "Back-UPS ES 750"

Configure upsd.conf

$ sudo vi /etc/nut/upsd.conf

Add the following config:

LISTEN 0.0.0.0 3493

Update Credentials

$ sudo vi /etc/nut/upsd.users
[admin]
  password = 123456
  actions = SET
  instcmds = ALL

[upsmon]
  password = 123456
  upsmon master

[deepthought]
  password = 123456
  upsmon slave

Update Nut.Conf

$ sudo vi /etc/nut/nut.conf 

Set Mode=netserver

MODE=netserver


Configuring UPS Monitor

$ vi /etc/nut/upsmon.conf

Add the following line

MONITOR qnapups@localhost 1 upsmon 123456 master
FINALDELAY 5


FINALDELAY seconds

When running in master mode, upsmon waits this long after sending the NOTIFY_SHUTDOWN to warn the users. After the timer elapses, it then runs your SHUTDOWNCMD. By default this is set to 5 seconds.

If you need to let your users do something in between those events, increase this number. Remember, at this point your UPS battery is almost depleted, so don’t make this too big.

Alternatively, you can set this very low so you don’t wait around when it’s time to shut down. Some UPSes don’t give much warning for low battery and will require a value of 0 here for a safe shutdown.



Verify Hardware Config

$ sudo upsdrvctl start 


Restart the Service

$ sudo service nut-server restart


Get Status

$ sudo service nut-server status

● nut-server.service - Network UPS Tools - power devices information server
   Loaded: loaded (/lib/systemd/system/nut-server.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2021-08-24 20:28:53 EDT; 7s ago
  Process: 12995 ExecStart=/sbin/upsd (code=exited, status=0/SUCCESS)
 Main PID: 12996 (upsd)
    Tasks: 1 (limit: 877)
   CGroup: /system.slice/nut-server.service
           └─12996 /lib/nut/upsd

Aug 24 20:28:53 pihole systemd[1]: Starting Network UPS Tools - power devices information server...
Aug 24 20:28:53 pihole upsd[12995]: fopen /var/run/nut/upsd.pid: No such file or directory
Aug 24 20:28:53 pihole upsd[12995]: listening on 127.0.0.1 port 3493
Aug 24 20:28:53 pihole upsd[12995]: listening on ::1 port 3493
Aug 24 20:28:53 pihole upsd[12995]: listening on 127.0.0.1 port 3493
Aug 24 20:28:53 pihole upsd[12995]: Connected to UPS [homeups]: usbhid-ups-homeups
Aug 24 20:28:53 pihole upsd[12995]: listening on ::1 port 3493
Aug 24 20:28:53 pihole upsd[12995]: Connected to UPS [homeups]: usbhid-ups-homeups
Aug 24 20:28:53 pihole systemd[1]: Started Network UPS Tools - power devices information server.
Aug 24 20:28:53 pihole upsd[12996]: Startup successful


Query the UPS

$ upsc qnapups@localhost

$ upsc qnapups@192.168.1.52

battery.charge: 100
battery.charge.low: 10
...
ups.vendorid: 051d


Client Setup QNAP

From the QNAP UI, open the control panel and navigate to External Sevices → UPS

Select Remote UPS Slave and enter the IP address of the master.

Click apply. The UPS information should now be updated. See screenshots below.


From a shell connection on the QNAP server, you can verify connection via the following command: (not required)

$ upsc qnapups@192.168.1.52

battery.charge: 100
...
ups.vendorid: 051d


Client Setup (Ubuntu)

Installation

Install nut-client

$ sudo apt-get install nut-client


Configure

Update nut-config

$ vi /etc/nut/nut.conf


Set mode to netclient

MODE=netclient


Update upsmon.conf

$ vi /etc/nut/upsmon.conf


RUN_AS_USER root
MONITOR qnapups@192.168.1.52 1 deepthought 123456 slave
NOTIFYCMD /usr/sbin/upssched
NOTIFYFLAG ONLINE WALL+EXEC
NOTIFYFLAG ONBATT WALL+EXEC

On our master we created a slave user called deepthought.

  • upsname = qnapups
  • user = deepthought
  • password = 123456


Setup Timer to Shutdown after 1 Minute

Edit the upssched.conf file

$ sudo vi /etc/nut/upssched.conf


Set  the following

CMDSCRIPT /etc/nut/upssched-cmd
  
PIPEFN /var/run/nut/upssched.pipe
LOCKFN /var/run/nut/upssched.lock

# Timer to shutdown machine after 30 seconds
AT ONBATT * START-TIMER onbattwarn 30
AT ONLINE * CANCEL-TIMER onbattwarn


Create the Command Script

$ sudo vi /etc/nut/upssched-cmd


Add the following content

case $1 in
        onbattwarn)
                logger -t upssched-cmd "Timer On Battery Warning has been triggered - Shutting Down!"
                wall "UPS timer expired: shutting down..."
                shutdown -h now
                ;;
        *)
                logger -t upssched-cmd "Unrecognized command: $1"
                ;;
esac



Restart Nut Client

Restart nut client:

$ systemctl restart nut-client


Get Status of Nut Client

Get Status:

$ systemctl status nut-client


● nut-monitor.service - Network UPS Tools - power device monitor and shutdown controller
Loaded: loaded (/lib/systemd/system/nut-monitor.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2021-08-25 13:11:09 EDT; 2s ago
Process: 376582 ExecStart=/sbin/upsmon (code=exited, status=0/SUCCESS)
Main PID: 376585 (upsmon)
Tasks: 2 (limit: 19043)
Memory: 1.1M
CGroup: /system.slice/nut-monitor.service
├─376583 /lib/nut/upsmon
└─376585 /lib/nut/upsmon

Aug 25 13:11:09 deepthought systemd[1]: Starting Network UPS Tools - power device monitor and shutdown controller...
Aug 25 13:11:09 deepthought upsmon[376582]: fopen /run/nut/upsmon.pid: No such file or directory
Aug 25 13:11:09 deepthought upsmon[376582]: Using power down flag file /etc/killpower
Aug 25 13:11:09 deepthought upsmon[376582]: UPS: qnapups@192.168.1.52 (slave) (power value 1)
Aug 25 13:11:09 deepthought systemd[1]: nut-monitor.service: Can't open PID file /run/nut/upsmon.pid (yet?) after start: Operati>
Aug 25 13:11:09 deepthought upsmon[376583]: Startup successful
Aug 25 13:11:09 deepthought systemd[1]: nut-monitor.service: Supervising process 376585 which is not our child. We'll most likel>
Aug 25 13:11:09 deepthought systemd[1]: Started Network UPS Tools - power device monitor and shutdown controller.


Check connection 

Check Connection:

$ upsc qnapups@192.168.1.52

Installing NUT on Mac

> brew install nut

==> Downloading https://ghcr.io/v2/homebrew/core/nut/manifests/2.8.1
######################################################################### 100.0%
==> Fetching nut
==> Downloading https://ghcr.io/v2/homebrew/core/nut/blobs/sha256:e89241f392bb9f
######################################################################### 100.0%
==> Pouring nut--2.8.1.arm64_sonoma.bottle.tar.gz
Warning: The post-install step did not complete successfully
You can try again using:
  brew postinstall nut
==> Caveats
To start nut now and restart at login:
  brew services start nut
Or, if you don't want/need a background service you can just run:
  /opt/homebrew/opt/nut/sbin/upsmon -D
==> Summary
🍺  /opt/homebrew/Cellar/nut/2.8.1: 94 files, 12MB
==> Running `brew cleanup nut`...

Didn't work.....

Troubleshooting

ProblemDetails 
UPS Shutting Down

Sometimes the UPS shutdown for no reason.  No power outage.

cat /var/log/syslog |grep ups

Mar 16 02:09:28 pihole upsmon[390]: UPS qnapups@localhost on battery
Mar 16 02:09:28 pihole upsd[381]: Client upsmon@127.0.0.1 set FSD on UPS [qnapups]
Mar 16 02:09:36 pihole upsmon[390]: Executing automatic power-fail shutdown
Mar 16 02:09:36 pihole upsmon[390]: Auto logout and shutdown proceeding
Mar 16 03:10:07 pihole kernel: [    0.000000] ftrace: allocated 60 pages with 4 groups
Mar 16 03:10:07 pihole dbus-daemon[275]: [system] Activating systemd to hand-off: service name='org.freedesktop.ColorManager' unit='colord.service' requested by ':1.1' (uid=0 pid=252 comm="/usr/sbin/cupsd -l ")
Mar 16 03:10:10 pihole upsdrvctl[342]: Using subdriver: APC HID 0.96
Mar 16 03:10:10 pihole upsdrvctl[342]: Network UPS Tools - Generic HID driver 0.41 (2.7.4)
Mar 16 03:10:10 pihole upsdrvctl[342]: USB communication driver 0.33
Mar 16 03:10:10 pihole upsdrvctl[342]: Network UPS Tools - UPS driver controller 2.7.4
Mar 16 03:10:10 pihole usbhid-ups[368]: Startup successful
Mar 16 03:10:11 pihole upsd[374]: fopen /var/run/nut/upsd.pid: No such file or directory
Mar 16 03:10:11 pihole upsd[374]: listening on 0.0.0.0 port 3493
Mar 16 03:10:11 pihole upsd[374]: listening on 0.0.0.0 port 3493
Mar 16 03:10:11 pihole upsd[374]: Connected to UPS [qnapups]: usbhid-ups-qnapups
Mar 16 03:10:11 pihole upsd[374]: Connected to UPS [qnapups]: usbhid-ups-qnapups
Mar 16 03:10:12 pihole upsd[381]: Startup successful
Mar 16 03:10:14 pihole upsmon[389]: fopen /var/run/nut/upsmon.pid: No such file or directory
Mar 16 03:10:14 pihole upsmon[389]: UPS: qnapups@localhost (master) (power value 1)
Mar 16 03:10:14 pihole upsmon[391]: Startup successful
Mar 16 03:10:14 pihole systemd[1]: nut-monitor.service: Can't open PID file /run/nut/upsmon.pid (yet?) after start: No such file or directory
Mar 16 03:10:14 pihole upsmon[392]: Init SSL without certificate database
Mar 16 03:10:14 pihole upsd[381]: User upsmon@127.0.0.1 logged into UPS [qnapups]
Mar 16 03:10:14 pihole upsmon[392]: UPS qnapups@localhost battery is low


Unplugging the power and querying the UPS showed a charge of 0!

> upsc qnapups@localhost |grep battery


Init SSL without certificate database
battery.charge: 0
battery.charge.low: 10
battery.charge.warning: 50
battery.date: 2001/09/25
battery.mfr.date: 2010/09/05
battery.runtime: 0
battery.runtime.low: 120
battery.type: PbAc
battery.voltage: 13.4
battery.voltage.nominal: 12.0


Get NUT status

sudo service nut-server status
● nut-server.service - Network UPS Tools - power devices information server
   Loaded: loaded (/lib/systemd/system/nut-server.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2024-03-16 03:10:12 EDT; 6h ago
  Process: 374 ExecStart=/sbin/upsd (code=exited, status=0/SUCCESS)
 Main PID: 381 (upsd)
    Tasks: 1 (limit: 877)
   CGroup: /system.slice/nut-server.service
           └─381 /lib/nut/upsd

Mar 16 03:10:11 pihole upsd[374]: fopen /var/run/nut/upsd.pid: No such file or directory
Mar 16 03:10:11 pihole upsd[374]: listening on 0.0.0.0 port 3493
Mar 16 03:10:11 pihole upsd[374]: listening on 0.0.0.0 port 3493
Mar 16 03:10:11 pihole upsd[374]: Connected to UPS [qnapups]: usbhid-ups-qnapups
Mar 16 03:10:11 pihole upsd[374]: Connected to UPS [qnapups]: usbhid-ups-qnapups
Mar 16 03:10:12 pihole upsd[381]: Startup successful
Mar 16 03:10:12 pihole systemd[1]: Started Network UPS Tools - power devices information server.
Mar 16 03:10:14 pihole upsd[381]: User upsmon@127.0.0.1 logged into UPS [qnapups]
Mar 16 09:26:09 pihole upsd[381]: User upsmon@127.0.0.1 logged into UPS [qnapups]
Mar 16 09:26:20 pihole upsd[381]: User deepthought@192.168.1.50 logged into UPS [qnapups]

 

Querying UPS
upsc qnapups@localhost

$ upsc qnapups@192.168.1.52

battery.charge: 100
battery.charge.low: 10
...
ups.vendorid: 051d

References

  • No labels