Redis Sentinel upgrade fails when coming from 6.0.6 (chris lea repo)

[Apollon77]

I have Redis clusters rrunning in lxc containers and all working fine with @chrislea ppa

Now I tried to switch to the official because veryion in Chris's PPA is 6.0.6 still.

Upgrading a Redis only system works fine, Upgrading a system that also installs redis sentinel gives an error dirtectly on upgrade:

Job for redis-sentinel.service failed because the service did not take the steps required by its unit configuration.
See "systemctl status redis-sentinel.service" and "journalctl -xe" for details.
invoke-rc.d: initscript redis-sentinel, action "start" failed.
* redis-sentinel.service - Advanced key-value store
     Loaded: loaded (/lib/systemd/system/redis-sentinel.service; enabled; vendor preset: enabled)
     Active: activating (auto-restart) (Result: protocol) since Tue 2021-05-25 15:06:40 CEST; 5ms ago
       Docs: http://redis.io/documentation,
             man:redis-sentinel(1)
    Process: 2100 ExecStart=/usr/bin/redis-sentinel /etc/redis/sentinel.conf (code=exited, status=0/SUCCESS)
   Main PID: 2100 (code=exited, status=0/SUCCESS)
dpkg: error processing package redis-sentinel (--configure):
 installed redis-sentinel package post-installation script subprocess returned error exit status 1
Processing triggers for systemd (245.4-4ubuntu3.6) ...
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for ureadahead (0.100.0-21) ...
Processing triggers for libc-bin (2.31-0ubuntu9.2) ...
Errors were encountered while processing:
 redis-sentinel
E: Sub-process /usr/bin/dpkg returned an error code (1)

The redis-sentinal log actually shows:

148:signal-handler (1621947997) Received SIGTERM scheduling shutdown...
148:signal-handler (1621947997) Received SIGTERM scheduling shutdown...
148:X 25 May 2021 15:06:37.824 # User requested shutdown...
148:X 25 May 2021 15:06:37.824 * Removing the pid file.
148:X 25 May 2021 15:06:37.824 # Sentinel is now ready to exit, bye bye...
2101:X 25 May 2021 15:06:40.603 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
2101:X 25 May 2021 15:06:40.603 # Redis version=6.2.3, bits=64, commit=00000000, modified=0, pid=2101, just started
2101:X 25 May 2021 15:06:40.603 # Configuration loaded
2101:signal-handler (1621948000) Received SIGTERM scheduling shutdown...
2101:X 25 May 2021 15:06:40.604 * monotonic clock: POSIX clock_gettime
2101:X 25 May 2021 15:06:40.605 * Running mode=sentinel, port=26379.
2101:X 25 May 2021 15:06:40.605 # Sentinel ID is 3339e57713bd9d0acb32df97e59b1886ca3f6951
2101:X 25 May 2021 15:06:40.605 # +monitor master states-master 192.168.178.128 6380 quorum 2
2101:X 25 May 2021 15:06:40.605 # +monitor master objects-master 192.168.178.129 6379 quorum 2
2101:X 25 May 2021 15:06:40.606 # User requested shutdown...
2101:X 25 May 2021 15:06:40.606 * Removing the pid file.
2101:X 25 May 2021 15:06:40.606 # Sentinel is now ready to exit, bye bye...
2113:X 25 May 2021 15:06:40.827 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
2113:X 25 May 2021 15:06:40.827 # Redis version=6.2.3, bits=64, commit=00000000, modified=0, pid=2113, just started
2113:X 25 May 2021 15:06:40.827 # Configuration loaded
2113:X 25 May 2021 15:06:40.828 * monotonic clock: POSIX clock_gettime
2113:X 25 May 2021 15:06:40.829 * Running mode=sentinel, port=26379.
2113:X 25 May 2021 15:06:40.829 # Sentinel ID is 3339e57713bd9d0acb32df97e59b1886ca3f6951
2113:X 25 May 2021 15:06:40.829 # +monitor master states-master 192.168.178.128 6380 quorum 2
2113:X 25 May 2021 15:06:40.829 # +monitor master objects-master 192.168.178.129 6379 quorum 2
2113:signal-handler (1621948090) Received SIGTERM scheduling shutdown...
...

The syslog log shows:

root@redistest1:~# systemctl status redis-sentinel.service
* redis-sentinel.service - Advanced key-value store
     Loaded: loaded (/lib/systemd/system/redis-sentinel.service; enabled; vendor preset: enabled)
     Active: failed (Result: protocol) since Tue 2021-05-25 15:08:12 CEST; 2min 50s ago
       Docs: http://redis.io/documentation,
             man:redis-sentinel(1)
    Process: 3714 ExecStart=/usr/bin/redis-sentinel /etc/redis/sentinel.conf (code=exited, status=0/SUCCESS)
   Main PID: 3714 (code=exited, status=0/SUCCESS)

May 25 15:08:12 redistest1 systemd[1]: redis-sentinel.service: Scheduled restart job, restart counter is at 7.
May 25 15:08:12 redistest1 systemd[1]: Stopped Advanced key-value store.
May 25 15:08:12 redistest1 systemd[1]: redis-sentinel.service: Start request repeated too quickly.
May 25 15:08:12 redistest1 systemd[1]: redis-sentinel.service: Failed with result 'protocol'.
May 25 15:08:12 redistest1 systemd[1]: Failed to start Advanced key-value store.

I tested on two different systems.

And it seems I'm not alone with this: https://stackoverflow.com/questions/67565522/redis-sentinel-not-starting

[yossigo]

@Apollon77 Does sentinel manage to start properly on a fresh install?
Can you provide the /lib/systemd/system/redis-sentinel.service configuration?

[Apollon77]

Ok, I installed the 6.2.4 on a new system and here the systemd file looks like

[Unit]
Description=Advanced key-value store
After=network.target
Documentation=http://redis.io/documentation, man:redis-sentinel(1)

[Service]
Type=notify
ExecStart=/usr/bin/redis-sentinel /etc/redis/sentinel.conf
ExecStop=/bin/kill -s TERM $MAINPID
PIDFile=/run/sentinel/redis-sentinel.pid
TimeoutStopSec=0
Restart=always
User=redis
Group=redis
RuntimeDirectory=sentinel
RuntimeDirectoryMode=2755

UMask=007
PrivateTmp=yes
LimitNOFILE=65535
PrivateDevices=yes
ProtectHome=yes
ReadOnlyDirectories=/
ReadWriteDirectories=-/var/lib/redis
ReadWriteDirectories=-/var/log/redis
ReadWriteDirectories=-/run/sentinel

NoNewPrivileges=true
CapabilityBoundingSet=CAP_SETGID CAP_SETUID CAP_SYS_RESOURCE
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
MemoryDenyWriteExecute=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectControlGroups=true
RestrictRealtime=true
RestrictNamespaces=true

# redis-sentinel can write to its own config file when in cluster mode so we
# permit writing there by default. If you are not using this feature, it is
# recommended that you replace the following lines with "ProtectSystem=full".
ProtectSystem=true
ReadWriteDirectories=-/etc/redis

[Install]
WantedBy=multi-user.target
Alias=sentinel.service

And the servce starts (with the default config):

1607:X 03 Jun 2021 22:38:00.663 * Supervised by systemd. Please make sure you set appropriate values for TimeoutStartSec and TimeoutStopSec in your service unit.
1607:X 03 Jun 2021 22:38:00.663 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1607:X 03 Jun 2021 22:38:00.663 # Redis version=6.2.4, bits=64, commit=00000000, modified=0, pid=1607, just started
1607:X 03 Jun 2021 22:38:00.663 # Configuration loaded
1607:X 03 Jun 2021 22:38:00.664 * monotonic clock: POSIX clock_gettime
1607:X 03 Jun 2021 22:38:00.664 * Running mode=sentinel, port=26379.
1607:X 03 Jun 2021 22:38:00.667 # Sentinel ID is 560008b666b596c08c40ae4cb7a99b9cb17fb5bd
1607:X 03 Jun 2021 22:38:00.667 # +monitor master mymaster 127.0.0.1 6379 quorum 2

So basically seems to work on a new fresh system.

On the system with the "old" version the systemd file (reverted to chris lea version):

[Unit]
Description=Advanced key-value store
After=network.target
Documentation=http://redis.io/documentation, man:redis-sentinel(1)

[Service]
Type=forking
ExecStart=/usr/bin/redis-sentinel /etc/redis/sentinel.conf
ExecStop=/bin/kill -s TERM $MAINPID
PIDFile=/var/run/sentinel/redis-sentinel.pid
TimeoutStopSec=0
Restart=always
User=redis
Group=redis
RuntimeDirectory=sentinel
RuntimeDirectoryMode=2755

UMask=007
PrivateTmp=yes
LimitNOFILE=65535
PrivateDevices=yes
ProtectHome=yes
ReadOnlyDirectories=/
ReadWriteDirectories=-/var/lib/redis
ReadWriteDirectories=-/var/log/redis
ReadWriteDirectories=-/var/run/sentinel

NoNewPrivileges=true
CapabilityBoundingSet=CAP_SETGID CAP_SETUID CAP_SYS_RESOURCE
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
MemoryDenyWriteExecute=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectControlGroups=true
RestrictRealtime=true
RestrictNamespaces=true

# redis-sentinel can write to its own config file when in cluster mode so we
# permit writing there by default. If you are not using this feature, it is
# recommended that you replace the following lines with "ProtectSystem=full".
ProtectSystem=true
ReadWriteDirectories=-/etc/redis

[Install]
WantedBy=multi-user.target
Alias=sentinel.service

Maybe the redis config also helps?

bind 0.0.0.0
#
protected-mode no

# port <sentinel-port>
# The port that this sentinel instance will run on
port 26379

# By default Redis Sentinel does not run as a daemon. Use 'yes' if you need it.
# Note that Redis will write a pid file in /var/run/redis-sentinel.pid when
# daemonized.
daemonize yes

# When running daemonized, Redis Sentinel writes a pid file in
# /var/run/redis-sentinel.pid by default. You can specify a custom pid file
# location here.
pidfile "/var/run/sentinel/redis-sentinel.pid"

# Specify the log file name. Also the empty string can be used to force
# Sentinel to log on the standard output. Note that if you use standard
# output for logging but daemonize, logs will be sent to /dev/null
logfile "/var/log/redis/redis-sentinel.log"

# sentinel announce-ip <ip>
# sentinel announce-port <port>
#
# The above two configuration directives are useful in environments where,
# because of NAT, Sentinel is reachable from outside via a non-local address.
#
# When announce-ip is provided, the Sentinel will claim the specified IP address
# in HELLO messages used to gossip its presence, instead of auto-detecting the
# local address as it usually does.
#
# Similarly when announce-port is provided and is valid and non-zero, Sentinel
# will announce the specified TCP port.
#
# The two options don't need to be used together, if only announce-ip is
# provided, the Sentinel will announce the specified IP and the server port
# as specified by the "port" option. If only announce-port is provided, the
# Sentinel will announce the auto-detected local IP and the specified port.
#
# Example:
#
# sentinel announce-ip 1.2.3.4

# dir <working-directory>
# Every long running process should have a well-defined working directory.
# For Redis Sentinel to chdir to /tmp at startup is the simplest thing
# for the process to don't interfere with administrative tasks such as
# unmounting filesystems.
dir "/var/lib/redis"

# sentinel monitor <master-name> <ip> <redis-port> <quorum>
#
# Tells Sentinel to monitor this master, and to consider it in O_DOWN
# (Objectively Down) state only if at least <quorum> sentinels agree.
#
# Note that whatever is the ODOWN quorum, a Sentinel will require to
# be elected by the majority of the known Sentinels in order to
# start a failover, so no failover can be performed in minority.
#
# Replicas are auto-discovered, so you don't need to specify replicas in
# any way. Sentinel itself will rewrite this configuration file adding
# the replicas using additional configuration options.
# Also note that the configuration file is rewritten when a
# replica is promoted to master.
#
# Note: master name should not include special characters or spaces.
# The valid charset is A-z 0-9 and the three characters ".-_".
sentinel myid 3339e57713bd9d0acb32df97e59b1886ca3f6951

# sentinel auth-pass <master-name> <password>
#
# Set the password to use to authenticate with the master and replicas.
# Useful if there is a password set in the Redis instances to monitor.
#
# Note that the master password is also used for replicas, so it is not
# possible to set a different password in masters and replicas instances
# if you want to be able to monitor these instances with Sentinel.
#
# However you can have Redis instances without the authentication enabled
# mixed with Redis instances requiring the authentication (as long as the
# password set is the same for all the instances requiring the password) as
# the AUTH command will have no effect in Redis instances with authentication
# switched off.
#
# Example:
#
# sentinel auth-pass mymaster MySUPER--secret-0123passw0rd

# sentinel down-after-milliseconds <master-name> <milliseconds>
#
# Number of milliseconds the master (or any attached replica or sentinel) should
# be unreachable (as in, not acceptable reply to PING, continuously, for the
# specified period) in order to consider it in S_DOWN state (Subjectively
# Down).
#
# Default is 30 seconds.
sentinel deny-scripts-reconfig yes

# sentinel parallel-syncs <master-name> <numreplicas>
#
# How many replicas we can reconfigure to point to the new replica simultaneously
# during the failover. Use a low number if you use the replicas to serve query
# to avoid that all the replicas will be unreachable at about the same
# time while performing the synchronization with the master.
sentinel monitor states-master 192.168.178.128 6380 2
sentinel failover-timeout states-master 20000

# sentinel failover-timeout <master-name> <milliseconds>
#
# Specifies the failover timeout in milliseconds. It is used in many ways:
#
# - The time needed to re-start a failover after a previous failover was
#   already tried against the same master by a given Sentinel, is two
#   times the failover timeout.
#
# - The time needed for a replica replicating to a wrong master according
#   to a Sentinel current configuration, to be forced to replicate
#   with the right master, is exactly the failover timeout (counting since
#   the moment a Sentinel detected the misconfiguration).
#
# - The time needed to cancel a failover that is already in progress but
#   did not produced any configuration change (SLAVEOF NO ONE yet not
#   acknowledged by the promoted replica).
#
# - The maximum time a failover in progress waits for all the replicas to be
#   reconfigured as replicas of the new master. However even after this time
#   the replicas will be reconfigured by the Sentinels anyway, but not with
#   the exact parallel-syncs progression as specified.
#
# Default is 3 minutes.
#sentinel config-epoch mymaster 0

# SCRIPTS EXECUTION
#
# sentinel notification-script and sentinel reconfig-script are used in order
# to configure scripts that are called to notify the system administrator
# or to reconfigure clients after a failover. The scripts are executed
# with the following rules for error handling:
#
# If script exits with "1" the execution is retried later (up to a maximum
# number of times currently set to 10).
#
# If script exits with "2" (or an higher value) the script execution is
# not retried.
#
# If script terminates because it receives a signal the behavior is the same
# as exit code 1.
#
# A script has a maximum running time of 60 seconds. After this limit is
# reached the script is terminated with a SIGKILL and the execution retried.

# NOTIFICATION SCRIPT
#
# sentinel notification-script <master-name> <script-path>
#
# Call the specified notification script for any sentinel event that is
# generated in the WARNING level (for instance -sdown, -odown, and so forth).
# This script should notify the system administrator via email, SMS, or any
# other messaging system, that there is something wrong with the monitored
# Redis systems.
#
# The script is called with just two arguments: the first is the event type
# and the second the event description.
#
# The script must exist and be executable in order for sentinel to start if
# this option is provided.
#
# Example:
#
# sentinel notification-script mymaster /var/redis/notify.sh

# CLIENTS RECONFIGURATION SCRIPT
#
# sentinel client-reconfig-script <master-name> <script-path>
#
# When the master changed because of a failover a script can be called in
# order to perform application-specific tasks to notify the clients that the
# configuration has changed and the master is at a different address.
#
# The following arguments are passed to the script:
#
# <master-name> <role> <state> <from-ip> <from-port> <to-ip> <to-port>
#
# <state> is currently always "failover"
# <role> is either "leader" or "observer"
#
# The arguments from-ip, from-port, to-ip, to-port are used to communicate
# the old address of the master and the new address of the elected replica
# (now a master).
#
# This script should be resistant to multiple invocations.
#
# Example:
#
# sentinel client-reconfig-script mymaster /var/redis/reconfig.sh

# SECURITY
#
# By default SENTINEL SET will not be able to change the notification-script
# and client-reconfig-script at runtime. This avoids a trivial security issue
# where clients can set the script to anything and trigger a failover in order
# to get the program executed.

#sentinel leader-epoch mymaster 0

# REDIS COMMANDS RENAMING
#
# Sometimes the Redis server has certain commands, that are needed for Sentinel
# to work correctly, renamed to unguessable strings. This is often the case
# of CONFIG and SLAVEOF in the context of providers that provide Redis as
# a service, and don't want the customers to reconfigure the instances outside
# of the administration console.
#
# In such case it is possible to tell Sentinel to use different command names
# instead of the normal ones. For example if the master "mymaster", and the
# associated replicas, have "CONFIG" all renamed to "GUESSME", I could use:
#
# SENTINEL rename-command mymaster CONFIG GUESSME
#
# After such configuration is set, every time Sentinel would use CONFIG it will
# use GUESSME instead. Note that there is no actual need to respect the command
# case, so writing "config guessme" is the same in the example above.
#
# SENTINEL SET can also be used in order to perform this configuration at runtime.
#
# In order to set a command back to its original name (undo the renaming), it
# is possible to just rename a command to itsef:
#
# SENTINEL rename-command mymaster CONFIG CONFIG

# To enable logging to the system logger, just set 'syslog-enabled' to yes,
# and optionally update the other syslog parameters to suit your needs.
syslog-enabled yes

# Specify the syslog identity.
syslog-ident "redis-sentinel-test"

# Specify the syslog facility. Must be USER or between LOCAL0-LOCAL7.
syslog-facility local0

# Generated by CONFIG REWRITE
maxclients 4064

sentinel config-epoch states-master 12
sentinel leader-epoch states-master 12
sentinel known-replica states-master 192.168.178.119 6380
sentinel known-replica states-master 192.168.178.129 6380
sentinel known-sentinel states-master 192.168.178.129 26379 3339e57713bd9d0acb32df97e59b1886ca3f6953
sentinel known-sentinel states-master 192.168.178.128 26379 3339e57713bd9d0acb32df97e59b1886ca3f6952
sentinel monitor objects-master 192.168.178.129 6379 2
sentinel failover-timeout objects-master 20000
sentinel config-epoch objects-master 13
sentinel leader-epoch objects-master 13
sentinel known-replica objects-master 192.168.178.128 6379
sentinel known-replica objects-master 192.168.178.119 6379
sentinel known-sentinel objects-master 192.168.178.129 26379 3339e57713bd9d0acb32df97e59b1886ca3f6953
sentinel known-sentinel objects-master 192.168.178.128 26379 3339e57713bd9d0acb32df97e59b1886ca3f6952
sentinel current-epoch 13
user default on nopass ~* +@all

[yossigo]

@Apollon77 It seems like the systemd unit file was not updated during the upgrade, as the new version uses Type=notify and the old version uses Type=forking. Do you see that happening only for the redis-sentinel service but not for redis-server?

AFAIR debhelper should take care of replacing this file on upgrade, so it's not clear why this does not happen. I can try to spin up a VM and reproduce this when I have some bandwidth - I guess all it takes is to add the chrislea PPA, install, then add the new PPA and upgrade. Is that correct?

[Apollon77]

I guess all it takes is to add the chrislea PPA, install, then add the new PPA and upgrade. Is that correct?

yes

I could also try an upgrade on a test system tomorrow to check if this is really the case. Could it be that systemd is confused because of that change when upgrading when sentinel is running in parallel?