Protect your dangerously insecure redis server

Protect your dangerously insecure redis server

If you’ve put Redis on the internet you’ve probably had your box hacked one way or another. Unfortunately, the service has very weak defaults with no authentication, encryption, or meaningful access control. While it’s true that redis is a back-end service that should only be used between servers, it’s often misused and abused.

For example, there are about 40,000 redis instances indexed on shodan.io, about half of which are not in protected mode.

Obviously, the best thing to do is not expose a backend service directly to the net, but if you can’t do that you can at least slow down the hackers a bit.

After a few days of exposing 6379/tcp, I started to get some log entries like this:

1:S 12 May 2022 15:05:14.873 * Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
1:S 12 May 2022 15:05:14.873 * Connecting to MASTER x.x.x.x:8886
1:S 12 May 2022 15:05:14.873 * MASTER <-> REPLICA sync started
1:S 12 May 2022 15:05:14.873 * REPLICAOF x.x.x.x:8886 enabled (user request from 'id=9 addr=y.y.y.y:60594 laddr=172.22.0.6:6379 fd=9 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=46 qbuf-free=20428 argv-mem=23 multi-mem=0 rbs=16384 rbp=0 obl=0 oll=0 omem=0 tot-mem=37679 events=r cmd=slaveof user=default redir=-1 resp=2')
1:S 12 May 2022 15:05:14.969 * Non blocking connect for SYNC fired the event.
1:S 12 May 2022 15:05:15.064 * Master replied to PING, replication can continue...
1:S 12 May 2022 15:05:15.253 * Trying a partial resynchronization (request 0e5a00d55e246642b7e09fd1e39e0ed79de7b4a2:1).
1:S 12 May 2022 15:05:15.353 * Full resync from master: ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ:1
1:S 12 May 2022 15:05:15.353 * MASTER <-> REPLICA sync: receiving 54992 bytes from master to disk
1:S 12 May 2022 15:05:15.552 * Discarding previously cached master state.
1:S 12 May 2022 15:05:15.552 * MASTER <-> REPLICA sync: Flushing old data
1:S 12 May 2022 15:05:15.552 * MASTER <-> REPLICA sync: Loading DB in memory
1:S 12 May 2022 15:05:15.554 # Wrong signature trying to load DB from file
1:S 12 May 2022 15:05:15.554 # Failed trying to load the MASTER synchronization DB from disk: Invalid argument
1:S 12 May 2022 15:05:15.554 * Reconnecting to MASTER x.x.x.x:8886 after failure
1:S 12 May 2022 15:05:15.554 * MASTER <-> REPLICA sync started
1:S 12 May 2022 15:05:15.648 * Non blocking connect for SYNC fired the event.
1:S 12 May 2022 15:05:15.741 # Error reply to PING from master: '-Reading from master: Connection reset by peer'

After this, there was a burst of activity, then the redis service would seize up. This was on a Docker container, so it was trivial to kill and restart the service. But, still annoying.

I should also note that despite an attempt to install a crypto miner, it was unsuccessful.

CONFIG – a file write exploit

First, I suspected it was a vulnerability that was causing this. Our redis was already on the very latest version, and nobody would be foolish enough to burn a very expensive zeroday on my stupid little server.

Upon digging a bit more, it appears to be an old and played out infection. TrendMicro did a great writeup on these exploits. In a nutshell, they use the ‘CONFIG’ command to write arbitrary files, then attempt to run them using cron and other schedulers.

Looking a little deeper, my container had a new executable in the ~/data folder in redis, red2.so.

See VirusTotal analysis for more…

I wasn’t able to surmise what this executable does beyond creating a crude reverse shell. Nevertheless, it’s absolutely malware! Wear gloves and handle with care.

REPLICAOF – an infoleak exploit

The crashloop I experienced was actually the server attempting to become a replica for a non-existant primary system. What the attacker was doing was actually commanding my server to send a full copy of its data to a primary, then sync and become a secondary. Of course they had no intention to helpfully replicate this data, just to have a copy of the keys & values for nefarious purposes.

This also had the effect of hanging up the system, since the ‘initial sync’ was unable to complete successfully.

Hardening the server to prevent exploit

To prevent infections, the correct solution was to just disallow these dangerous commands by all connected clients.

rename-command FLUSHALL "" 
rename-command MIGRATE ""
rename-command RESTORE ""
rename-command SORT ""
rename-command KEYS ""
rename-command CLIENT ""
rename-command DEBUG ""
rename-command CONFIG ""
rename-command SAVE ""
rename-command REPLICAOF ""
rename-command SLAVEOF ""

This prevents both attack vectors, both the CONFIG file write issue, and the REPLICAOF infoleak exploit.

Of course, you should evaluate your application to ensure this won’t break any functionality. It’s possible to accidentally disallow a command that is used, and cause problems with app functionality.

To go further, it should be said that any non-containerized redis install should be hardened. Not just using redis-native auth, but also the installation itself. Systemd sandboxing can restrict access to filesystem paths, cgroups and namespaces can place restrictions on various system resources, and good old fashioned filesystem ACLs do a great job of keeping the blast radius to a minimum.