5

I'm using a Postgres container to run some small non-critical apps and sites. It's been stable for a while, but now the container has started to consume some serious CPU after it's been running for a short period of time. I have removed all other containers which use the Postgres container, and even after starting a new instance, the excessive CPU utilisation reoccurs. In my host (docker stats), I see this:

CONTAINER ID        NAME                                              
CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS cd553249727d        data_postgresql.1.ft2gof5jci25xs5w5uqw6eezi                
814.52%             22.11MiB / 46.95GiB   0.05%               129kB / 116kB       0B / 692kB          23

And this (top):

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
28923 70        20   0  633580  19664    488 S 696.7  0.0   2408:51 Dp2N

In the container (top), I see this:

Mem: 42042244K used, 7183656K free, 3622600K shrd, 1952K buff, 30585480K cached
CPU:  63% usr   9% sys   0% nic  26% idle   0% io   0% irq   0% sirq
Load average: 9.77 9.70 9.66 13/508 11090
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
   94     1 postgres S     618m   1%   3  58% ./Dp2N    <----- WTF?!?!?
   53    52 postgres S     1588   0%   1   1% {systemd} /bin/sh ./systemd
   47     1 postgres S     163m   0%   8   0% postgres: postgrea67 postgres 10.2
   22     1 postgres S     161m   0%   0   0% postgres: autovacuum launcher proc
   20     1 postgres S     161m   0%   8   0% postgres: writer process
   21     1 postgres S     161m   0%   5   0% postgres: wal writer process
    1     0 postgres S     161m   0%   0   0% postgres
   19     1 postgres S     161m   0%   8   0% postgres: checkpointer process
   23     1 postgres S    19988   0%   1   0% postgres: stats collector process
11081    53 postgres R     1588   0%   4   0% [systemd]
   33     0 root     S     1576   0%   9   0% sh
   52    47 postgres S     1568   0%  10   0% sh -c setsid ./systemd
   39    33 root     R     1508   0%  11   0% top
11083 11081 postgres Z        0   0%   5   0% [grep]
11084 11081 postgres Z        0   0%   4   0% [awk]

Query activity (no idea what select fun308928987('setsid ./systemd') does):

postgres=# select backend_start, usename, application_name, client_addr, client_hostname, query from pg_stat_activity;
         backend_start         |  usename   | application_name | client_addr | client_hostname |                                                    query
-------------------------------+------------+------------------+-------------+-----------------+-------------------------------------------------------------------------------------------------------------
 2018-05-23 07:34:14.694057+00 | postgres   | psql             |             |                 | select backend_start, usename, application_name, client_addr, client_hostname, query from pg_stat_activity;
 2018-05-23 01:26:55.235556+00 | postgrea67 |                  | 10.255.0.2  |                 | select fun308928987('setsid ./systemd');
 2018-05-23 07:26:03.519231+00 | postgrea67 |                  | 10.255.0.2  |                 | select fun308928987('setsid ./systemd');

In the service logs there are also a large amount of instances of this error:

data_postgresql.1.ft2gof5jci25@IS-57436    | ps: bad -o argument 'command', supported arguments: user,group,comm,args,pid,ppid,pgid,etime,nice,rgroup,ruser,time,tty,vsz,stat,rss

If I kill the Dp2N process within the container, CPU usage returns to normal, but then something immediately spins that process back up. I have googled to see if I can find any info on Dp2N, but to no avail. It's located in an externally mounted volume:

/ # ls -al /var/lib/postgresql/data/pgdata/Dp2N
-rwxrwxrwx    1 postgres postgres   1886536 May 22 23:25 /var/lib/postgresql/data/pgdata/Dp2N

but is seemingly created as it's not part of the base image as far as I can see.

I'm using postgres:9.6.9-alpine. The problem started with postgres:9.6.8-alpine, but upgrading didn't fix it. Any help would be greatly appreciated as this is driving me nuts!

Additional details

Results of running file:

sudo file /var/data/pgdata/pgdata/Dp2N
/var/data/pgdata/pgdata/Dp2N: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, for GNU/Linux 2.6.24, BuildID[sha1]=bcb5ccf2bc22d1fcb0676506d7c7f31a9b7148bc, stripped

It turns out that Alpine comes with a limited version of the ps command. Running this:

apk --no-cache add procps

gets the enhanced version and prevents the ps related error in the logs. I've updated the Postgres image to include this, and so far the problem hasn't resurfaced. Speculation is that the CPU is being thrashed trying to execute the command repeatedly after failure.

Diagnosis

As per the answer below, it turns out I've been hacked. I'm currently at a loss to how they got in though. Server locked down to specific user with SSH cert / no password access and root disabled. ('last' only shows my accesses - unless it's been hacked.) No public access to postgresql. Very strong database admin password. Only accessed from 1 other container currently. Seems likely that they got in via the web sites on the server but only got as far as the container operating system in this case, not the host OS. FWIW I'm running a Wordpress site, Grafana, Kibana, Traefik, Portainer and my own .NET based API. I'm starting off with a Wordpress shakedown first, as I've experienced plug-in related infections with it before.

For educational purposes:

https://www.imperva.com/blog/2018/03/deep-dive-database-attacks-scarlett-johanssons-picture-used-for-crypto-mining-on-postgre-database/

  • What does "file /var/lib/postgresql/data/pgdata/Dp2N" give? It's started by systemd, as far as I care to see. Look for it in the systemd services? – Gerard H. Pille May 23 '18 at 7:37
  • Hi Gerard, thanks for your swift response. I've added the requested info to the question. Still investigating as systemd seems a likely candidate for starting / restarting processes. It look as though running 'apk --no-cache add procps' should fix the 'ps' related logs issue as alpine comes with a basic version of the command. – vipes May 23 '18 at 8:34
  • Systemd is also a likely candidate for other things. What is an executable doing in that directory??? – Gerard H. Pille May 23 '18 at 8:39
  • No idea what it's doing there, it gets created on the fly. First hypothesis: CPU is being thrashed as for some reason, the container is continually executing the 'ps' command and erroring. So restarted the container and immediately ran the apk command to get the updated version of ps. Quiet so far - we'll see. – vipes May 23 '18 at 8:51
  • Why "file /var/data/pgdata/pgdata/Dp2N" when I asked for "file /var/lib/postgresql/data/pgdata/Dp2N"? cfr your "ls -al /var/lib/postgresql/data/pgdata/Dp2N" – Gerard H. Pille May 23 '18 at 8:59
3

You have been hacked, and are now mining cryptocurrency for the hacker.

They got in by guessing the password for your postgresql server's super-user account. Then they used the lo_export facility to drop the binary for a user-defined-function which executes arbitrary shell commands. That is what fun308928987 is, the SQL function which was created to wrap this binary.

Best clean up is to just destroy the server and rebuild it, this time setting up an actual strong password for the superuser account. Or better yet, also change pg_hba.conf to not allow super users connections, or preferably any connections, from the outside world.

  • Hi, thanks for this. I'm currently at a loss to how they got in though. Server locked down to specific user with SSH cert / no password access and root disabled. ('last' only shows my accesses - unless it's been hacked.) No public access to postgresql. Very strong admin password. Only accessed from 1 other container currently. There are around 5 web entrypoints on the server, so I guess that's where they came in. Will have to do some more digging to see how I can prevent this in future. I think they only got as far as the container operating system in this case, not the host os. – vipes May 23 '18 at 19:10
  • The password I'm talking about is the database user password, not the OS user. last wouldn't help you, but log_connections = on would (for the next time). Are you sure your settings in pg_hba are locked down? Your pg_stat_activity does show the connection coming from a private address, 10.255.0.2, so maybe they compromised that machine and then compromised PostgreSQL service from there. – jjanes May 23 '18 at 22:25
  • I was using 'last' just to see if they'd managed to gain login access to the server - the strong admin password was a strong database password. Suggestions appreciated though, I will continue looking. – vipes May 24 '18 at 11:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.