1

In my company we have implemented a distributed cluster using the CitusData extension, the virtual machines we've using are located on a privative virtual server provider. About two or three weeks ago we have noticed that one of the nodes where extremely slow (when doing queries to the master node), and the cores at full capacity. So the first move where to check the processes load and this was what we got issuing (only a few entries shown):

top

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
20398 postgres  20   0  462040   5088    836 S  10.3  0.1  20:00.38 mule
23139 postgres  20   0  462040   5156    864 S  10.3  0.1 303:25.07 mule
 3361 postgres  20   0  462040   5056    784 S  10.0  0.1  52:28.02 mule
13159 postgres  20   0  462040   5076    772 S  10.0  0.1 113:03.39 mule
16580 postgres  20   0  396504   4792    796 S  10.0  0.1  27:28.25 mule
32372 postgres  20   0  265432   5144    920 S   9.6  0.1   0:03.29 mule
19135 postgres  20   0  462040   5120    840 S   9.3  0.1 423:10.94 mule
24734 postgres  20   0  462040   4984    876 S   9.3  0.1  73:49.95 mule
28094 postgres  20   0  330968   5144    904 S   9.3  0.1   6:20.76 mule
 9255 postgres  20   0  462040   4752    696 S   9.0  0.1 130:48.49 mule
.....
.....

ps aux | grep mule

postgres  1365  5.0  0.1 396504  4904 ?        Ssl  13:04   8:12 /tmp/mule
postgres  1896 11.8  0.1 462040  5028 ?        Ssl  Nov24 262:02 /tmp/mule
postgres  2377  5.6  0.1 462040  4988 ?        Ssl  04:03  39:43 /tmp/mule
postgres  3361  8.2  0.1 462040  5124 ?        Ssl  Nov24 141:37 /tmp/mule
postgres  3629  6.6  0.1 462040  4916 ?        Ssl  Nov24  82:51 /tmp/mule
postgres  4692  4.9  0.1 330968  4996 ?        Ssl  14:04   5:00 /tmp/mule
postgres  5577 11.1  0.1 462040  5044 ?        Ssl  Nov24 240:06 /tmp/mule
postgres  5697  5.5  0.1 462040  5092 ?        Ssl  05:03  35:19 /tmp/mule
postgres  7288  6.5  0.1 462040  5180 ?        Ssl  Nov24  77:43 /tmp/mule
postgres  8111  4.9  0.1 265432  4912 ?        Ssl  15:04   2:03 /tmp/mule
postgres  8675  7.9  0.1 462040  4892 ?        Ssl  Nov24 132:40 /tmp/mule
.....
.....

htop screenshot htop screenshot

We restarted that node and all went back to normal, but a few days later, the same ~160 mule processes where initiated by the postgres user. This happends only in one particular node. All of them are replicas (2cores and 8gb ram each) so all of them run postgresql-9.6.5 and CitusData_7.1 over CentOS 7.

We've done some research but with no luck and we can't find any docs or issues regarding these processes, only documentation about MuleSoft, but we never installed anything of such.

We also contacted the CitusData developers and they also have no idea what that process is. We've been using postgresql for quite a while but never seeing this. Is this any kind of malware ?. Feel free to ask for more details.

UPDATE

Fixed information: postgresql server version -> 9.6 to 9.6.5

select version();

                                             version
----------------------------------------------------------------------------------------------------------
PostgreSQL 9.6.5 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-11), 64-bit
  • 1
    I think Mulesoft must be used in your stack! What is the software that is accessing the database? Is it 3rd party? – Vérace Nov 27 '17 at 19:25
  • 1
    No it isn't. The cluster is ingested with data provided by some web services developed in Erlang. Those services are managed by us and aren't using anything related to Mulesoft. We don't use any 3rd party softwares. – Emiliano Nov 27 '17 at 19:54
  • Can you run /tmp/mule --version and/or --help from the command line? At the risk of sounding like a broken record, you could look here. – Vérace Nov 27 '17 at 21:22
  • The mule command does not exist in our environment. -bash: /tmp/mule: No such file or directory. Even when is running we can't find anything inside the /tmp folder, only we can see the output of the top or ps commands. – Emiliano Nov 28 '17 at 12:56
  • OK - so, just to humour me :-), try sudo find / -iname "*mule*" - the items in your processlist must be coming from somewhere! – Vérace Nov 28 '17 at 14:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.