0

We've been experiencing huge CPU usage spikes on our Postgres 9.6 AWS RDS so the point we've had to upgrade to a cost prohibitive instance just to handle these once daily spikes. This spike always occurs at 23:30 UTC and lasts for about 30 minutes before tapering off to half the intensity, then are gone completely by 00:30.

After looking thru the enhanced monitoring logs at 1 second granularity, I can see that at 22:30:03 there is suddenly an entry for OS processes in the processList that did not exist before.

I cannot tell any information about what this mystery process is, however, I see that it immediately requires +33% of the overall cpuUsedPc.

No slow queries show up when I watch this output:

SELECT pid, age(query_start, clock_timestamp()), usename, query
FROM pg_stat_activity 
WHERE query != '<IDLE>' AND query NOT ILIKE '%pg_stat_activity%' AND query not ilike '%DEALLOCATE%' AND query not ilike '%COMMIT%'
ORDER BY query_start desc;

There are no crons running at that time.

There are no backups running until 02:00.

What could this be or how can I diagnose what is causing it?

Example:

    {
        "vss": 0,
        "name": "OS processes",
        "tgid": 0,
        "vmlimit": "",
        "parentID": 0,
        "memoryUsedPc": 0,
        "cpuUsedPc": 32.22,
        "id": 0,
        "rss": 0
    }
  • Did you try to contact Amazon support? With shell access I would probably turn on process accounting or simply have a cron job record ps snapshots (or run some top command in continuous log mode) – eckes Mar 24 '18 at 4:49
  • How do you know there's no crons running? How does the vacuuming start if not by cron? – Gerard H. Pille Mar 24 '18 at 16:06
  • I contacted AWS Business Support and they can't tell exactly what is causing it. I know there are crons running, but none are set for that time period. We generally do not rely on a manual vac and instead have our auto-vac fined tuned for each table. – eComEvo Mar 24 '18 at 19:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.