42

Is there a systematic way to force PostgreSQL to load a specific table into memory, or at least read it from disk so that it will be cached by the system?

33

You may be interessted in one of the mailing lists topics, it's answerd by Tom Lane (core dev):

[..] But my opinion is that people who think they are smarter than an LRU caching algorithm are typically mistaken. If the table is all that heavily used, it will stay in memory just fine. If it's not sufficiently heavily used to stay in memory according to an LRU algorithm, maybe the memory space really should be spent on something else. [..]

You might also be interessted in an SO question: https://stackoverflow.com/questions/486154/postgresql-temporary-tables and maybe more suiteable https://stackoverflow.com/questions/407006/need-to-load-the-whole-postgresql-database-into-the-ram

1
  • 28
    Yes and no. We lock some Oracle tables in-memory because we know that they might not be used that often, but in the situation they are used, latency will be a killer. A DB should always give the DBA final say (another example is hinting the query optimizer).
    – Gaius
    Apr 4 '11 at 9:17
45

Postgres 9.4 finally added an extension to preload data from relations into the OS or database buffer cache (at your choice):

pg_prewarm

This allows reaching full operating performance more quickly.

Run once in your database (detailed instructions here):

CREATE EXTENSION pg_prewarm;

Then it's simple to preload any given relation. Basic example:

SELECT pg_prewarm('my_tbl');

Finds the first table named my_tbl in the search path and loads it to the Postgres buffer cache.

Or:

SELECT pg_prewarm('my_schema.my_tbl', 'prefetch');

prefetch issues asynchronous prefetch requests to the operating system, if this is supported, or throws an error otherwise. read reads the requested range of blocks; unlike prefetch, this is synchronous and supported on all platforms and builds, but may be slower. buffer reads the requested range of blocks into the database buffer cache.

The default is buffer, which has the greatest impact (higher cost, best effect).

Read the manual for more details.
Depesz blogged about it, too.

3
  • If your table is quite large, how does one go about determining the correct block numbers? For instance, we have a content table and we want to preload the last 24 hours only. How could I programmatically do it with pg_prewarm? We're on AWS Aurora PostgreSQL Serverless, pg 10.14ish.
    – Brandon
    Jul 1 '21 at 17:06
  • @Brandon: Start a new question, please. You can always link to this answer for context. And drop a line here to link back and get my attention. Jul 1 '21 at 22:22
5

In the general case if you have enough RAM you can generally trust the database service to do a good job of keeping the things you regularly use in RAM. Some systems allow you to hint that the table should always be held in RAM (which is useful for smallish tables that are not used often but when they are used it is important that they respond as quickly as possible) but if pgsql has such table hints you need to be very careful about using them as you are reducing the amount of memory available for caching anything else so you might slow down your application overall.

If you are looking to prime the database's page cache on startup (for instance after a reboot or other maintainence operation that causes the DB to forget everything that is cached) then write a script that does the following:

SELECT * FROM <table>
SELECT <primary key fields> FROM <table> ORDER BY <primary key fields>
SELECT <indexed fields> FROM <table> ORDER BY <indexed fields>

(that last step repeated for each index, or course, and be careful to have the fields in the ORDER BY clause in the right order)

After running the above every data and index page should have been read and so will be in the RAM page cache (for the time being at least). We have scripts like this for our application databases, which are run after reboot so that the first users logging into the system afterwards don't experience slower responsiveness. You are better off hand-writing any such script, instead of scanning the db definition tables (like sys.objects/sys.indexes/sys.columns in MSSQL), then you can selectively scan the indexes that are most commonly used rather than scanning everything which will take longer.

1
  • In Postgres 9.5, I tried SELECT * FROM schema.table and saw it load the entire 60GiB table into my 100GiB PostgreSQL buffer cache.
    – sudo
    Mar 7 '17 at 15:46
1

I had similar problem:
After restarting server service and all cashed data dropped, many queries called first time where really really slow, cause of specific complexity of the queries, until all necessary indexes and data was cashed. that means, for example users must hit once every "item" (1-3 sec exec time) and related data from 50 million rows, so users would not experience any unwanted delays anymore. It takes first 3 hours for users to experience annoying hangs, till most used data is cashed and programs are ruining top notch with production performance, end even then, 2 days a few sudden short delays, when hitting less first time accessed data ..., for statistics data etc.

To solve this, did write a small python script which does perform selects on heaviest used tables with large indexes. It took 15 min to run, and no performance delays.

0

Hmmm, may be COPY command would help. Just execute COPY to stdout and read from it. It is possible to do it using pg_dump:

pg_dump -U <user> -t <table> <database> > /dev/null

Other way is to find all table files and run cat <files> > /dev/null.

Here is the example on how to get table filenames:

# SELECT oid, datname FROM pg_database ;
  oid  |  datname  
-------+-----------                                                                                                                                          
<...>
 16384 | test
-- out of database is 16384
# SELECT oid, relname FROM pg_class WHERE relname like 'fn%';
  oid  | relname 
-------+---------
 24576 | fn
(1 row)
-- oid of our table is 24576

so, table's file(s) is /path/to/pgsql/data/base/16384/24576*

You migth want to read indexes and toast tables as well, get their oids in the same way.

BTW, why do you need it? I believe postgresql and OS is smart enough to cache hottest data and maintain good. cache efficiency.

0

I use RamDrive from QSoft, which was benchmarked as the fastest ramdisk for Windows. I just used

initdb -D e:\data

where e:\ is the place of the RamDisk.

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.