0

I'm trying to better understand Checkpoint internals, specifically in PostgreSQL.

My current understanding is that Crash Recovery will always start from the latest REDO location that is marked when a Checkpoint starts. Data in all WAL records before the REDO location are guaranteed to be on disk (in data files), and those WAL records are no longer needed.

I'm having a hard time understanding the following:

1) How is the REDO location determined exactly. When a Checkpoint starts, does it literally say "REDO location is the current WAL record + 1"?

2) When we do determine the REDO location, how is it that we are guaranteed that all WAL records before this are no longer needed?

Specifically, how is the following scenario handled:

  • Checkpoint 1 runs and writes out dirty page A to disk.
  • Dirty page A then has new rows inserted
  • Checkpoint 1 finishes
  • Checkpoint 2 starts and determines REDO location.

How can Checkpoint 2 be sure that all previous data is written to disk if pages can be dirtied after Checkpoint 1 flushes them, is the gist of the question.

1

How is the REDO location determined exactly. When a Checkpoint starts, does it literally say "REDO location is the current WAL record + 1"?

In essence, yes. Exactly, no. There is code in there about rounding up for cases where the record will not fit in the remainder of the log file and other corner cases. To get an exact answer you have to read the source code. See CreateCheckPoint in src/backend/access/transam/xlog.c

When we do determine the REDO location, how is it that we are guaranteed that all WAL records before this are no longer needed?

A checkpoint starts (after computing the REDO pointer) by surveying all pages in the buffer cache, and marking the ones which were dirty at that time. Before a checkpoint can end successfully, all of those marked pages need to have been written and flushed. Once an initially dirty page has been flushed, the WAL that protected that initial dirtying of the page is no longer needed, because the page itself is safely on disk and can speak for itself. If the page starts out dirty and is then re-dirtied multiple times throughout the checkpoint, then the subsequent dirtying actions will each have their own WAL records, which will necessarily fall after the REDO pointer. Those WAL records will be retained, at least until the successful completion of the next checkpoint after the current one.

  • Thanks for the great answer @jjanes. I have a follow up question. What happens if Postgres crashes after a checkpoint has started but before all dirty pages have been written out. In that case would we start at the prior REDO location (the one before the REDO location that we just defined)? And if so, does PostgreSQL have some notion of marking a checkpoint as having completed? Thanks! – Dave Stibrany May 26 '16 at 20:27
  • 1
    When a checkpoint is completed, a record is written into the WAL to indicate the successful completion, and the control file is update as well. Until the checkpoint is completed and that record is written, the REDO pointer exists only in memory, and upon a crash it is no different than if the checkpoint had never started. – jjanes May 26 '16 at 20:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.