Do writes to TempDB always result in an actual physical write to disk,
or are TempDB writes cached by SQL Server for delayed write like in
the Windows file system cache?
Do they always? Most definitely not. Do they ever? Yes but not as a result of the typical mechanism. Reference here is What does checkpoint do for tempdb?.
In a "well behaved" system, writes to a user database file occur on checkpoint. In a badly behaved system, writes will also occur when the lazywriter needs to flush pages from the buffer pool to make room for other pages.
A checkpoint is only done for tempdb when the tempdb log file reaches
70% full - this is to prevent the tempdb log from growing if at all
possible (note that a long-running transaction can still essentially
hold the log hostage and prevent it from clearing, just like in a user
But there is no need to flush tempdb to disk, as crash recovery is never run on tempdb, it's always recreated on startup.
Tempdb is not recovered in the event of a crash, and so there is no
need to force dirty tempdb pages to disk, except in the case where the
lazywriter process (part of the buffer pool) has to make space for
pages from other databases.
This remarkably (it was a surprise to me) is the only mechanism by which tempdb pages will be written to disk. If there is buffer pool pressure, tempdb pages may be flushed to disk. If there isn't, it should not occur.
Edit: Debatable whether "badly behaved" is an appropriate description for a user database that is writing pages outside of checkpoint. Unusual, atypical, or just not ideal maybe?
Additional edit (following comments/chat with @SqlKiwi):
The glaring omission above is that temporary tables are not sole source of tempdb traffic. Quoting from Understanding Hash, Sort and Exchange Spill Events:
Certain SQL Server query execution operations are calibrated to
perform best by using a (somewhat) large amount of memory as
intermediate storage. The Query Optimizer will choose a plan and
estimate the cost based on these operators using this memory
scratch-pad. But this is, of course, only an estimate. At execution
the estimates may prove wrong and the plan must continue despite not
having enough memory. In such an event, these operators spill to disk.
I had incorrectly assumed that the mechanism behind a physical write for a spill operation were exactly the same as described earlier i.e. the lazywriter forcing tempdb pages to disk as a result of pressure on the buffer pool (caused by the spill).
@SqlKiwi explained where I was wrong (thanks Paul!):
I think you are asking why physical tempdb activity occurs when a
query exceeds its workspace memory grant, rather than just using
tempdb-in-memory If it did that, the query would be using more memory
than its grant, defeating the point of restricting memory grants in
the first place.
Spills are indeed special in writing through to
storage. Physical tempdb activity is seen on a spill, even in the face
of oodles of free memory and zero pressure on tempdb.
Paul also pointed me to his blog post Advanced TSQL Tuning: Why Internals Knowledge Matters which includes example scripts for demonstrating spills, for those that want to delve deeper.