Daniel's answer focuses on the cost of reading individual rows. In this context: Putting fixed size
NOT NULL columns first in your table helps a little. Putting relevant columns first (the ones you query for) helps a little. Minimizing padding (due to data alignment) by playing alignment tetris with your columns can help a little. But the most important effect has not been mentioned, yet, especially for big tables.
Additional columns obviously make a row cover more disk space, so that fewer rows fit on one data page (8 kB by default). Individual rows are spread out over more pages. The database engine generally has to fetch whole pages, not individual rows. It matters little whether individual rows are somewhat smaller or bigger - as long as the same number of pages has to be read.
If a query fetches a (relatively) small portion of a big table, where the rows are spread out more or less randomly over the whole table, supported by an index, this will result in roughly the same number of page reads, with little regard to row-size. Irrelevant columns will not slow you down much in such a (rare) case.
Typically, you will fetch patches or clusters of rows that have been entered in sequence or proximity and share data pages. Those rows are spread out due to the clutter, more disk pages have to be read to satisfy your query. Having to read more pages is typically the most important reason for a query to be slower. And that is the most important factor why irrelevant columns make your queries slower.
With big databases, there is typically not enough RAM to keep all of it in cache memory. Bigger rows occupy more cache, more contention, fewer cache hits, more disk I/O. And disk reads are typically much more expensive. Less so with SSDs, but a substantial difference remains. This adds to the above point about page reads.
It may or may not matter if irrelevant columns are TOAST-ed. Relevant columns may be TOAST-ed as well, bringing back much of the same effect.