Here's my stab in the dark at what may be happening.
With a small resultset and a sufficiently simple query plan, the query memory stolen pages (that portion of the query memory grant actually used) is under the grant size, so no spill to tempdb occurs. In this case, the logical reads are what you would expect - the query thread references to user object pages in the buffer pool.
But. what if the query stolen pages reaches the query memory grant size, and there's still lots of query work left to do? Once all granted memory has been stolen for the query, the remaining "query memory" contents spill to tempdb. These bypass the usual buffer pool LRUQ and get written to tempdb right away. If the query needs them again later to finish its work, it'll read the temdb pages into the buffer pool and the query threads will incur logical reads when accessing these intermediate results. So, work similar to query memory stolen pages references (which I don't believe get counted in logical reads) becomes logical reads of tempdb contents.
Looking through sys.dm_exec_query_stats (for query read and write stats), sys.dm_exec_query_memory_grants (for query memory grant request size and used query memory), sys.dm_db_session_space_usage (cumulative tempdb for completed tasks in the session), and sys.dm_db_task_usage (tempdb usage for in-progress tasks) should make it clear whether this "maxed out stolen pages so adding tempdb logical reads to finish the query" is occurring.