It's true that the InnoDB buffer pool may still contain pages for the table you want to drop. When you drop a table, MySQL has a tendency to "lock" the buffer pool as it scans for pages associated with that table. The larger your buffer pool, the longer this scan takes. RAM is fast, but it's not infinitely fast.
But after your queries stop using data from that table, over time the pages in the buffer pool associated with that table are replaced by other query activity requesting pages from other tables. This is a gradual process.
What I've done at past jobs is when I want to drop a large table, I instead RENAME TABLE
to move the table to another schema, for which the applications don't have privileges.
This ensures any applications that used to access this table won't be able to find it.
Then wait a while for the buffer pool to gradually evict the pages for that table id.
We used a delay of 7 days, then an automatic job would check once per day and delete any tables in that "pending drops" schema.
By then, we assumed the buffer pool had recycled all the pages for that table, and then dropping it would be quick.
Was 7 days long enough? Was it too long? It depends on a lot of factors, like how large the table is, how much of that table was occupying pages in the buffer pool, how fast do other queries cause the idle pages to become evicted, etc. There's not a good way to predict how long it will take precisely.