I think I figured it out. SSMS appears to be trying to interact with the "master" database, even if that isn't the database which is being connected. I suspect it is quietly failing/retrying - and the final result is the slowness that we experience (... even though no error messages are actually being reported to us).
There are official docs that explain how to change the selected database when connecting via SSMS. You can do this in order to connect to something other than "master". See here:
While the changing of the selected database does technically fix the connection problem, the slowness will remain. It appears to be related to the the fact that your database user still doesn't have full-fledged access to the "master" database .
To fix: Connect to "master" database with sql server administrator. Then run the following.
CREATE USER [sql-problem-user]
FOR LOGIN [sql-problem-user]
WITH DEFAULT_SCHEMA = dbo
After doing this, your sql-problem-user will be able to connect more quickly to the original database that you wanted to connect to in the first place. (It will now be able to connect to "master" as well). Hope this is clear. Once I discovered this, I was able to find others that had used this approach long ago as well. See https://stackoverflow.com/questions/29561876/azure-sql-database-slow
I'm using SSMS v18.5 (15.0.18330.0).
I had also noticed that Azure Data Studio never experienced this connection delay. So maybe some folks can think about migrating over to that tool instead. I suspect that is the tool that Microsoft believes will replace SSMS.