Mine went from 2.5 mins to open, to 1-2 seconds.
MSDB.dbo.backupset keeps a log of backups (separate to what's inside backupfiles themselves).
Try
select *
from msdb.dbo.backupset
where database_name = 'Your-DB-Name-Here'
To clean up:
EXEC msdb.sp_delete_backuphistory @oldest_date = '2019-06-27 10:00:00.000';
WARNING: the sp_delete_backuphistory
procedure isn't limited to one database, there are no other parameters for this procedure call
The date above is an example.
It is recommended to include this kind of thing in your maintenance plan.