All your assumptions are correct.
You need to ANALYZE the materialized view to get correct estimates – this should be done automatically by autovacuum, so I am not sure what could be wrong there.
It also looks like your materialized view is terribly bloated. You would need to run VACUUM (FULL) on it to fix that.
Both point in the direction of autovacuum not ...