This absolutely matters.
When the Core cluster resources are owned by the AG secondary node that is the Windows Failover Cluster home. At any time the node owning the Core Cluster Resource may not see another node in the Windows Failover Cluster and will try to restart the cluster service on that node. At that point the AG Role goes offline and either migrates,stays offline, or something else depending on your AG Role settings on the WFC and the settings of the AG Group.
No matter how any of that is configured all connectivity to the AGL and any AG databases is down at this point until the Role comes back online on the old primary or another one.
Looks something like this in the SQL Error Log on the Primary Replica:
"A connection timeout has occurred on a previously established connection to availability replica 'XXXXX02' with id [DD7AA510-53B2-4AA4-AF12-D6348A838398]. Either a networking or a firewall issue exists or the availability replica has transitioned to the resolving role."
NOTE: This came from a server this morning that has all the latest updates and is newly built (current date: 2022-11-23).
When you have AG Roles with Replicas that are windows failover cluster nodes in a separate subnet this can be a common thing if your Cluster Name resource is running on a node that is not the Primary Replica.
Recommendation is to have some window server level monitoring and alerting for when the Cluster Name resource is not running on the same cluster node that the AG Role is running on.