Can a SQL Server AG listener name be used to access an administrative share
I can RDP using the listener name but can't use the listener name to access the
$ shares (as mentioned above).
You cannot use the listener name to access administrative shares on the cluster nodes. When accessing a cluster network name (which is what a listener is), shares are scoped to clustered shared folders only.
Please see Deploying a two-node clustered file server for reference.