Creating a listener creates (permissions must allow, otherwise the entry will be made by the network team) a DNS entry, and has a dependency to an IP address. It logically follows that each listener needs its own IP address, because how else will traffic know where to be directed?
See Failover Cluster Manager>[Cluster Name]>Services and applications> [Availability Group name]. Under Server Name section, the listener name is displayed and shows the IP Address underneath, which is reflected as a Dependencies entry if you open the properties of the listener name.
If you find a way around it, I'd sure like to know, because it doesn't make sense to me that it would work. Good luck!