This behavior occurs due to the time to live (TTL) on the DNS records. A client does not do a DNS lookup every time it attempts to connect to a server. It first looks in its DNS cache to see if there is a cached entry. You can see what is in the cache by running
ipconfig /displaydns. Once you run this you'll see the "Time To Live" property on each record. This indicates how many more seconds the record will stay cached.
So your client will not connect to the new IP address until the old record is timed out of the cache. If you dig into the Microsoft documentation, they recommend setting the time to live on listener DNS records to 5 minutes to minimize this delay.
Additionally, if the SQL client in use supports "multi-subnet failover", you need to enable that option. When this option is enabled, the client will try to connect to all IP addresses that are registered for the listener name. So for this to work nicely, you will have both IP addresses registered for the listener hostname at all times.
See SQL Server Multi-Subnet Clustering (SQL Server)