Do you have in place read-only routing? Try to review read-only routing URL in the Availability Group Properties / Read-Only Routing.
Maybe the connection at that time is using this feature and the routing URL for the secondary is not right.
Eventually try to verify what encryption algorythm you are using.
SELECT name , type_desc , state_desc , role_desc , is_encryption_enabled , connection_auth_desc , encryption_algorithm_desc FROM sys.database_mirroring_endpoints;
Usually AES. Veryfy if you are using the same algorythm in both node.