Got it, unfortunately we didn't have binary logs enabled on those replicas to make a good bug report. This has happened first time and only so far, we had the replica working for over a week. The only common things were, it was at the same time and same table, but different primary ids/keys on the two MySQL8 replica. Only MySQL 8 replicas failed, the MySQL 5.7s were fine. We had STRICT slave_exec_mode on all of them.

@RickJames Curious, what suggests we might have hit a bug?

@RickJames yes, the topology is 1 Primary and 4 replicas. Primary and 2 replicas are on 5.7.40, 2 replicas on MySQL 8.0.36. Added a bunch of info that could be relevant

The two concurrent inserts are definitely invalid in business logic sense and ideally shouldn't trigger. I am trying to debug why is that even happening with the owners of that code. But yes, separately, couldn't understand the mysql deadlock log.

Hi @Vérace and @danblack, thank you checking and responding. Updated with table description, token is a unique key on the table. Still dont follow what the second txn is waiting on. Thanks again!

Yes. Rails framework.