InnoDB deadlocks with IX locks

Question

Table schema

  `id` varchar(255) NOT NULL,
  `created_time` bigint DEFAULT NULL,
  `updated_time` bigint DEFAULT NULL,
  `tdi` varchar(255) NOT NULL,
  `gid` varchar(255) NOT NULL,
  `auth_id` varchar(64) NOT NULL,
  `version` bigint DEFAULT NULL,
  `f_cdc` bit(1) DEFAULT b'1',
  `user_id` bigint NOT NULL DEFAULT (_utf8mb4'0'),
  `series_id` bigint NOT NULL DEFAULT (_utf8mb4'0'),
  `cd` bigint NOT NULL DEFAULT (_utf8mb4'0'),
  `notification_status` enum('NOTIFIED','SKIPPED','UNPROCESSED') DEFAULT NULL,
  PRIMARY KEY (`tdi`,`series_id`,`user_id`),
  UNIQUE KEY `id$tdi` (`id`,`tdi`)

Innodb deadlock log

LOCK WAIT 3 lock struct(s), heap size 1136, 2 row lock(s)
MySQL thread id 2420106, OS thread handle 140412549347072, query id 4960324289 xxxxxxx.xxxx.xxx xxxxxxx.xxxx.xxx tidbadmin update
Insert into emaillog (`id`,`gid`,`version`,`tdi`,`auth_id`,`created_time`,`updated_time`,`f_cdc`,`cc`,`click_details`,`clicked`,`cd`,`completion_time`,`email_category`,`email_delivery_status`,`email_subject`,`from_email`,`from_name`,`mailer`,`mailjob_id`,`open_details`,`opened`,`reply_to`,`schedule_time`,`status_detail`,`template`,`user_email`,`user_id`) values ("userJob|JOB-xozrD168621886603817217b3bdf5b20041686","",1,"1592758856318018638","",1686218871,1686218871,true,"",NULL,NULL,1643243617115175397,0,"other","done","Invitation | SeriesInvite1686218819","[email protected]","tuser91680614785080 ","SESMailer","MJHo8Tb1686218871371",NULL,NULL,"[email protected]",NULL,NULL,NULL,"[email protected]",0)

*** (1) HOLDS THE LOCK(S):
RECORD LOCKS space id 890 page no 21509 n bits 112 index PRIMARY of table `rtid$staging$gci`.`emaillog` trx id 1496376200 lock_mode X locks gap before rec
Record lock, heap no 25 PHYSICAL RECORD: n_fields 31; compact format; info bits 0
 0: len 30; hex 757365724a6f627c4a4f422d786f7a724431363836323138383636303338; asc userJob|JOB-xozrD1686218866038; (total 50 bytes);
 1: len 19; hex 31353932373538383536333138303138363338; asc 1592758856318018638;;
 2: len 6; hex 00005930e36e; asc   Y0 n;;
 3: len 7; hex 82000000910110; asc        ;;
 4: len 8; hex 800000006481a877; asc     d  w;;
 5: len 8; hex 800000006481a877; asc     d  w;;
 6: len 0; hex ; asc ;;
 7: len 0; hex ; asc ;;
 8: len 8; hex 8000000000000001; asc         ;;
 9: len 1; hex 01; asc  ;;
 10: len 8; hex 8000000000000000; asc         ;;
 11: len 30; hex 6e65775f6c6561726e657238313638363231383836323539344031736563; asc new_learner81686218862594@1sec; (total 38 bytes);
 12: len 8; hex 96cdf95d0d93f9e5; asc    ]    ;;
 13: SQL NULL;
 14: len 8; hex 8000000000000000; asc         ;;
 15: SQL NULL;
 16: len 4; hex 646f6e65; asc done;;
 17: SQL NULL;
 18: len 5; hex 6f74686572; asc other;;
 19: len 0; hex ; asc ;;
 20: len 20; hex 61646d696e406d696e647469636b6c652e636f6d; asc [email protected];;
 21: len 20; hex 7475736572393136383036313437383530383020; asc tuser91680614785080 ;;
 22: len 30; hex 74757365723931363830363134373835303830406d742d6f6a6c656f2e63; asc [email protected]; (total 32 bytes);
 23: len 30; hex 496e7669746174696f6e207c20536572696573496e766974653136383632; asc Invitation | SeriesInvite16862; (total 35 bytes);
 24: len 20; hex 4d4a4c6d486d4131363836323138383731313737; asc MJLmHmA1686218871177;;
 25: SQL NULL;
 26: len 9; hex 5345534d61696c6572; asc SESMailer;;
 27: SQL NULL;
 28: SQL NULL;
 29: SQL NULL;
 30: SQL NULL;


*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 890 page no 21509 n bits 112 index PRIMARY of table `rtid$staging$gci`.`emaillog` trx id 1496376200 lock_mode X locks gap before rec insert intention waiting
Record lock, heap no 25

*** (2) TRANSACTION:
TRANSACTION 1496376201, ACTIVE 0 sec inserting
mysql tables in use 1, locked 1
LOCK WAIT 3 lock struct(s), heap size 1136, 2 row lock(s)
MySQL thread id 2420096, OS thread handle 140432547460864, query id 4960324291 xxxxxxx.xxxx.xxx  xxxxxxx.xxxx.xxx  tidbadmin update
Insert into emaillog (`id`,`gid`,`version`,`tdi`,`auth_id`,`created_time`,`updated_time`,`f_cdc`,`cc`,`click_details`,`clicked`,`cd`,`completion_time`,`email_category`,`email_delivery_status`,`email_subject`,`from_email`,`from_name`,`mailer`,`mailjob_id`,`open_details`,`opened`,`reply_to`,`schedule_time`,`status_detail`,`template`,`user_email`,`user_id`) values ("userJob|JOB-xozrD168621886603817217b3bdf5b20051686","",1,"1592758856318018638","",1686218871,1686218871,true,"",NULL,NULL,1643243617115175397,0,"other","done","Invitation | SeriesInvite1686218819","[email protected]","tuser91680614785080 ","SESMailer","MJ83Evy1686218871372",NULL,NULL,"[email protected]",NULL,NULL,NULL,"[email protected]",0)

*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 890 page no 21509 n bits 112 index PRIMARY of table `rtid$staging$gci`.`emaillog` trx id 1496376201 lock_mode X locks gap before rec
Record lock, heap no 25

*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 890 page no 21509 n bits 112 index PRIMARY of table `rtid$staging$gci`.`emaillog` trx id 1496376201 lock_mode X locks gap before rec insert intention waiting
Record lock, heap no 25
*** WE ROLL BACK TRANSACTION (2)

Observations

1. Both txs hold 1 lock
2. both txs are waiting for an IX lock on gap before page Id 890.

Queries:

1. Can someone explain the locks acquired and locks in waiting? Both txs say they hold X lock on page id 890. How can both have x lock on same page?
2. How can both txs be waiting on same gap lock which causes a deadlock? Such a scenaro should cause wait for 1 tx and possibly timeout, but not a deadlock.
3. I am using innodb in repetable read isolation level. As per MySQL documentation, IX locks allow insertion by multiple txs as long as id to be inserted is not same in these txs. It appears as if this is not true here.
4. Based on table schema and deadlock information, what is causing the deadlock?
5. Since there are 2 txs logged by innodb here, I am assuming in the cyclical dependency for locks, no other txn is involved.