If your MySQL Server is causing out of memory problems, it's probably not related to the transaction size. I think that's a red herring.
InnoDB transactions can be larger than your RAM, because MySQL saves uncommitted work to disk as you go. If you run out of innodb transaction log space, it must flush modified pages from the buffer pool to disk, but these can still be related to an uncommitted transaction.
Another limit on transaction size is max_binlog_cache_size
. If your MySQL Server uses a binary log, and a given transaction is larger than this configuration value, there's a risk that you'll corrupt your binary log, and replicas will stop with errors (I've seen this happen). Despite the reference to cache, this is not related to RAM either. Binlog cache uses both RAM and temporary files on disk.
MySQL Server uses the largest amount of RAM for the InnoDB Buffer Pool, not for transactions. But the Buffer Pool uses a fixed amount of RAM, which is allocated at MySQL Server startup.
In addition, there is a smaller amount of RAM allocated per connection and per query. I've seen this push total RAM usage over the system's physical RAM capacity if you have too many sessions running complex queries concurrently. Each query individually may be fine, but if 100 of them are running at the same time, it adds up to too much.
You can use the performance schema in MySQL 5.7 and later to measure memory usage, and you may be able to figure out the culprit. Here's a blog that describes how to do this: https://www.percona.com/blog/2020/11/02/understanding-mysql-memory-usage-with-performance-schema/