At my last job we used pt-online-schema-change for such changes, or any other ALTER TABLE changes, hundreds of times per week on tables much larger than yours. I worked on an internal service and dashboard to allow developers to run schema changes on their own. I know -- that's madness!
For such large tables, you have to be careful about restarts. If the database has a failover event or if the host where pt-online-schema-change runs has a restart, then you have to start over. We actually developed patches for pt-online-schema-change to save its state, so we could resume where it left off if the script was interrupted. Unfortunately those patches are not public and I've left that job.
At least run pt-online-schema-change in a screen
or tmux
session, so you don't have to depend on an uninterrupted ssh session.
How much time does it take for a very large table? It varies, because pt-online-schema-change monitors a couple of performance indicators, and it slows itself down dynamically if it thinks the table copying workload is causing a performance drop. So if your database is normally serving high traffic levels, pt-online-schema-change would take more time than it would if the database were idle. Therefore it's worthwhile to schedule your schema changes during off-hours if possible.
Tables that are large may take more than 24 hours to complete a schema change. I think the longest I saw was 4 weeks. That was probably a single table over 1TB, on a very busy database server. It was unfortunate, because I recall in that case, the developers thought they could drop an index. Once they dropped it, it turned out they really did need that index after all for certain queries. But it took 4 weeks to do the alter table to recreate the dropped index. Because of using pt-online-schema-change, the table could still be queried during that 4 weeks, but the performance of certain queries was bad without the needed index. That was painful.
I kept telling the developers that allowing tables to grow so large is asking for trouble, for reasons like that. But they didn't listen.
Another caveat of pt-online-schema-change is that since it must create triggers at the start and do a rename at the end, it must have exclusive access to the table briefly at the start and the end. This means it waits for an exclusive metadata lock, if there are any transactions outstanding against the table. So if you have long-running queries, or even short queries that leave their transaction uncommitted, it will block the startup or the rename at the end. And while pt-online-schema-change is blocked waiting for that metadata lock, it blocks all other queries. This can cause a serious problem.
So we found a way to invoke pt-online-schema-change with a 2-second timeout on the metadata locking. If it can't get its job done in 2 seconds, it stops waiting, and must try again. This prevents long logjams like I described. Sometimes it means pt-online-schema-change must retry many times to start or to finish. But that's better than an outage.
Ideally, you would not have such long-running transactions, but that's up to your application code. It might be difficult to know if you have such cases, or which code is responsible for them.
The final caveat I can think of right now is if you do joins on string columns anywhere, changing the character set and hence the collation means that if those joins were depending on indexes before, they can't now. Such joins may be at a great disadvantage on performance until you can alter the joined table to be compatible. This has nothing to do with pt-online-schema-change, but would apply to any method you use to change character sets.
I hope you are upgrading to utf8mb4, not just utf8. The utf8mb4 is becoming the preferring character set, and utf8 (the 3-byte type) is becoming deprecated.
I'm not sure about the replication issue you mentioned. I suggest you test it, not with your production database, but in a test environment. My suspicion is that statement-based replication would work, but I'm not sure that row-based replication would.