I'm running 1 single medium RDS instance for my websites.
How do I determine when I will need to scale it up? Is there a way to scale it horizontally or I can only upgrade it to a bigger instance?
You need to scale it up when resource of CPU and disk io is the bottleneck and the cost of optimising the database is higher than scale up.
There are many way to scale it horizontally for sure. Master slave replication is one of the most used way to scale horizontally. You can also run MySQL cluster too.