Running multiple instances on the same server is possible and in some cases necessary. The oratab is designed specifically to handle managing multiple instances. In cases where you have multiple independent applications it may be preferable to have an instance for each application.
Having sufficient memory is critical. Oracle builds the SGA in shared memory. If you don't have sufficient memory the SGA or processes will start swapping in and out. This is known as thrashing, and results in a significant performance penalty.
It may be possible to tune the instances to run with a smaller SGA. Oracle provides tools to help tune the SGA size. If any of the instances have an excessively large SGA it will penalize all instances.
Another use for memory which is important is buffer space. This acts as secondary cache, and can eliminate significant read I/O.
sar (assuming a Unix bases O/S) can give you good diagnostics on where the problem is. Likely problems are either swapping or disk I/O saturation. Adding RAM will likely address either of these problems.
Disk I/O saturation may also be dealt with by moving some tablespaces to other disks. I generally configure Oracle so that I/O is distributed across as many disks as possible.
EDIT: These are a few cases which may require separate instances.
- It is always necessary if you run different versions of Oracle on the same server. Applications are not always certified or capable working with the latest version.
- You may want an instances that can be automatically rolled back to a point in time. (Although I discourage it, I have used this for an automated testing environment.)
- If you have hard-coded schema reference in the applications, you may need different instances to deal with namespace collisions.
- Security requirements may be easier to deal with using separate instances. It may be appropriate to use different servers in this case.
- Running different versions of the same database (development, testing, user acceptance, training, and production) on the same server is safer with separate instances. I would not run all of these environments on the same server, but often run two or more on the same server.