We have an Amazon RDS SQL Server Web Edition on Windows Server 2012 with 2 processors and 8GB RAM. Should I configure the Max Memory similar to that of an instance on a local server i.e keeping back 3-4GB for OS, memory pool etc. or does RDS require a different calculation?
No, I wouldn't treat this any differently than you would an on-prem instance with regards to max server memory. The Windows Server 2012 box is almost certainly a VM, but hopefully Amazon is a managing that well to avoid VM problems.
I would definitely reserve 3 GB at minimum for the OS and other overhead.