Database servers use a lot of memory by design, as the alternative is to continually pull data off disk - which is orders of magnitude slower than memory.
Changing the max memory setting would be the primary method of reducing the amount of memory allocated to SQL Server. You've said that you don't want to actually reduce the amount of memory because application performance will suffer - so the alternative is to try to tune the workload, or potentially add more RAM to the server.
If you're going to try to tune, this resource is as good as any. Tuning can be over-whelming if you've just been given a problem, and you're new to databases. If it's a vendor product, you might want to reach it to them for support.
http://www.brentozar.com/sql/sql-server-performance-tuning/