SQL Server is kind of a memory hog. You could try clearing the cache, BUT I would highly recommend doing some reading on these commands first before you using them. From what I hear, their can be negative side effects when using these especially in a production environment.
DBCC FREESYSTEMCACHE ('ALL') DBCC FREEPROCCACHE
Here are some links I found useful: