It's not unusual for SQL Server to have rather high memory utilization, even when the server is idle.
SQL Server will increase memory utilization up to the configured maximum--then hold on to that memory, even if it is not being actively used. This increase in memory usually corresponds to data being read from disk, into memory, and held in cache (thus improving the performance the next time that data needs to be used). The only time you might see SQL Server release memory is if the system is under extreme memory pressure and SQL Server is also idle.
The default configuration for SQL Server is to use a minimum of 16 MB, and a maximum of 2147483647 MB (ie, unlimited). Under this configuration, you are essentially allowing SQL Server to self-manage it's own memory.
To see how your instance of SQL Server is configured, you can run this in Management Studio to see the minimum & maximum memory configuration:
EXEC sp_configure 'show advanced options',1
EXEC sp_configure 'min server memory (MB)'
EXEC sp_configure 'max server memory (MB)'
If you wanted to configure SQL Server to allocate 128MB of memory immediately on startup, and max out at 1024MB, you would use this script:
EXEC sp_configure 'min server memory (MB)',128
EXEC sp_configure 'max server memory (MB)',1024