You could NEVER, EVER trust Task Manager to tell you how much memory SQL Server is using (maybe you are remembering a 32-bit system with a very small amount of memory). Stop using Task Manager for this, period. Use the performance counter - you can also query the performance counter using DMVs:
SELECT object_name, cntr_value
FROM sys.dm_os_performance_counters
WHERE counter_name = 'Total Server Memory (KB)';
You could save that as a query shortcut in Tools > Options > Environment > Keyboard > Query Shortcuts, and get accurate results in a query window much faster than getting inaccurate results from Task Manager.
You can also check for memory pressure (and whether you can do anything about it) using these queries:
SELECT object_name, counter_name, cntr_value
FROM sys.dm_os_performance_counters
WHERE counter_name IN
('Total Server Memory (KB)', 'Target Server Memory (KB)');
-- SQL Server 2012:
SELECT physical_memory_kb FROM sys.dm_os_sys_info;
-- Prior versions:
SELECT physical_memory_in_bytes FROM sys.dm_os_sys_info;
EXEC sp_configure 'max server memory';