Have you tried specifying both the file name and the target size in MB, e.g.
DBCC SHRINKFILE('filename', 20);
Or the TRUNCATEONLY option, which doesn't try any of the nasty stuff like reorganizing all of your physical data:
DBCC SHRINKFILE('filename', TRUNCATEONLY);
Also you could try making sure the database is in single user mode first:
ALTER DATABASE db SET SINGLE_USER WITH ROLLBACK IMMEDIATE;
...and turn off any backup / log backup or other maintenance jobs that may be conflicting with your shrink operation. If it still occurs, please run a CHECKDB against this database and let us know the results:
DBCC CHECKDB (db) WITH NO_INFOMSGS, ALL_ERRORMSGS;
But just as a side note, you do know that shrinking is bad and should usually be reserved for extraordinary events and circumstances, right?