If it's saying the file is not empty, it's not empty. You need to figure out what's there and either rebuild it on another filegroup manually, or drop it. `EMPTYFILE` only works _on the last file_ if there are no objects at all, and of course this is not possible on the `PRIMARY` filegroup.

My first guess is there's a table or index still associated with this filegroup:

    DECLARE @fgid int;

    SELECT @fgid = data_space_id
      FROM sys.filegroups
      WHERE name = N'the_filegroup_name';

    SELECT [object] = o.name, [index] = i.name
      FROM sys.objects AS o
      INNER JOIN sys.indexes AS i
      ON o.[object_id] = i.[object_id]
      WHERE i.data_space_id = @fgid;

    -- If that isn't it, there might be a partition scheme 
    -- no longer associated directly with any objects:

    SELECT name
      FROM sys.partition_schemes 
      WHERE data_space_id = @fgid;