If it's saying the file is not empty, it's not empty. You need to figure out what's there and either rebuild it on another filegroup manually, or drop it. `EMPTYFILE` only works _on the last file_ if there are no objects at all, and of course this is not possible on the `PRIMARY` filegroup.

My first guess is there's a table, index, or partition scheme still associated with this filegroup:

    DECLARE @fgid int;

    SELECT @fgid = data_space_id
      FROM sys.filegroups
      WHERE name = N'the_filegroup_name';

    SELECT [object] = o.name, [index] = i.name
      FROM sys.objects AS o
      INNER JOIN sys.indexes AS i
      ON o.[object_id] = i.[object_id]
      WHERE i.data_space_id = @fgid;

    SELECT ps.name
      FROM sys.destination_data_spaces AS ds
      INNER JOIN sys.partition_schemes AS ps
      ON ds.partition_scheme_id = ps.data_space_id
      WHERE ds.data_space_id = @fgid
      GROUP BY ps.name;