A differential backup actually only reflects the changes that were made on your database since the last full backup; more specifically: the extents that were changed since your last full backup.
A reason for the differential backup still backing up the no longer used extents may be that you have deleted a bunch of data using TSQL DELETE and not dropped the tables involved.
When rows are deleted from a heap the Database Engine may use row or
page locking for the operation. As a result, the pages made empty by
the delete operation remain allocated to the heap. When empty pages
are not deallocated, the associated space cannot be reused by other
objects in the database.
Since the pages remain allocated, they will also be included in the backup.
My guess is that a DBCC SHRINKFILE will deallocate the empty pages, thus decreasing the size of DIFFERENTIAL backups, however I was not able to confirm this.
Shanky's comment of course also is a valid question as backup compression surely will not store empty pages themselves but only a reference to them and therefore vastly make a difference in size for you (in case you have Enterprise Edition, though).