Basically the philosophy seems to be that its not worth putting much effort on compacting the DB if it will grow again. The way the BDB engine works makes it hard to really reclaim any freed space on workloads that have much insert/update activity, and I think JMS persistence quite possibly is such a workload. The gain from this philosophy, of course, is that on a burst of new messages the DB doesn't need to allocate more pages but can directly write the data on existing pages in the most efficient way. But if the hit on retrieval performance is significant it might be that BDB is indeed not right choice for you workload.
I wonder if the answers provided in these posts in the Oracle forums provide any light to the mystery (the quote comes from the second link).
Understanding compact and database file size
Berkeley DB database file size keeps growing
There is no specific maximum size that the database must reach before
page reuse starts. Also, you don't need to close and reopen the
database handles in order to influence page reusing. BDB reuses pages
when they are emptied and it does not perform any kind of automatic
key/page balancing (as it will lead to deadlocks). The factors that
you should be looking into are the insert, update and delete rates, if
transactions are long-lived, the deadlock detection policy, the size
of the database page size and the page fill factor etc.
One example that might explain the behavior you're seeing is one where
the delete process/thread(s) gets "late" to a page when trying to
remove on key found on that page. By the time it acquires the write
lock on that page, there are no more consecutive keys there, and at
best the delete process/thread only manages to remove just part of
those keys (the other keys are ones with higher values, as the insert
and update processes/threads are far ahead), thus the page will not be
emptied so that it can be placed onto the free list for reuse. If the
insert and update processes are very active, this may result in rapid
page splits (or new page allocations), hence sets of keys are moved
over onto new pages. In addition, the new keys inserted may land on
already populated pages, thus making things more difficult for the
delete process. If the new keys being inserted can be distributed
based on their values to already populated pages where there is room
for them, then the pages on the free list, if any, will not be reused.
