6

I'm using Berkeley DB (BDB) as a persistent store for a JMS queue. When I consume an entry from the queue the underlying BDB files do not immediately shrink, but do eventually. I'm running into issues with the BDB files taking up lots of space on the file system while retrieval performance degrades.

My entry size varies considerably but it is not uncommon to have 400,000 messages of around 32kb each in the persistent queue.

I'd like to understand how BDB manages the files so that I can throttle the number of entries for file size/retrieval performance. Or so I can rule out BDB as my persistent store mechanism.

I am probably searching for the wrong terms but have not found what I'm looking for in the Oracle documentation or The Berkeley DB Book. I would not be surprised if BDB doesn't want me messing with its internals but I would be surprised if (at least) an overview of how it does handle its internals is not available.

2

Basically the philosophy seems to be that its not worth putting much effort on compacting the DB if it will grow again. The way the BDB engine works makes it hard to really reclaim any freed space on workloads that have much insert/update activity, and I think JMS persistence quite possibly is such a workload. The gain from this philosophy, of course, is that on a burst of new messages the DB doesn't need to allocate more pages but can directly write the data on existing pages in the most efficient way. But if the hit on retrieval performance is significant it might be that BDB is indeed not right choice for you workload.

I wonder if the answers provided in these posts in the Oracle forums provide any light to the mystery (the quote comes from the second link).

Understanding compact and database file size

Berkeley DB database file size keeps growing

There is no specific maximum size that the database must reach before page reuse starts. Also, you don't need to close and reopen the database handles in order to influence page reusing. BDB reuses pages when they are emptied and it does not perform any kind of automatic key/page balancing (as it will lead to deadlocks). The factors that you should be looking into are the insert, update and delete rates, if transactions are long-lived, the deadlock detection policy, the size of the database page size and the page fill factor etc.

One example that might explain the behavior you're seeing is one where the delete process/thread(s) gets "late" to a page when trying to remove on key found on that page. By the time it acquires the write lock on that page, there are no more consecutive keys there, and at best the delete process/thread only manages to remove just part of those keys (the other keys are ones with higher values, as the insert and update processes/threads are far ahead), thus the page will not be emptied so that it can be placed onto the free list for reuse. If the insert and update processes are very active, this may result in rapid page splits (or new page allocations), hence sets of keys are moved over onto new pages. In addition, the new keys inserted may land on already populated pages, thus making things more difficult for the delete process. If the new keys being inserted can be distributed based on their values to already populated pages where there is room for them, then the pages on the free list, if any, will not be reused.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.