The cappedCollection concept works well for most of my projects where cleaning old data without care makes sense.
For another projects, I need a more complex and safe concept. The requirement is nearly the same as
The data is appended to the main collection, without compression/compact and no index except a timestamp for simple queries by time. => The focus is on writing and persistent data.
Similar to the logrotate rules, I'd like the main collection not to become too large => capped by size; if possible, capping by timestamp might be a plus.
This sounds like a cappedCollection, but I do not want any data loss when it's capped. The old data should be stored into another db's collection that must be
compact:true and a non-capped collection. It's name depends on the current month and makes sure that there will be max 12 "archive" collections per year.
liveDB.mainCollection_capped grows and starts capping.
Before removing old documents, these are savely moved into
No data is lost and the main collection remains small and fast. Storing the data in another database avoids db locks, e.g. repairDatabase tasks on an archive file will not affect or delay the main collection.
Is there a good practice or how to achieve this - as reliable and automated as possible - without writing all the data transfer for a cronjob which handles the data transfer but should never ever be missed because data is lost if capping starts before old data is copied into the archive.