We're trying to build a stats server that stores visits for each document on a section for a specific date, so we need a table that has the following parameters:

  • documentID
  • sectionID
  • date
  • visits

The plan is then to increment the visits field.

Now, we tried to do this using MongoDB (albeit a very simple instance of it) and it turns out this is too write intensive for it. Is there another database system that's better suited for this task? Or should we just start growing the MongoDB infrastructure that we're using?

  • 2
    How much intensity is "too intensive"? What's the current set up? – FrustratedWithFormsDesigner Mar 7 '13 at 20:02
  • At peak moments, we have a few thousand writes a minute, and our lowly single-server setup (on a 2.2Ghz Quad-Core AMD Opteron, 8gb RAM Rackspace server w/Ubuntu 12.04) for mongo has proven not enough. – Roberto Mar 7 '13 at 20:19
  • 1
    Are you doing fire and forget writes (i.e. no fsync/safe/getLastError)? If your visits data fits in memory Redis will do tens of thousands increments/sec. – Justin Case Mar 7 '13 at 20:49
  • 1
    It would definitely seem like you want to cache this in the web layer somehow if you are getting thousands of visits a minute to the same document. If it's just thousands of visits a minute across all documents, I would think most relational DBs could handle it. – Cade Roux Mar 7 '13 at 20:51
  • 1
    @Rob If you can afford to lose say 1 minute of data without repercussions RDB would be fine. Otherwise it's suggested to use both (explained in detailed @ redis.io/topics/persistence). – Justin Case Mar 7 '13 at 21:25

To be honest I think NoSQL would be a bad match here. The eventual consistency traps may be an issue with very frequently updated information and this has problems here.

A better approach IMO would be to start with a standard SQL db (PostgreSQL is a good choice) and do things by logging visits and, asynchronously (separate process in the background) occasionally set counters at roll-forward points. This avoids lock contention.

I doubt your problem is actually a matter of write intensiveness so much as it is lock contention. What's probably happening is that you have several processes trying write to the same counter and only one can be allowed to autoritatively do this at a time. By doing a log, aggregate, and snapshot approach you can avoid this lock contention because the high writes will occur mostly in sequential disk I/O (at least in PostgreSQL).

If I am right and the issue is lock contention I don't really see a viable alternative. You have to have some way of incrementing a counter atomically and you don't want several concurrent attempts to be treated as a single attempt, incrementing by 1.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.