1

I'm designing a system that checks a given website for any security vulnerabilities. The system includes a client (firefox plugin) and a server. The server does all the scanning while the client just relays that info to the user. If a website is dangerous, it is blacklisted; otherwise whitelisted.

The system must hypothetically be able to handle several thousands of requests and updates to the database simultaneously.

Although the database is expected to have a very simple structure, I am still considering using NoSQL because my understanding is that it can handle a greater amount of queries. Is this true? Which db technology is better suited for my system?

4 Answers 4

6

It's really too early to tell from what you've described - not enough detailled information. There are those who say (probably correctly) that NoSQL systems can update rapidly - even more rapidly than RDBMS technology. However, that speed comes at a price (no SQL/poor indexing) - this is the ONE thing that people should realise about NoSQL.

It does have use cases, but IMHO, a decently designed RDBMS can cope with most (the vast majority) of work given to them. Indeed, some of the reference users of NoSQL (e.g. Google) have been rowing back on its benefits and not using indexes and SQL. Take a look at Brian Aker's talk here - he was MySQL AB's chief architect.

What you should also realise is that NoSQL represents a gamut of technologies, some of which may suit your use case, but an ordinary RDBMS may well be suitable also. You should perform some tests using both RDBMSs and NoSQL systems (or hire a consultant). Also, consider your own areas of expertise - maybe the devil you know is better than the one you don't? If you're comfortable with an RDBMS solution, maybe that will be more important than a 5% performance gain?

2

There's two parts to this question:

Where should the data be persisted permanently?

Let's hold off on that part until we answer the second one, which is much more important:

Where should the clients query to check for vulnerabilities?

A caching layer. Your clients should never hit the underlying data store first. There's not a need for atomic transactions here, and your queries would be fine dealing with data that's a few seconds, a few minutes, or perhaps even a few hours old.

For that caching layer, you can use things like Redis, Memcached, Elasticache, or whatever in-memory store is available at your chosen cloud provider.

Your app should check the cache first, and if the required listing isn't in cache, query it from the database, and then populate it into the cache along with an expiration date. This logic should be built into your app, not into the data persistence layer or the caching layer.

Now, back to that first question: where should the data be persisted permanently? Once you design the app to check the cache layer first, you can see why the persistence layer isn't quite as important of a decision. It's easy to scale a caching layer to thousands (or hundreds of thousands) of concurrent connections.

The answer: use the database you're already comfortable with, and start getting comfortable with a caching layer.

1

The volume of queries that a system can handle depends on more than just the database management system you use. SQL / NoSQL systems will both require a solid hardware foundation to handle high volumes.

If you are more concerned with masssive throughput and being able to store unstructured data I would lean towards NoSQL.

If you are more concerned with accuracy/consistency of data, I would lean towards SQL.

My rule of thumb is financial transactions belong in a SQL database. Entertaining data (facebook status updates, tweets, pinterest postings, sensor data) are more suited for a NoSQL solution.

In your case with the volumes you are talking about I would go with NoSQL solution distributed across several nodes.

0

Your use case looks like a simple key-value lookup, where the key is the website and the value part will say if the website is black/whitelisted. Also, you may not need the typical RDBMS functionalities like joins etc.

I think a NoSQL solution is a better fit for you. Why have the baggage when you dont need it.

2
  • What do you mean by "baggage"? Concurrency control, transactions, security - that sort of thing? Jul 5, 2014 at 12:07
  • Yes. On top of what you said, traditional RDBMS's also do query optimization, plan generation etc which is un-necessary for simple key-value lookups. In the list you mentioned, I think concurrency control is needed in any system and nosql systems offer it.
    – sunil
    Jul 7, 2014 at 5:47

Not the answer you're looking for? Browse other questions tagged or ask your own question.