30

I only heard about Robert Martin today, and it seems like he's a notable figure in the software world, so I don't mean for my title to appear as if it's a click bait or me putting words in his mouth, but this is simply how I interpreted what I heard from him with my limited experience and understanding.

I was watching a video today (on software architecture), on a talk by Robert C. Martin, and in the latter half of the video, the topic of databases was the main focus.

From my understanding of what he said, it seemed like he was saying that SSDs will reduce the usefulness of databases (considerably).

To explain how I came to this interpretation:

He discussed how with HDDs/spinning disks, retrieving data is slow. However, these days we use SSDs, he noted. He starts off with "RAM is coming" and then continues by mentioning RAM disks, but then says he can't call it RAM disk, so resorts to just saying RAM. So with RAM, we don't need the indexes, because every byte takes the same time to get. (this paragraph is paraphrased by me)

So, him suggesting RAM (as in computer memory) as a replacement for DBs (as that's what I interpreted his statement as) doesn't make sense because that's like saying all the records are in-memory processed in the lifetime of an application (unless you pull from a disk file on demand)

So, I resorted to thinking by RAM, he means SSD. So, in that case, he's saying SSDs reduce the usefulness of databases. He even says "If I was Oracle, I'd be scared. The very foundation of why I exist is evaporating."

From my little understanding of SSDs, unlike HDDs, which are O(n) seek time (I'd think), SSDs are near O(1), or almost random. So, his suggestion was interesting to me, because I've never thought about it like that. The first time I was introduced to databases a few years ago, when a professor was describing the benefits over regular filesystem, I concluded the primary role of a database is essentially being a very indexed filesystem (as well as optimizations, caching, concurrent access, etc), thus, if indexes aren't needed in SSD, this kind of does make databases less useful.

Regardless of that though, prefacing that I'm a newb, I find it hard to believe that they become less useful, as everyone still uses DBs as the primary point of their application, instead of pure filesystem, and felt as if he was oversimplifying the role of databases.

Note: I did watch till the end to make sure he didn't say something different.

For reference: 42:22 is when the whole database topic comes up, 43:52 is when he starts off with "Why do we even have databases"

This answer does say SSDs speed DBs up considerably. This question asks about how optimization is changed.

To TL;DR my question, does the advent of widespread SSD use in the server market (whether it's upcoming or has happened already) reduce the usefulness of databases?

It seemed like what the presenter was trying to convey was that with SSDs, one can store the data on disk, and not have to worry about how slow it would be to retrieve it as with older HDDs, as with SSDs, seek times are near O(1) (I think). So, in the event of that being true, that would hypothetically lose one of the advantages it had: indexing, because the advantage of having indexes for faster seek times is gone.

Improve this question
0

4 Answers 4

Reset to default
62

There are some things in a database that should be tweaked when you use SSDs. For instance, speaking for PostgreSQL you can adjust effective_io_concurrency, and random_page_cost. However, faster reads and faster random access isn't what a database does. It ensures

He's just wrong about indexes. If the whole table can be read into ram, an index is still useful. Don't believe me? Let's do a thought experiment,

  • Imagine you have a table with one indexed column.

      CREATE TABLE foobar ( id text PRIMARY KEY );

  • Imagine that there are 500 million rows in that table.

  • Imagine all 500 million rows are concatenated together into a file.

What's faster,

  1. grep 'keyword' file
  2. SELECT * FROM foobar WHERE id = 'keyword'

It's not just about where data is at, it's about how you order it and what operations you must do to find what you're looking for. PostgreSQL supports B-tree, Hash, GiST, SP-GiST, GIN and BRIN indexes (and Bloom through an extension). You'd be foolish to think that all of that math and functionality goes away because you have faster random access.

Improve this answer
2
  • 32
    Just an addendum - OP should be careful not to conflate "random access" with "content-addressable access". As OP noted, "random access" means that getting to each byte of memory is O(1). However, FINDING data in that "random-access memory" still requires sequentially searching through it; that is, you can't ask the memory "find me the data that looks like this" and have it magically handed to you.
    – Bob Jarvis - Слава Україні
    Dec 22, 2016 at 20:04
  • 2
    @BobJarvis You're correct. Your comment helps clear up even more @EvanCarroll's "What's faster" example on why indexing and even subindexing matter, and just grabbing in O(1) isn't sufficient for the use cases a DB provides
    – Honinbo Shusaku
    Dec 22, 2016 at 20:09
12

Based on your post, it appears the clear message is that RDBMS lookup time optimizations are being replaced with hardware which makes IO time negligible.

This is absolutely true. SSD on database servers combined with high (actual) RAM makes IO waiting significantly shorter. However, RDBMS indexing and caching is still of value because even systems with this huge IO boon can and will have IO bottlenecks from poorly performing queries caused by bad indexing. This is typically only found under high workload applications or poorly written applications.

The key value to RDBMS systems in general is data consistency, data availability, and data aggregation. Utilizing an excel spreadsheet, csv file, or other method of keeping a "data base" yields no guarantees.

SSD doesn't protect you from your primary server become unavailable for any reason (network, OS corruption, power loss). SSD doesn't protect you from a bad data modification. SSD doesn't make it faster to run analytics compared to "just having" them.

Improve this answer
2
  • Although I've gained better insight, I was asking in the context of raw SSD data storage vs data storage on a DB w/ HDD, and your answer is in the context of DB on SSD (due to poor question phrasing from me)
    – Honinbo Shusaku
    Dec 22, 2016 at 20:12
  • 4
    @Abdul That comparison is apples-to-suspension bridges. A raw device gets you a big expanse of storage; a database gets you a way to organize and access that storage according to a data model. Josh's point here is that if you go into this with the starry-eyed idea that a raw SSD is a wonderful thing because it's "fast" and that you're just going to write code to do all of your data storage on that raw volume, you will eventually end up writing a database.
    – Blrfl
    Dec 23, 2016 at 11:32
9

Uncle Bob probably was talking about in-memory databases such as Redis or Gemfire. In these databases, everything in the database really is contained in RAM. The database could start out empty and be filed with short-lived data (being used as a cache) or it start by loading everything in from disk and periodically checkpoint changes to disk.

This is becoming more and more popular because RAM is getting cheap, and it becomes feasible to have a terabyte of data stored in an in-memory clustered database. There are a lot of use cases where the speed from having instant access to things makes it valuable to put in RAM rather than even a fast disk like SSD. You can even continue using SQL for some of these if it makes sense.

Why should this worry Oracle? Data is growing and it's unlikely that RDBMSes will go away. However, a lot of Oracle's engineering time over the years has gone into ways to make data retrieval on spinning disks really fast. Oracle will need to adapt to a completely different storage tier. They are, with Oracle Database In Memory, but they're exposed to different competition than in the past. Think of how much time has gone into making sure the query optimizer chooses the right strategies based on the layout of things on disk....

Improve this answer
2
  • Ah. I never knew there things such as in-memory databases
    – Honinbo Shusaku
    Dec 23, 2016 at 20:04
  • 1
    As another example SQLite can run in memory so no need to use a different database
    – mmmmmm
    Dec 24, 2016 at 10:20
8

Community Wiki post collecting answers originally left as question comments

I would say just the opposite. Since read/write speeds are so fast, now you can get a GPU accelerated database (e.g. BlazingDB or Alenka) to crunch numbers even faster. Now you can have even more complex queries run faster. Now queries which people wouldn't even consider running can be run at a reasonable speed. The more complex, and the more data the better off you are - cybernard

While Bob Martin has been around for a long time and his opinions are generally worth listening to (if not agreeing with :-), in this case I think he's diving into the "The Death Of Relational Databases Is Upon Us" crowd (of which I'm an associate member :-). For some things under limited circumstances a somewhat convincing argument can be made that non-relational database technologies can provide an edge. That having been said, however, IMO the relational model, flawed in various and sundry ways as it may be, still provides the best general purpose database model available today. YMMV. - Bob Jarvis

The primary reason that we use databases isn't because disks are slow (indeed, originally, that was cited as a reason not to use databases), but rather because data is complicated. The primary purpose of a database is to enable multiple apps/users to be able to find the correct data and even to be able to simultaneously alter it in a controlled manner. Doing that quickly is only a secondary goal of databases. - RBarryYoung

RDBMS isn't going away anytime soon; they're the best choice for some types of application, and NoSQL (Mongo, etc.) is the best choice for others. Horses for courses. - sh1rts

Database helps organize data. It was not really designed for fast access of data in the first place anyways. - JI Xiang

Improve this answer
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.