126 votes
Accepted

Best database and table design for billions of rows of data

This is exactly what I do every day, except instead of using the hourly data, I use the 5 minute data. I download about 200 million records everyday, so the amount you talk about here is not a problem....
Mr.Brownstone's user avatar
81 votes

Best database and table design for billions of rows of data

PostgreSQL and BRIN indexes Test it for yourself. This isn't a problem on a 5 year old laptop with an ssd. EXPLAIN ANALYZE CREATE TABLE electrothingy AS SELECT x::int AS id, (x::int % ...
Evan Carroll's user avatar
  • 63.2k
49 votes
Accepted

If NoSQL stands for "Not only SQL", is SQL a subset of NoSQL?

TL;DR The main distinction between database systems is not the language used to query the database, but rather the consistency model of the system. The Venn diagram should be two intersecting sets - ...
Vérace's user avatar
  • 29.9k
24 votes

If NoSQL stands for "Not only SQL", is SQL a subset of NoSQL?

NoSQL was, is, and always will be a vague buzzword, not a well-defined term. Its origin is related to a historical trend: at the beginning of this century, if you said "database", the ...
IMSoP's user avatar
  • 397
16 votes

Why is it more difficult to horizontally scale a relational database than a NoSQL database like MongoDB?

Why is it more difficult to horizontally scale a relational database than a NoSQL database I'm afraid your assumption is a bit flawed. It is not difficult to scale a relational (or any other) DBMS ...
mustaccio's user avatar
  • 26k
15 votes
Accepted

Is a single-table relational database similar to a simple NoSQL database?

No - a single table relational database is more akin to a spreadsheet than a NoSQL database. NoSQL is not about having no schema, it's more about having a flexible schema! Wikipedia puts it very ...
Vérace's user avatar
  • 29.9k
14 votes

Best database and table design for billions of rows of data

It amazes me me that nobody here has mentioned benchmarking - that is until @EvanCarroll came along with his excellent contribution! If I were you, I would spend some time (and yes, I know it's a ...
Vérace's user avatar
  • 29.9k
13 votes
Accepted

What kind of database to store Quotes

If the app remains as simple as this...then is a NoSQL approach If the app is truly as simple as this and will remain so, and dev time is the key factor, then a pair of files and a bash script (or a ...
David Spillett's user avatar
11 votes
Accepted

Redesign the storage of large amounts of sensor data

Since (a) the information you are working with appears to be, in a of itself, a very valuable organizational resource, and (b) the volume of data will be considerable, I would decidedly (c) build a ...
MDCCL's user avatar
  • 8,520
11 votes
Accepted

Why Cassandra is not considered a relational database?

Cassandra is not an RDBMS because it does not support the relational data model. The fundamental assumption of the relational model is that all data is represented as mathematical n-ary relations, an ...
David Browne - Microsoft's user avatar
10 votes
Accepted

ALLOW FILTERING vs NOT; Cassandra Data Model Question

Right now the table only has about 320k records and I can use ALLOW FILTERING with no problem, but I realize this might not always be the case. So here's the thing: Cassandra is very good at querying ...
Aaron's user avatar
  • 4,075
9 votes
Accepted

What is the point of column families?

I've just uncovered some interesting information from the RocksDB FAQ. (RocksDB is a K-V store.) Here are some relevant extracts. Q: What are column families used for? A: The most common ...
Noldorin's user avatar
  • 443
9 votes

Is a single-table relational database similar to a simple NoSQL database?

Yes, you can use a single-table relational database as if it were a NoSQL database. You can also use a spoon to paint your house. You have to use a tool the way it was intended else you're fighting ...
Michael Green's user avatar
8 votes

MongoDB vs RDBMS DBA duties

DBA is a small acronym but a large role. At various times I have seen a DBA look after storage network VMs compute nodes installation and configuration of the above backups, and testing restores DR ...
Michael Green's user avatar
8 votes
Accepted

Storing arrays in MYSQL?

My question really is, is MySQL a good choice if you want to store multiple values in one column? Codd's Rules Rule 2: The guaranteed access rule: Each and every datum (atomic value) in a relational ...
Vérace's user avatar
  • 29.9k
8 votes

If NoSQL stands for "Not only SQL", is SQL a subset of NoSQL?

SQL is a language. NoSQL is an approach to choosing database systems. It makes no sense to compare them. The origin of "Not Only SQL" lies in the fact that not too long ago, when you had ...
Jörg W Mittag's user avatar
7 votes
Accepted

MongoDB vs RDBMS DBA duties

I have been in the NoSQL field since last 3 years. Being a MongoDB DBA you need to work closely with Development and Ops teams. Following are the things you need to do as day to day tasks as a MongoDB ...
Atish's user avatar
  • 344
7 votes

Best database and table design for billions of rows of data

If you have not already, take a look at a time series DBMS, since it is optimized for storing and querying data where the primary focus is the date/time type. Typically time series databases are used ...
FloorDivision's user avatar
6 votes

Best database and table design for billions of rows of data

Clearly this is not a NoSQL problem, but I would suggest that while an RDBMS solution would work, I think an OLAP approach will fit much better and given the very limited data ranges involved, I would ...
Paul Smith's user avatar
5 votes
Accepted

How do you design schema for tracking change history in NoSQL?

Add three attributes to the project and asset documents - WrittenWhen, WrittenBy and IsActive. Populate these in the "real" data. When a record changes copy it but with IsActive flag flipped. Retain ...
Michael Green's user avatar
5 votes

Why can't I take RDBMS Database Design practices, and use for NoSQL MongoDB Database Designs?

If you want to use a relational structure, and you're thinking about doing it like this with a NoSQL system, use an RDBMS, that's what they're designed to do. Using the right tool for the job will ...
Hannah Vernon's user avatar
  • 70k
5 votes

Are relational databases even viable for large eCommerce platforms?

There are really three different questions in here. Q: Is a table with hundreds of thousands of rows huge? Generally, no. For example, you can download the Stack Overflow public data export, and it ...
Brent Ozar's user avatar
4 votes
Accepted

How to store data having a fixed number of columns and a variable number of rows

Based on your image, something like the following will be fine (using a RDBMS solution): create table articles ( article_id integer PRIMARY KEY, title varchar(256) NOT NULL, url ...
Philᵀᴹ's user avatar
  • 31.8k
4 votes
Accepted

What is considered as a document in mongoDB?

In the context of MongoDB, a document means any piece of valid BSON* (binary JSON). The word does not have the everyday meaning of, say, an MS Word file or PDF. A collection is a container for zero ...
Michael Green's user avatar
4 votes

how to design a good database design which suits for both RDMS and noSQL

The database designs for relational and document stores will be entirely different. You will not be able to take one and simply implement it in the other. What will endure, however, is the data model ...
Michael Green's user avatar
4 votes

Why are NoSQL databases not ACID compliant?

Not having ACID properties means that the database works well on clusters. But ACID is something very fundamental. How can a database work well if there is no atomicity, consistency, isolation and ...
Md Haidar Ali Khan's user avatar
4 votes

How to design a graph database in this scenario?

Should I create 1 MandatoryData for each book? Yes. If you cannot get rid of MandatoryData object, you should have one such object per book. Otherwise I would simplify the model and move any ...
filiprem's user avatar
  • 6,539
4 votes

How to tell if a project needs a NoSQL database solution?

How to tell if a project needs NoSQL database solution? I think that SQL limits define it, if project needs bigger database or bigger fields than it needs NoSQL. Am I right? You are not ...
Josh Darnell's user avatar
  • 29.2k
4 votes

Microsoft Column Store Database?

It sounds like what you're looking for is a "non-relational database," often called a document or NoSQL database. Amazon defines that here: What is a document database A document database is a ...
Josh Darnell's user avatar
  • 29.2k

Only top scored, non community-wiki answers of a minimum length are eligible