Hot answers tagged nosql
61
Distributed Database Systems 101
Or, Distributed Databases - what the FK does 'web scale' actually mean?
Distributed database systems are complex critters and come in a number of different flavours. If I dig deep in to the depths of my dimly remembered distributed systems papers I did at university (roughly 15 years ago) I'll try to explain some of the ...
24
If I was going to put this into SQL Server, I would suggest a table something like:
CREATE TABLE tcp_traffic
(
tcp_traffic_id bigint constraint PK_tcp_traffic primary key clustered IDENTITY(1,1)
, tcp_flags smallint /* at most 9 bits in TCP, so use SMALLINT */
, src_as int /* Since there are less than 2 billion A.S.'s possible, use INT ...
18
In a company I work for we are dealing with similar amount of data (around 10 TBs of realtime searchable data). We solve this with Cassandra and I would like to mention couple of ideas that will allow you to do O(1) search on a multi TBs database. This is not specific to Cassandra db though, you can use it with other db as well.
Theory
Shard your data. ...
12
Relational databases can cluster like NoSQL solutions. Maintaining ACID properties may make this more complex and one must be aware of the tradeoffs made to maintain these properties. Unfortunately, exactly what the trade-offs are depends on the workload and of course the decisions made while designing the database software.
For example, a primarily OLTP ...
8
The fundamental answer is that the consistency model is different. I am writing this to expand ConcernedOfTunbridge's answer which really ought to be the reference point for this.
The basic point of the ACID consistency model is that it makes a bunch of fundamental guarantees as to the state of the data globally within the system. These guarantees are ...
7
IMO you are making what is probably a pretty common mistake when it comes to web pages which is to assume that the answer to performance problems due to initial result size on MySQL is to jump to NoSQL solutions often with little understanding of what the tradeoffs are or how to use them appropriately and effectively.
I would be surprised if a well-tuned db ...
6
CAP is basically a continuum along which BASE and ACID are on opposite ends.
CAP is Consistency, Availability, and Partition tolerance. Basically you can pick 2 of those but you can't do all 3.
ACID focuses on Consistency and availability.
BASE focuses on Partition tolerance and availability and throws consistency out the window.
5
The answer to your question is "no." And no, I don't think it will ever be possible at least in any helpful way.
Relational databases are designed to operate over sets of tuples. Non-relational databases are not. This means that one of the key tradeoffs is being able to have very fast access (with "good enough" consistency controls) to single pieces of ...
4
Barry Morris here from NuoDB. Answers below:
1) DBaaS (can be self hosted too) the can easily scale to terabytes of
data (ie. big data)
NuoDB is a downloadable product that runs anywhere (laptop, rack, public cloud). We have not announced DBAAS at this point. NuoDB uses Key/Value stores at the storage layer so supported DB sizes will be related ...
4
Given that you say you believe this data model screams "relational" the thought that comes to mind is, "I have a nail and I would really like to use a crescent wrench.... Can you? Yes. Are you aware of the tradeoffs? Doubtful."
The basic point about NoSQL is that it makes sense when your data use cases are fairly simple and you have encapsulated that ...
4
My answer won't be as well-written as the previous one, but here goes.
Michael Stonebraker of Ingres fame has created a MPP shared-nothing column-store (Vertica) and a MPP shared-nothing New SQL database (VoltDB) which distributes data between different nodes in a cluster and maintains ACID. Vertica has since been bought by HP.
I believe other New SQL ...
4
Tens of terabytes of data in 3 tables sounds expensive in many ways.
Do you have budgetary constraints for this solution? Storing tens of terabytes of data in an in-memory NoSQL platform may be exactly what you're after, but it's likely to require dozens of servers.
You might get some good ideas from High Scalability. For example, it's interesting to see ...
3
The key words here are:
"heavily updated"
"in the table for 2-3 hours".
Point 1. is indication for a lower fill factor, while 2. is the opposite. It helps performance if multiple row versions are stored on the same data page. H.O.T. updates would achieve that. Read here or here. They need some wiggle room on the data page - like dead tuples or space ...
3
Typically No-SQL solutions were implemented due to massive scale and not needing 100% ACID compliance. Flattening data out and de-normalizing it for performance gains were critical in getting us to where we are today with solutions such as GFS, Hadoop, Mongo, Basho RIAK, etc.
Your case doesn't scream No-SQL, if anything you'd probably be better served ...
3
The first primer which will save you research is by Martin Fowler.
NoSQL Distilled. Available on Kindle from Amazon.
Reviews the types of NoSQL databases.
Key Value Pair {Redis, Raik}, Uses {Web Sessions, Shopping Cart, User Profiles}
Document {MongoDB, CouchDB...}, Uses {EventLogging, CMS, Blog}
Column {Cassandra, Hbase...}, Uses {EventLogging, CMS, ...
3
100 million rows in a table does not make a database "huge" by today's standards
This is also not MySql related in most ways - because you leave pretty fat the server software level and talk hardware.
The standard answer to this is that it all depends on your workload profile. Is it transactional, reporting, OLAP?
You mentioned that your largest table is ...
3
This seems a little outside the scope of a StackExchange question. However.....
NoSQL databases are, typically, build to resolve specific issues with the relational model. The most common issue addressed is scalability. Are you planning to need to handle terabytes of data? However, because they're all designed to address different aspects of certain ...
3
There are a number of reasons not to use MongoDB as a pure key-value store, and there are some reasons to consider it. Mongo is optimized as a document store - it indexes all the fields in a document, and has rich primitives for JSON objects and hierarchies. You can use it as a key-value store, but the single-threaded nature means you won't be getting good ...
2
I would recommend trying PostgreSQL to be honest. You can always move away later but that's a good place to start. It's free. It's not a db specializing in running code written for other db's (like MySQL is). It is enterprise-grade. It is advanced and you can take the features it offers and go very far. With regard to advanced data modelling it is on a ...
2
even that the way to distribute data in graph databases is not that easy as in key-value stores, where the keys are distributed by ranges, there are techniques to distribute a graph.
InfiniteGraph (made by Objectivity) for instance is a highly distributed graph database. It uses a P2P technology to increase read and write performance by growing the ...
2
Pretty much any database you choose will work for this.
You need to think carefully about concurrency issues no matter which DB you're using, as you'll have many concurrent workers inserting new records, sending messages and flagging them as sent, etc. You're highly likely to need a queueing system like PgQ, because queueing is harder than you think and any ...
2
You should probably start by reading up on some of the basics. Google is a good aid. Good words to search for would be for example Big Data, Hadoop or RavenDB. I barely know the basics myself, but basically NOSQL typically refers to a non-relational, file-based database system built on clusters of drives.
NOSQL solutions such as Hadoop are used when the ...
2
Oracle NoSQL Database integrates well with the Oracle database. You can run an SQL query in Oracle Database that can reference data in the Oracle database as well as NoSQL Database.
Please see the Oracle NoSQL Database documentation for more details on external tables support.
You can also get an overview of the latest release here:
...
2
I would recommend HBase. You can store all the raw data in one or more HBase tables, depending on what you need to query. HBase can handle large data-sets and does auto-sharding through region splits.
In addition, if you design row keys well, you can get extremely fast, even O(1) queries. Note that if you are retrieving a large data set, that is still ...
2
Which version did you download and install? If you have 64-bit Windows 8, you should download and install the build marked with "2008R2+". Their download page indicates that:
The Windows 2008+ build uses newer features of Windows to enhance
performance. Use this build if you are running with 64-bit Windows
Server 2008 R2, Windows 7, or greater.
...
1
I second the recommendation to look at Informix TimeSeries. IBM literature claims TimeSeries can store this kind of information in 1/5th the space and perform 5 times as fast as traditional relational tables.
Additional benefits would be the Virtual Table Interface that can make TimeSeries data appear like traditional relational tables to the end user ...
1
Said this :
...we are not opposed to looking at proprietary solutions for this
project
I suggest consider IBM Informix database + TimeSeries datablade.
Opposite what some people says, Informix is alive and going very well. The last version was released last month (March/2013 ,version 12.10).
TimeSeries is like a "plugin" (no-cost) able to deal ...
1
I think the first thing to do is make sure you know exactly what you need. Chances are you want to start with SQL as AceCTO suggested. My reasoning for starting there is different however.
The basic point is that standard SQL is a pretty mature technology. You know what you get. It is robust also, so you don't have to deal with the questions that things ...
1
The short answer is that NoSQL, NewSQL, OODB or just plain SQL are all options, but SQL is probably the most viable. Many NoSQL implementations can be configured for ACID compliance at the cost of performance/scalability, but they will most likely still scale beyond most reasonable SQL implementations. Another option is to use application sharded SQL to ...
1
Yes, you don't need a NoSQL database. As the earlier answer said, you don't have a scale issue. Moreover you have requirement for a richer query set. NoSQLs don't offer a very rich query capabilities. If you normalize your database schema, the future requirements can be properly accommodated by adding more tables. NoSQLs specialize in having loose schema ...
Only top voted, non community-wiki answers of a minimum length are eligible
