Tag Info

Hot answers tagged

87

Distributed Database Systems 101 Or, Distributed Databases - what the FK does 'web scale' actually mean? Distributed database systems are complex critters and come in a number of different flavours. If I dig deep in to the depths of my dimly remembered distributed systems papers I did at university (roughly 15 years ago) I'll try to explain some of the ...


45

NoSQL stands for "Not only SQL" and usually means that the database is not a relational database, which have been very popular the last decades. The reason why NoSQL has been so popular the last few years is mainly because, when a relational database grows out of one server, it is no longer that easy to use. In other words, they don't scale out very well ...


27

Are you familiar with the concept of a Key/Value Pair? Presuming you're familiar with Java or C# this is in the language as a map/hash/datatable/KeyValuePair (the last is in the case of C#) The way it works is demonstrated in this little sample chart: Color Red Age 18 Size Large Name Smith Title The Brown Dog Where ...


26

If I was going to put this into SQL Server, I would suggest a table something like: CREATE TABLE tcp_traffic ( tcp_traffic_id bigint constraint PK_tcp_traffic primary key clustered IDENTITY(1,1) , tcp_flags smallint /* at most 9 bits in TCP, so use SMALLINT */ , src_as int /* Since there are less than 2 billion A.S.'s possible, use INT ...


24

In a company I work for we are dealing with similar amount of data (around 10 TBs of realtime searchable data). We solve this with Cassandra and I would like to mention couple of ideas that will allow you to do O(1) search on a multi TBs database. This is not specific to Cassandra db though, you can use it with other db as well. Theory Shard your data. ...


17

NoSQL is a very broad term and typically is referred to as meaning "Not Only SQL." The term is dropping out of favor in the non-RDBMS community. You'll find that NoSQL database have few common characteristics. They can be roughly divided into a few categories: key/value stores Bigtable inspired databases (based on the Google Bigtable paper) Dynamo ...


15

Relational databases can cluster like NoSQL solutions. Maintaining ACID properties may make this more complex and one must be aware of the tradeoffs made to maintain these properties. Unfortunately, exactly what the trade-offs are depends on the workload and of course the decisions made while designing the database software. For example, a primarily OLTP ...


13

Horizontal Scaling Horizontal Scaling is essentially building out instead of up. You don't go and buy a bigger beefier server and move all of your load onto it, instead you buy 1+ additional servers and distribute your load across them. Horizontal scaling is used when you have the ability to run multiple instances on servers simultaneously. Typically it is ...


13

In general, for such a structured dataset I suspect you could write a custom data format which was faster for most daily operations (i.e. small data pulls from an arbitrary time). The benefit of moving to a standard DB tool is likely in some of the extras, for example ad hoc queries, multiple access, replication, availability etc. It's also easier to hire ...


13

In SQL terms, a NoSQL database is a single table with two columns: one being the (Primary) Key, and the other being the Value. And that's it, that's all the NoSQL magic. You would use NoSQL for one main reason: scalability. If your application needs to handle millions of queries per second, the only way to achieve it is to add more servers. That is very ...


10

There are a lot of differences between the two of them. MongoDB is more like a traditional RDBMS (nobody shoot). CouchDB performs master-master replication. It's pretty well documented in this much ballyhooed blog post.


10

NoSQL is a kind of database that doesn't have a fixed schema like a traditional RDBMS does. With the NoSQL databases the schema is defined by the developer at run time. They don't write normal SQL statements against the database, but instead use an API to get the data that they need. The NoSQL databases can usually scale across different physical servers ...


10

I assume you have basic understanding of NoSQL movement and non-relational databases models. Key Value store is one of the non-relation database model, like graph, document oriented database models. Key Value stores and the NoSQL movement In general, SQL managed to deal with specially structured data and allowed highly dynamic queries ...


10

BigTable doesn't use SQL (a query language) so SQL can't be used directly to query the database. And BigTable doesn't have "relations" in the same way as relational databases, it's more like bare tables. If you want to get data from two tables, you have to do two lookups, and combine the result set in the application code. In other words the "join" ...


10

IMO you are making what is probably a pretty common mistake when it comes to web pages which is to assume that the answer to performance problems due to initial result size on MySQL is to jump to NoSQL solutions often with little understanding of what the tradeoffs are or how to use them appropriately and effectively. I would be surprised if a well-tuned db ...


9

The Quick Answer - Yes. Happens all the time. There are plenty of good solutions. What solutions are already in your environment? I am helping one client that takes their web site/session activity information from their web application, they write it to xml then deserialize that xml into Hadoop. They then use Hive on top of Hadoop to create aggregations and ...


9

If you can't scale a major RDBMS then your database design (includes indexing, queries and the like) or hardware is wrong. The choice of platform is almost irrelevant. It is that simple. Especially when you mention "few hundred megabytes" which implies low volumes (I mean a few dozen writes per second)


9

CAP is basically a continuum along which BASE and ACID are on opposite ends. CAP is Consistency, Availability, and Partition tolerance. Basically you can pick 2 of those but you can't do all 3. ACID focuses on Consistency and availability. BASE focuses on Partition tolerance and availability and throws consistency out the window.


9

Option 1 There are several reasons for this, which I'll explain below. First, here's how to do it. Use your choice of standard RDBMS platform. Set up your schema with several user-configurable fields, and make your application facilitate the configuration on a per-tenant basis. From the per-tenant metadata, you can create a per-tenant view of their data, ...


9

To be very fast, the database should only use memory and not disc since disc operations usually takes much longer time. But then your data aren't persistent in case of a crash. What you could do is assynchronous disk operations, by that most of your writes will be persistent but it's not guaranteed that the last few writes are persistent. If this is okey for ...


8

Others have explained this, but I'm going to take a stab anyway. A key/value database stores data by a primary key. This lets us uniquely identify a record in a bucket. Since all values are unique, lookups are incredibly fast: it's always a simple disk seek. The value is just any kind of value. The way the data is stored is opaque to the database itself. ...


8

This question is really far too vague to answer effectively. There are dozens of "NoSQL" data stores out there which have various use cases. Here is a 10,000 foot view of what's out there. In my mind, there are basically 3 main categories of NoSQL data stores commonly used, key/value stores, document databases, and big data (hadoop). This is a somewhat ...


8

The key words here are: "heavily updated" "in the table for 2-3 hours". Point 1. is indication for a lower fill factor, while 2. is the opposite. It helps performance if multiple row versions are stored on the same data page. H.O.T. updates would achieve that. Read here or here. They need some wiggle room on the data page - like dead tuples or space ...


8

The fundamental answer is that the consistency model is different. I am writing this to expand ConcernedOfTunbridge's answer which really ought to be the reference point for this. The basic point of the ACID consistency model is that it makes a bunch of fundamental guarantees as to the state of the data globally within the system. These guarantees are ...


7

As far as I know there are no "nosql" databases that promise ACID transactions, so for banking purposes they are a non starter. Referential consistency support is not usually in their key feature sets either. mySQL claims ACID transactions when using innodb tables, but I believe there are some caveats around that which may be show stoppers (any mix of other ...


6

If you have a relational database, then you can easily experiment with this: create table keyvalue (my_key varchar2(255), my_value varchar2(255)); create unique index ix_keyvalue on keyvalue (my_key, my_value); This is how all databases used to be, with Berkeley DBM being a good example, from 1979. Since then, things have advanced (you can have many ...


6

Basically dispensing with the relational setup, with primary and foreign keys, and with the additional overhead involved in keeping transactional safety, often gives you extreme increases in performance. However this is not unique to the new databases/datastores, as eg MySQL has been tuned to perform at "NoSQL levels" by bypassing layers. In short, you can ...



Only top voted, non community-wiki answers of a minimum length are eligible