Hot answers tagged

35

In a company I work for we are dealing with similar amount of data (around 10 TBs of realtime searchable data). We solve this with Cassandra and I would like to mention couple of ideas that will allow you to do O(1) search on a multi TBs database. This is not specific to Cassandra db though, you can use it with other db as well. Theory Shard your data. ...


34

If I was going to put this into SQL Server, I would suggest a table something like: CREATE TABLE tcp_traffic ( tcp_traffic_id bigint constraint PK_tcp_traffic primary key clustered IDENTITY(1,1) , tcp_flags smallint /* at most 9 bits in TCP, so use SMALLINT */ , src_as int /* Since there are less than 2 billion A.S.'s possible, use INT ...


12

The whole point of NoSQL is that it doesn't use the same type of datastructures as an SQL database. It requires a whole different way of thinking and approach with your app, so there isn't something to handily copy the schema over because it is going to be unique to your solution.


9

You can index collection types in cassandra 2.1 and later. You are after: SELECT * FROM <table> WHERE <field> CONTAINS <value_in_list/map/set> Detailed example: cqlsh> USE ks; cqlsh:ks> CREATE TABLE data_points ( id text PRIMARY KEY, created_at timestamp, previous_event_id varchar, ...


7

cqlsh -k mykeyspace -e 'COPY fromTable(columnNames) TO STDOUT' | head -n -1 | cqlsh -k mykeyspace -e 'COPY toTable(columnNames) FROM STDIN'


6

This really depends on your requirements. Many systems built on top of open source products are rock solid. Note that your example does not impress me - Twitter is not reliable at all by my customers' standards - it has frequent downtimes, it loses tweets and followers, its search is not 100% correct etc. I work in finances, and my job is to develop robust ...


6

Sounds like my Cassandra Cluster Admin is exactly what you want! Take a look at it here: https://github.com/sebgiroux/Cassandra-Cluster-Admin


5

Cassandra 1.0 uses a minimum of 1MB of heap per CF. So, 1000 or 2000 CFs will be okay for typical heap sizes, but 10000 will probably not be. JVM GC does poorly with very large heaps; I recommend staying under 8GB.


5

I created a chat program a while back which is available on github https://github.com/akc42/MBChat which has some of the characteristics you are talking about. In particular it provides both rooms which people can enter and discuss on an open way, and whisper boxes where a few people can get together and discuss things. When people enter a room, or join ...


5

Despite the mainstream of noSQL databases IMHO the decision about adopting such technology should be made according to the achievements needed according to the information stored, not only attending to the performance you currently have. This means that maybe your best option is to stick to the SQL database and improve your HW. But additionally I read ...


4

Great question! There's a handful of ways of doing this, depending on how deeply you want to be involved in the configuration and long term maintenance. You certainly could use Cassandra on top of an encrypted filesystem, such as TrueCrypt or eCryptfs, and it should function normally. While possible, it's a little complicated to set up encrypted filesystem ...


4

I'm not a fan. It's about as good an idea as creating a relational table named OrdersOrCustomers with columns defined for both. The storage-engine penalty is slightly lower in Cassandra because of the sparse-cell storage under the hood, but it's still bad practice. This bites you later when you want to map/reduce over your data; each task will have to ...


4

You can try to make each node unique for some of the ports, but this isn't really a good path to go down. Cassandra nodes need to see each other on the ports listed here for the best setup: http://www.datastax.com/documentation/cassandra/1.2/index.html#cassandra/security/secureFireWall_r.html One option that I have seen work well is a host VPN like ...


4

The best way to think of a cassandra database is not as a set of databases on different nodes, but as one single database. Adding the column to the first node adds it to all other nodes. The actual number of times your data is written is determined by your replication strategy and replication factor, but each copy of the data will be the same. Thus, if you ...


4

To monitor - OpsCenter: http://www.datastax.com/what-we-offer/products-services/datastax-opscenter To query - DevCenter: http://www.datastax.com/what-we-offer/products-services/devcenter


4

You can use check Java Heap memory for each node. Like Total Java Heap Memory and Using Java Heap Memory. One most important is CPU Utilization for each node. Set alerts for errors. system.log have lots of information about. You may set alerts for data disk and log disk. Heartbeat check of servers like if you dont receive in few min it will get alerts to ...


3

Let's say I add a column to a dynamic column set. I write this column to a single node. I then later add another column to the same column store, but this time to another node If you're talking about adding 2 different columns (with different names) to a single row/partition (column set?) then there will be no merge - those are different columns. If ...


3

There's a lot to consider. I'm going to narrow this down to a single problem and offer one solution, although there's many approaches. Problem: You need AD-HOC queries on lots of rows. First, setup a server to handle writes and a replicated slave (running some flavor of MySQL 5.5.x) to handle reads. My personal preference is Percona Server. Your mileage ...


3

As with any other piece of software, it's a question of features vs. cost. Enterprise class commercial databases (Oracle, DB2, and SQL Server primarily) generally have a variety of technical and non-technical features that are very helpful for large companies. Focusing just on the non-technical, commercial databases will generally have a much more mature ...


3

In looking through the documentation, I am having a hard time finding a reference for this, but the short answer is "yes" you need to run nodetool repair on each node in your cluster. The closest I can find is the documentation on repairing nodes which suggests that you should not run repair on more than one node in your cluster at a time. You can also run ...


3

[EDIT - added several public dataset sites]. First off, there is no real evidence that NoSQL databases are "better" at handling large datasets than traditional (OldSQL) RDBMSs. Check out Ted Dziuba's article about how he can't wait for NoSQL to die. He makes the point that Walmart continue to use RDBMSs - and they're not a small company! He says that NoSQL ...


3

In my clusters, I use jolokia so I can access JMX interface using HTTP. This way I can create scripts with Python that monitor Cassandra and send alerts to Nagios. What you should monitor: Java HEAP and GC Compactions pending Flushes pending Hinted Handoff pending Gossip Tasks pending Read / Write latency Key cache hit rate Down end point count ...


3

There are a couple of concepts which need to be distinguished. One is about structure and the other about schema. Structured data is one where the application knows in advance the meaning of each byte it receives. A good example is measurements from a sensor. In contrast a Twitter stream is unstructured. Schema is about how much of the structure is ...


3

Schema design in Cassandra, for efficient tables, will grate against your RDBMS experience; for efficiency, the Cassandra prefers denormalization, not normalization. By this, I mean that if you have some user information and you want to look up that data using two different primary keys, then using Cassandra, it actually is better to use two tables (and ...


2

If you are going to go multi-headed (which you probably need to if you really need 3K active connections) I'd probably look at Riak or maybe Cassandra. It really depends on what your app does as to how well these will fit, but from what you have described I think it would fit in something like Riak. That said, a sharded approach seems pretty doable, if you ...


2

If you are using MySQL as the main database, you may want to consider using a Star Topology via MySQL Replication. Now, before you say UGHHH, ROFL and OMG to MySQL Replication, hear me out. A star topology allows you to write to one DB server (called a Distribution Mster [DM]) and send the SQL commands to several DB servers. How do you setup such a DB ...


2

A few db's that come close: 1: Postgres-XC, meets most areas except that sharding takes some design effort 2: VoltDB, meets most areas except for data larger than memory The key question though is why do you need the autosharding? Anything automatic adds complexity and if you are big enough to require sharding, you can probably afford to have the ...


2

Said this : ...we are not opposed to looking at proprietary solutions for this project I suggest consider IBM Informix database + TimeSeries datablade. Opposite what some people says, Informix is alive and going very well. The last version was released last month (March/2013 ,version 12.10). TimeSeries is like a "plugin" (no-cost) able to deal ...


2

I would recommend HBase. You can store all the raw data in one or more HBase tables, depending on what you need to query. HBase can handle large data-sets and does auto-sharding through region splits. In addition, if you design row keys well, you can get extremely fast, even O(1) queries. Note that if you are retrieving a large data set, that is still ...


2

I started this out as just a comment reply to Randy but was running out of space. Innodb slave side only seems odd to me. The main reason you should look toward that engine is data consistency and disaster recovery. This means you want it on the master if you have to choose one. Further, row level locking innodb provides will better enable you to ...



Only top voted, non community-wiki answers of a minimum length are eligible