Hot answers tagged

34

If I was going to put this into SQL Server, I would suggest a table something like: CREATE TABLE tcp_traffic ( tcp_traffic_id bigint constraint PK_tcp_traffic primary key clustered IDENTITY(1,1) , tcp_flags smallint /* at most 9 bits in TCP, so use SMALLINT */ , src_as int /* Since there are less than 2 billion A.S.'s possible, use INT ...


34

In a company I work for we are dealing with similar amount of data (around 10 TBs of realtime searchable data). We solve this with Cassandra and I would like to mention couple of ideas that will allow you to do O(1) search on a multi TBs database. This is not specific to Cassandra db though, you can use it with other db as well. Theory Shard your data. ...


12

The whole point of NoSQL is that it doesn't use the same type of datastructures as an SQL database. It requires a whole different way of thinking and approach with your app, so there isn't something to handily copy the schema over because it is going to be unique to your solution.


9

You can index collection types in cassandra 2.1 and later. You are after: SELECT * FROM <table> WHERE <field> CONTAINS <value_in_list/map/set> Detailed example: cqlsh> USE ks; cqlsh:ks> CREATE TABLE data_points ( id text PRIMARY KEY, created_at timestamp, previous_event_id varchar, ...


7

cqlsh -k mykeyspace -e 'COPY fromTable(columnNames) TO STDOUT' | head -n -1 | cqlsh -k mykeyspace -e 'COPY toTable(columnNames) FROM STDIN'


6

This really depends on your requirements. Many systems built on top of open source products are rock solid. Note that your example does not impress me - Twitter is not reliable at all by my customers' standards - it has frequent downtimes, it loses tweets and followers, its search is not 100% correct etc. I work in finances, and my job is to develop robust ...


5

Sounds like my Cassandra Cluster Admin is exactly what you want! Take a look at it here: https://github.com/sebgiroux/Cassandra-Cluster-Admin


5

Cassandra 1.0 uses a minimum of 1MB of heap per CF. So, 1000 or 2000 CFs will be okay for typical heap sizes, but 10000 will probably not be. JVM GC does poorly with very large heaps; I recommend staying under 8GB.


5

I created a chat program a while back which is available on github https://github.com/akc42/MBChat which has some of the characteristics you are talking about. In particular it provides both rooms which people can enter and discuss on an open way, and whisper boxes where a few people can get together and discuss things. When people enter a room, or join ...


5

Despite the mainstream of noSQL databases IMHO the decision about adopting such technology should be made according to the achievements needed according to the information stored, not only attending to the performance you currently have. This means that maybe your best option is to stick to the SQL database and improve your HW. But additionally I read ...


4

The best way to think of a cassandra database is not as a set of databases on different nodes, but as one single database. Adding the column to the first node adds it to all other nodes. The actual number of times your data is written is determined by your replication strategy and replication factor, but each copy of the data will be the same. Thus, if you ...


4

I'm not a fan. It's about as good an idea as creating a relational table named OrdersOrCustomers with columns defined for both. The storage-engine penalty is slightly lower in Cassandra because of the sparse-cell storage under the hood, but it's still bad practice. This bites you later when you want to map/reduce over your data; each task will have to ...


4

You can try to make each node unique for some of the ports, but this isn't really a good path to go down. Cassandra nodes need to see each other on the ports listed here for the best setup: http://www.datastax.com/documentation/cassandra/1.2/index.html#cassandra/security/secureFireWall_r.html One option that I have seen work well is a host VPN like ...


4

Great question! There's a handful of ways of doing this, depending on how deeply you want to be involved in the configuration and long term maintenance. You certainly could use Cassandra on top of an encrypted filesystem, such as TrueCrypt or eCryptfs, and it should function normally. While possible, it's a little complicated to set up encrypted filesystem ...


3

As with any other piece of software, it's a question of features vs. cost. Enterprise class commercial databases (Oracle, DB2, and SQL Server primarily) generally have a variety of technical and non-technical features that are very helpful for large companies. Focusing just on the non-technical, commercial databases will generally have a much more mature ...


3

There's a lot to consider. I'm going to narrow this down to a single problem and offer one solution, although there's many approaches. Problem: You need AD-HOC queries on lots of rows. First, setup a server to handle writes and a replicated slave (running some flavor of MySQL 5.5.x) to handle reads. My personal preference is Percona Server. Your mileage ...


3

To monitor - OpsCenter: http://www.datastax.com/what-we-offer/products-services/datastax-opscenter To query - DevCenter: http://www.datastax.com/what-we-offer/products-services/devcenter


3

Let's say I add a column to a dynamic column set. I write this column to a single node. I then later add another column to the same column store, but this time to another node If you're talking about adding 2 different columns (with different names) to a single row/partition (column set?) then there will be no merge - those are different columns. If ...


3

There are a couple of concepts which need to be distinguished. One is about structure and the other about schema. Structured data is one where the application knows in advance the meaning of each byte it receives. A good example is measurements from a sensor. In contrast a Twitter stream is unstructured. Schema is about how much of the structure is ...


3

You can use check Java Heap memory for each node. Like Total Java Heap Memory and Using Java Heap Memory. One most important is CPU Utilization for each node. Set alerts for errors. system.log have lots of information about. You may set alerts for data disk and log disk. Heartbeat check of servers like if you dont receive in few min it will get alerts to ...


3

[EDIT - added several public dataset sites]. First off, there is no real evidence that NoSQL databases are "better" at handling large datasets than traditional (OldSQL) RDBMSs. Check out Ted Dziuba's article about how he can't wait for NoSQL to die. He makes the point that Walmart continue to use RDBMSs - and they're not a small company! He says that NoSQL ...


2

For Graph database on Cassandra have a look at TitanDB: What you need is already implemented in TitanDB. Implementing your own Graph Database is not trivial, and would be very time consuming. In most cases, a proven solution is best. (BTW, I am not involved in TitanDB development or business.) I have no idea about your use case, but I do not see a reason to ...


2

In my clusters, I use jolokia so I can access JMX interface using HTTP. This way I can create scripts with Python that monitor Cassandra and send alerts to Nagios. What you should monitor: Java HEAP and GC Compactions pending Flushes pending Hinted Handoff pending Gossip Tasks pending Read / Write latency Key cache hit rate Down end point count ...


2

According to the Cassandra repair documentation, nodetool repair should be run in the following situations: As a best practice, you should schedule repairs weekly. Note: If deletions never occur, you should still schedule regular repairs. Be aware that setting a column to null is a delete. During node recovery. For example, when bringing a node back into ...


2

In looking through the documentation, I am having a hard time finding a reference for this, but the short answer is "yes" you need to run nodetool repair on each node in your cluster. The closest I can find is the documentation on repairing nodes which suggests that you should not run repair on more than one node in your cluster at a time. You can also run ...


2

My largest obstacle right now with Cassandra are the no-join, no order by (since everything is ordered by key for you), and mass update. If your current data structure depends on joins (or aggregation) to pull off the queries you need you won't find a nosql option that includes these. You do have some options: Consider reviewing your current data ...


2

Although Cassandra allows the definition of "columns" within a "table", these are much less strict than a relational database schema. As it says here different rows in the same column family do not have to share the same set of columns, and a column may be added to one or multiple rows at any time. In this world, the new value is only added to the ...


2

If you are going to go multi-headed (which you probably need to if you really need 3K active connections) I'd probably look at Riak or maybe Cassandra. It really depends on what your app does as to how well these will fit, but from what you have described I think it would fit in something like Riak. That said, a sharded approach seems pretty doable, if you ...


2

Did you take a look at sstableloader http://www.datastax.com/documentation/cassandra/2.0/cassandra/tools/toolsBulkloader_t.html ?


2

See https://github.com/thinkaurelius/titan/wiki/Using-Cassandra scroll down to "Cassandra Specific Configuration", and use property "storage.keyspace=" to use multiple keyspaces in cassandra. Each keyspace will hold a graph. (default keyspace is 'titan'. So, if you want 3 graphs, create 3 config files, with diff "storage.keyspace" property. When you open a ...



Only top voted, non community-wiki answers of a minimum length are eligible