Tag Info

Hot answers tagged

30

In a company I work for we are dealing with similar amount of data (around 10 TBs of realtime searchable data). We solve this with Cassandra and I would like to mention couple of ideas that will allow you to do O(1) search on a multi TBs database. This is not specific to Cassandra db though, you can use it with other db as well. Theory Shard your data. ...


30

If I was going to put this into SQL Server, I would suggest a table something like: CREATE TABLE tcp_traffic ( tcp_traffic_id bigint constraint PK_tcp_traffic primary key clustered IDENTITY(1,1) , tcp_flags smallint /* at most 9 bits in TCP, so use SMALLINT */ , src_as int /* Since there are less than 2 billion A.S.'s possible, use INT ...


12

The whole point of NoSQL is that it doesn't use the same type of datastructures as an SQL database. It requires a whole different way of thinking and approach with your app, so there isn't something to handily copy the schema over because it is going to be unique to your solution.


7

You can index collection types in cassandra 2.1 and later. You are after: SELECT * FROM <table> WHERE <field> CONTAINS <value_in_list/map/set> Detailed example: cqlsh> USE ks; cqlsh:ks> CREATE TABLE data_points ( id text PRIMARY KEY, created_at timestamp, previous_event_id varchar, ...


6

This really depends on your requirements. Many systems built on top of open source products are rock solid. Note that your example does not impress me - Twitter is not reliable at all by my customers' standards - it has frequent downtimes, it loses tweets and followers, its search is not 100% correct etc. I work in finances, and my job is to develop robust ...


5

Sounds like my Cassandra Cluster Admin is exactly what you want! Take a look at it here: https://github.com/sebgiroux/Cassandra-Cluster-Admin


5

Cassandra 1.0 uses a minimum of 1MB of heap per CF. So, 1000 or 2000 CFs will be okay for typical heap sizes, but 10000 will probably not be. JVM GC does poorly with very large heaps; I recommend staying under 8GB.


5

I created a chat program a while back which is available on github https://github.com/akc42/MBChat which has some of the characteristics you are talking about. In particular it provides both rooms which people can enter and discuss on an open way, and whisper boxes where a few people can get together and discuss things. When people enter a room, or join ...


5

cqlsh -k mykeyspace -e 'COPY fromTable(columnNames) TO STDOUT' | head -n -1 | cqlsh -k mykeyspace -e 'COPY toTable(columnNames) FROM STDIN'


4

Great question! There's a handful of ways of doing this, depending on how deeply you want to be involved in the configuration and long term maintenance. You certainly could use Cassandra on top of an encrypted filesystem, such as TrueCrypt or eCryptfs, and it should function normally. While possible, it's a little complicated to set up encrypted filesystem ...


4

I'm not a fan. It's about as good an idea as creating a relational table named OrdersOrCustomers with columns defined for both. The storage-engine penalty is slightly lower in Cassandra because of the sparse-cell storage under the hood, but it's still bad practice. This bites you later when you want to map/reduce over your data; each task will have to ...


4

The best way to think of a cassandra database is not as a set of databases on different nodes, but as one single database. Adding the column to the first node adds it to all other nodes. The actual number of times your data is written is determined by your replication strategy and replication factor, but each copy of the data will be the same. Thus, if you ...


4

You can try to make each node unique for some of the ports, but this isn't really a good path to go down. Cassandra nodes need to see each other on the ports listed here for the best setup: http://www.datastax.com/documentation/cassandra/1.2/index.html#cassandra/security/secureFireWall_r.html One option that I have seen work well is a host VPN like ...


3

Let's say I add a column to a dynamic column set. I write this column to a single node. I then later add another column to the same column store, but this time to another node If you're talking about adding 2 different columns (with different names) to a single row/partition (column set?) then there will be no merge - those are different columns. If ...


3

As with any other piece of software, it's a question of features vs. cost. Enterprise class commercial databases (Oracle, DB2, and SQL Server primarily) generally have a variety of technical and non-technical features that are very helpful for large companies. Focusing just on the non-technical, commercial databases will generally have a much more mature ...


3

There's a lot to consider. I'm going to narrow this down to a single problem and offer one solution, although there's many approaches. Problem: You need AD-HOC queries on lots of rows. First, setup a server to handle writes and a replicated slave (running some flavor of MySQL 5.5.x) to handle reads. My personal preference is Percona Server. Your mileage ...


3

[EDIT - added several public dataset sites]. First off, there is no real evidence that NoSQL databases are "better" at handling large datasets than traditional (OldSQL) RDBMSs. Check out Ted Dziuba's article about how he can't wait for NoSQL to die. He makes the point that Walmart continue to use RDBMSs - and they're not a small company! He says that NoSQL ...


2

If you are using MySQL as the main database, you may want to consider using a Star Topology via MySQL Replication. Now, before you say UGHHH, ROFL and OMG to MySQL Replication, hear me out. A star topology allows you to write to one DB server (called a Distribution Mster [DM]) and send the SQL commands to several DB servers. How do you setup such a DB ...


2

I would recommend HBase. You can store all the raw data in one or more HBase tables, depending on what you need to query. HBase can handle large data-sets and does auto-sharding through region splits. In addition, if you design row keys well, you can get extremely fast, even O(1) queries. Note that if you are retrieving a large data set, that is still ...


2

Said this : ...we are not opposed to looking at proprietary solutions for this project I suggest consider IBM Informix database + TimeSeries datablade. Opposite what some people says, Informix is alive and going very well. The last version was released last month (March/2013 ,version 12.10). TimeSeries is like a "plugin" (no-cost) able to deal ...


2

A few db's that come close: 1: Postgres-XC, meets most areas except that sharding takes some design effort 2: VoltDB, meets most areas except for data larger than memory The key question though is why do you need the autosharding? Anything automatic adds complexity and if you are big enough to require sharding, you can probably afford to have the ...


2

Did you take a look at sstableloader http://www.datastax.com/documentation/cassandra/2.0/cassandra/tools/toolsBulkloader_t.html ?


2

I started this out as just a comment reply to Randy but was running out of space. Innodb slave side only seems odd to me. The main reason you should look toward that engine is data consistency and disaster recovery. This means you want it on the master if you have to choose one. Further, row level locking innodb provides will better enable you to ...


2

To monitor - OpsCenter: http://www.datastax.com/what-we-offer/products-services/datastax-opscenter To query - DevCenter: http://www.datastax.com/what-we-offer/products-services/devcenter


2

The easiest (non-invasive) way is probably to download the tarball installation (you'll need to select either a Mac or Linux-based OS for it to allow you to download the tarball). Based-on your mention of disabling the service, I'm going to guess that you want to accomplish this on Windows. If that's not the case, please indicate so in the comments. ...


2

According to the Cassandra repair documentation, nodetool repair should be run in the following situations: As a best practice, you should schedule repairs weekly. Note: If deletions never occur, you should still schedule regular repairs. Be aware that setting a column to null is a delete. During node recovery. For example, when bringing a node back into ...


2

My largest obstacle right now with Cassandra are the no-join, no order by (since everything is ordered by key for you), and mass update. If your current data structure depends on joins (or aggregation) to pull off the queries you need you won't find a nosql option that includes these. You do have some options: Consider reviewing your current data ...


2

Although Cassandra allows the definition of "columns" within a "table", these are much less strict than a relational database schema. As it says here different rows in the same column family do not have to share the same set of columns, and a column may be added to one or multiple rows at any time. In this world, the new value is only added to the ...


1

It is not a strange question. I have the same problem every day; how to get things to run faster, faster. We use MS SQL Server 2012 with 512 GB RAM and a SAN at 50TB. We have about ten tables with more than a billion rows, our largest table is 75 billion rows. What really helps is ensuring that the right datatypes are used; no bigints when tinyints will do. ...


1

Have you tried Helenos: https://github.com/tomekkup/helenos It seems good



Only top voted, non community-wiki answers of a minimum length are eligible