Tag Info

Hot answers tagged

26

If I was going to put this into SQL Server, I would suggest a table something like: CREATE TABLE tcp_traffic ( tcp_traffic_id bigint constraint PK_tcp_traffic primary key clustered IDENTITY(1,1) , tcp_flags smallint /* at most 9 bits in TCP, so use SMALLINT */ , src_as int /* Since there are less than 2 billion A.S.'s possible, use INT ...


21

In a company I work for we are dealing with similar amount of data (around 10 TBs of realtime searchable data). We solve this with Cassandra and I would like to mention couple of ideas that will allow you to do O(1) search on a multi TBs database. This is not specific to Cassandra db though, you can use it with other db as well. Theory Shard your data. ...


6

This really depends on your requirements. Many systems built on top of open source products are rock solid. Note that your example does not impress me - Twitter is not reliable at all by my customers' standards - it has frequent downtimes, it loses tweets and followers, its search is not 100% correct etc. I work in finances, and my job is to develop robust ...


4

Great question! There's a handful of ways of doing this, depending on how deeply you want to be involved in the configuration and long term maintenance. You certainly could use Cassandra on top of an encrypted filesystem, such as TrueCrypt or eCryptfs, and it should function normally. While possible, it's a little complicated to set up encrypted filesystem ...


3

The best way to think of a cassandra database is not as a set of databases on different nodes, but as one single database. Adding the column to the first node adds it to all other nodes. The actual number of times your data is written is determined by your replication strategy and replication factor, but each copy of the data will be the same. Thus, if you ...


3

You can try to make each node unique for some of the ports, but this isn't really a good path to go down. Cassandra nodes need to see each other on the ports listed here for the best setup: http://www.datastax.com/documentation/cassandra/1.2/index.html#cassandra/security/secureFireWall_r.html One option that I have seen work well is a host VPN like ...


3

As with any other piece of software, it's a question of features vs. cost. Enterprise class commercial databases (Oracle, DB2, and SQL Server primarily) generally have a variety of technical and non-technical features that are very helpful for large companies. Focusing just on the non-technical, commercial databases will generally have a much more mature ...


3

I'm not a fan. It's about as good an idea as creating a relational table named OrdersOrCustomers with columns defined for both. The storage-engine penalty is slightly lower in Cassandra because of the sparse-cell storage under the hood, but it's still bad practice. This bites you later when you want to map/reduce over your data; each task will have to ...


2

A few db's that come close: 1: Postgres-XC, meets most areas except that sharding takes some design effort 2: VoltDB, meets most areas except for data larger than memory The key question though is why do you need the autosharding? Anything automatic adds complexity and if you are big enough to require sharding, you can probably afford to have the ...


2

I would recommend HBase. You can store all the raw data in one or more HBase tables, depending on what you need to query. HBase can handle large data-sets and does auto-sharding through region splits. In addition, if you design row keys well, you can get extremely fast, even O(1) queries. Note that if you are retrieving a large data set, that is still ...


2

Let's say I add a column to a dynamic column set. I write this column to a single node. I then later add another column to the same column store, but this time to another node If you're talking about adding 2 different columns (with different names) to a single row/partition (column set?) then there will be no merge - those are different columns. If ...


2

You can index collection types in cassandra 2.1 and later. You are after: SELECT * FROM <table> WHERE <field> CONTAINS <value_in_list/map/set> Detailed example: cqlsh> USE ks; cqlsh:ks> CREATE TABLE data_points ( id text PRIMARY KEY, created_at timestamp, previous_event_id varchar, ...


1

It is not a strange question. I have the same problem every day; how to get things to run faster, faster. We use MS SQL Server 2012 with 512 GB RAM and a SAN at 50TB. We have about ten tables with more than a billion rows, our largest table is 75 billion rows. What really helps is ensuring that the right datatypes are used; no bigints when tinyints will do. ...


1

To monitor - OpsCenter: http://www.datastax.com/what-we-offer/products-services/datastax-opscenter To query - DevCenter: http://www.datastax.com/what-we-offer/products-services/devcenter


1

In regard to a standard implementation of Cassandra the answer is Yes. So everything in the cluster regardless of the data center would have access. From the Apache Cassandra site: Cassandra's support for replicating across multiple datacenters is best-in-class, providing lower latency for your users and the peace of mind of knowing that you can ...


1

I second the recommendation to look at Informix TimeSeries. IBM literature claims TimeSeries can store this kind of information in 1/5th the space and perform 5 times as fast as traditional relational tables. Additional benefits would be the Virtual Table Interface that can make TimeSeries data appear like traditional relational tables to the end user ...


1

Said this : ...we are not opposed to looking at proprietary solutions for this project I suggest consider IBM Informix database + TimeSeries datablade. Opposite what some people says, Informix is alive and going very well. The last version was released last month (March/2013 ,version 12.10). TimeSeries is like a "plugin" (no-cost) able to deal ...


1

I started this out as just a comment reply to Randy but was running out of space. Innodb slave side only seems odd to me. The main reason you should look toward that engine is data consistency and disaster recovery. This means you want it on the master if you have to choose one. Further, row level locking innodb provides will better enable you to ...


1

There's a lot to consider. I'm going to narrow this down to a single problem and offer one solution, although there's many approaches. Problem: You need AD-HOC queries on lots of rows. First, setup a server to handle writes and a replicated slave (running some flavor of MySQL 5.5.x) to handle reads. My personal preference is Percona Server. Your mileage ...


1

Cassandra is a Columnar Store NoSQL Database, Which goes beyond the Key Value pairs. A Column family in cassandra is a container for rows, like table in any RDBMS. And These are Stored and represented in Json Formats. Below are some links where you can get more info: http://wiki.apache.org/cassandra/DataModel http://www.datastax.com/docs/0.8/ddl/index ...


1

If you are going to go multi-headed (which you probably need to if you really need 3K active connections) I'd probably look at Riak or maybe Cassandra. It really depends on what your app does as to how well these will fit, but from what you have described I think it would fit in something like Riak. That said, a sharded approach seems pretty doable, if you ...


1

If you are using MySQL as the main database, you may want to consider using a Star Topology via MySQL Replication. Now, before you say UGHHH, ROFL and OMG to MySQL Replication, hear me out. A star topology allows you to write to one DB server (called a Distribution Mster [DM]) and send the SQL commands to several DB servers. How do you setup such a DB ...



Only top voted, non community-wiki answers of a minimum length are eligible