The tag has no wiki summary.

learn more… | top users | synonyms

0
votes
0answers
14 views

Architecture for low-latency RESTful server on Amazon EC2 with Node.js and a database

This is a question for DBAs with a lot of practical experience with large scale deployments. We are trying to build an app with very low latency. (We really want to create the best user experience, ...
0
votes
1answer
47 views

Optimization for 100 GB Real-time info table

I have a table of device information feed which is filled by data from remote device units. There will be like 1000 units each sending an update per minute which will be stored as a row in that ...
2
votes
1answer
45 views

SQL Server many-to-many relationship, row reuse?

Below is your typical many-to-many relationship. Pretend there are 2 tasks that have the exact same Tags. because of foreign key/primary key relationship constraints, there is no way for the rows in ...
1
vote
1answer
26 views

Are there any inherently non-partitionable datasets?

Are there any practical datasets that, by their nature, cannot be partitioned in any practical way? I am wondering about this because such datasets would be ill-suited to scaling out on commodity ...
0
votes
0answers
6 views

What is the preferred mode of replication for a MySQL Cluster?

Reading about MySQL Cluster, I'd like to know about what are the preferred models of replication. I understand that it supports asynchronous replication out of the box, but I'm not sure of how it ...
0
votes
0answers
15 views

Would an INNER JOIN query in a sharded relational database hit the same shard when using a compound primary key?

In this question I have asked about modulus calculation in sharded tables, this design is used: CREATE TABLE Orders (order_id INT PRIMARY KEY, ...); CREATE TABLE LineItems (order_id INT, lineitem_id ...
0
votes
0answers
9 views

Modulus calculation for compound primary keys in MySQL Cluster (and other sharded databases)

In MySQL Cluster, and generally in sharded databases, how is a compound primary key's modulus calculated? Say, for example, there are two tables that have primary keys like so: CREATE TABLE Orders ...
0
votes
0answers
21 views

Performance and maintainablity of a large MySQL table with hundreds of columns

For a project of mine I need to have a table with around 400 hundred columns of type nullable indexed VARCHAR(128), this table is going to store at least 50 million rows and has one column of type ...
1
vote
1answer
45 views

Reference implementation of a scalable storage system with strong consistency?

Is there a storage system with these properties? horizontally scalable - can handle arbitrary read and write throughput if you add enough nodes to the cluster consistent, in the sense that all reads ...
2
votes
4answers
90 views

What are the risks of building a database outside IT control?

I've been looking for the best StackExchange community to ask this. I hope it fits well here. As a risk analyst in a trading business, I'm often confronted with large volumes of data, way beyond the ...
3
votes
1answer
317 views

Confused about load balancing and horizontal scaling in postgresql [closed]

Please correct me if I am wrong but I guess handling more requests and load by adding more machines or balancing the load between multiple servers is horizontal scalling. So, if I add more servers, ...
0
votes
1answer
95 views

Scaling out mongodb on the same storage

We are planning to have some tests to predict how mongo db operates and functions with 2 terabyte of inforamtion on a single collection. So the plan is to have 15-20 sharded machines (on the same ...
3
votes
1answer
1k views

PostgreSQL High Availability/Scalability using HAProxy and PGBouncer

I have multiple PostgreSQL servers for a web application. Typically one master and multiple slaves in hot standby mode (asynchronous streaming replication). I use PGBouncer for connection pooling: ...
1
vote
0answers
90 views

PostgreSQL how to create a scalable location-based query

I need suggestions because I am not good enough. I have a DB in PostgreSQL that runs on AWS (Amazon Web Services). I have a table "user_location" where there are stored the locations of each user ...
0
votes
0answers
68 views

MySQL: Optimizing for large but discrete data sets

In brief: I'm developing a database that handles GTFS datasets from multiple transit agencies. Each dataset contains millions of rows in the stop_times.txt file (and thus its corresponding table). ...
1
vote
1answer
259 views

optimizing MySQL for traffic analytics system

background : I've developed a URL shortener system like Bitly with same features , so the system also tracks clickers info and represent as graphs to the person who has shorten the link as ...
3
votes
2answers
171 views

Scalability: Desiging a message table

This question might be deadly simple but I wanted to make sure I am going through the correct path and not making any simple and irrevertible mistakes. My project will be based on messaging and it ...
1
vote
1answer
362 views

Mitigating MySQL replication delay in single master/multiple slave setup

For a particular web application that requires high availability I am thinking of scaling out by setting-up MySQL replication with one "write" master and multiple "read" slaves (as explained here ...
0
votes
0answers
95 views

MySql Scalability, Master Slave replication a good idea? Read description please

I have a database on mysql which is scaling on daily basis and right now its on 5M records and in a month it may double itself. I have a table containing users data which is the largest one. I have ...
4
votes
4answers
324 views

In Memory Database for high transaction and volume spikes on a website [closed]

I am looking at the IA for a hypothetical application scenario, for example: A high demand application such as an online ticketing website where there can be demand for 100,000 tickets to be ...
-3
votes
1answer
72 views

Are database servers less scalable than web servers? [closed]

Isn't it possible to scale database servers through virtualization, like forever? I like to keep most of the workload on database server because it optimizes queries and the architecture decreases the ...
1
vote
2answers
2k views

Stress test MySQL with queries captured with general log in MySQL

Is there any tool available which can do stress testing using the log file created by MySQL general log? After a lot of search on google I found few stress testing tools which only use some benchmarks ...
4
votes
3answers
3k views

Is it better to seperate a big query into multiple smaller queries?

There are situations which require to have really big query joining several tables together with sub select statements in them to produce the desired results. My question is, should we consider using ...
41
votes
5answers
5k views

Why can't RDBM's cluster the way NoSQL does?

One of the big plusses for nosql DBMS is that they can cluster more easily. Supposedly with NoSQL you can create hundreds of cheap machines that store different pieces of data and query it all at ...
0
votes
1answer
86 views

Looking for a scalable relational database [closed]

I'm looking for a database that achieves the following: fully ACID. relational with joins. has transactions. allows the administrator to specify sharding rules (for example: users from Europe should ...
3
votes
1answer
1k views

Very Slow MySQL Queries, even with indexes

I have a relatively large 4-deep relational data setup like this: client_applications: (potentially 1,000's of records)    - ...    - account_id    - ...
4
votes
1answer
1k views

What are the practical limitations on a column family in Cassandra?

In Cassandra, it's not recommended to have more than a few thousand column families, let's say 2,000 for the sake of argument. In cases where more than 2,000 types of data need to be persisted, one ...
2
votes
1answer
196 views

How to scale MySQL transaction wrapped queries?

So I'm working on an app that wraps many of it's SQL queries in a transaction for it's SQL usage and I'm trying to think of some ways to scale the DB layer up.... Up until now we've been throwing ...
4
votes
1answer
646 views

Postgres scalability - What is the impact of connection pooling?

We had a question over on Server Fault that raised an interesting question: In light of the scalability improvements in Postgres 9.2 scalability improvements, is it better to use connection pooling ...
1
vote
2answers
109 views

How to store massive quantities of machine generated data?

I am tasked with developing a system that stores massive quantities of machine generated data (syslogs) from a number of server farms and I am wondering what tools you fine folks use such instances. ...
1
vote
1answer
686 views

What are the pros and cons of using CLIENT_RESULT_CACHE_SIZE and RESULT_CACHE_MODE

I have multiple questions For Oracle Database 11g Release 11.2.0.1.0 what are the benefits of using CLIENT_RESULT_CACHE_SIZE and RESULT_CACHE_MODE. How will the CLIENT side cache be kept sync if ...
4
votes
2answers
2k views

Proper storage size estimation technique for PostgreSQL

We are preparing a PostgreSQL database for production usage and we need to estimate the storage size for this database. We are a team of developers with low expertise on database administration, so we ...
1
vote
4answers
494 views

database sharding on mysql

I need to do database sharding(horizontal) based on MySQL. My database schema is as following: I have tables A,B,C A,B is Global(which need not to be sharding) C have huge data so it need to be ...
3
votes
0answers
222 views

SQL Server 2008 R2 - Merge Replication - Can the Distribution Database size cause replication problems?

Setup: We have a SQL Server 2008 R2 instance running on a central server. The server is a merge-replication publisher. We have 'N' subsribers running SQL Server 2008 Express. 'N' can grow to any ...
5
votes
1answer
5k views

Scalability limitations of PostgreSQL and MySQL

I've heard that performance of non-sharded relational database such as MySQL or PostgreSQL "breaks" beyond 10 TB. I suspect that limits as such do exist as one would not come up with Netezza, ...
1
vote
1answer
280 views

MySQL table locks solution -> InnoDb / Partitions

I'm experiencing many table locks in a MyISAM MySQL table that is pretty big (Constx10xGB and 15M rows). It's mainly caused due many heavy inserts. My current thoughts to optimize: Changing to ...
3
votes
1answer
426 views

NuoDB with Amazon S3

I think about developing my new mobile and web app against NuoDB. I believe in their technology. The thing that I care the most is backup and restore in case of data corruption and read / write speed ...
3
votes
2answers
567 views

Scalable DBaaS that can scale to hundreds of terabytes with easy backup and restore

I've gone over many DBaaS solutions, including Xeround (limited to 50G, in memory), NuoDB (currently in beta 7), DrawnToScale (not out yet), MongoDB (in memory), RavenDB (for ASP.NET, but NoSQL) and ...
1
vote
2answers
1k views

MySQL Cluster On Cloud vs MySQL Cluster On Dedicated Cluster

I saw this interesting article : http://mikaelronstrom.blogspot.com/2012/02/105bn-qpm-using-mysql-cluster-72.html and it made me get interested in MySQL Cluster. After little chat with hosting ...
5
votes
3answers
594 views

How to store many smallish time-series in a relational DB?

Input Data Multiple Tests Beds generate measurement data of various complexity. In it's most basic form, not considering any meta-data, one measurement on a Test Bed will be a small (1 - a few ...
2
votes
2answers
164 views

A Huge Database: Picking the right schema and data model

We want to store some genomic variant data but there are some problems, more important ones like problem of the data's immense size and variability. 1) Variant data can be huge. For example, a single ...
1
vote
2answers
679 views

Which is better for long term database scalability: adding columns or having key/value store

For application, which has few (5-10) settings at the moment but will have a lot more (up to 100) in the future which would be better approach: to add new columns to settings table every time new ...
2
votes
2answers
498 views

Is it OK to create hundreds of databases in SQL Azure versus one big one and run the risk of deadlocking

I need to create highly scalable solution - where field devices in thousands of sites are delivering data in real time to a back end system, and SQL Azure seems to fit the bill nicely in terms of ...
3
votes
1answer
242 views

to what “scale” of applications does nosql support?

Recently i came across the Nosql database concept, though i learnt how to and why of it, i did not get a proper answer to the question , to what scale of project does it support? Will it support ...
5
votes
3answers
6k views

Why can't relational databases meet the scales of Big Data?

It is often repeated that the Big Data problem is that relational databases can not scale to process the massive volumes of data that are now being created. But what are these scalability limitations ...
9
votes
3answers
2k views

Database suggestion for a social network/ knowledge base community?

I am looking into various database types and DBMS's for a new project I am wanting to start in the summer. I have built systems in MySQL and postgreSQL , now I am wanting to expand my knowledge and ...
4
votes
3answers
1k views

MySQL tables/databases into different disks

I am aware that there is a possibility to have a single database server with many tables/databases, and you can separate some of those tables/databases into separate disks. Lets say you have 5 ...
7
votes
3answers
2k views

SQL Server - Multiple running totals

I've got a base table with transactions and I need to create a table with running totals. I need them to be per account and and also have a few running totals for each account (depending on the ...
4
votes
3answers
5k views

How does one scale SQL Server 2008 or 2012?

How does one scale SQL Server 2008 (or 2012)? At its basic, I understand there are two options: Scale up: If CPU bound, I can clearly see going from 1 CPU core to 2 to 4. Or if RAM usage rockets, ...
4
votes
1answer
735 views

Database as a service (DbaaS) over dimensions--performance, scalability, & reliability?

MySql Database as a service: Amazon Xeround Another eg. PostgreSQL Database-as-a-Service. You can get the list of similar products here. Does anyone have details on their performance, ...