The tag has no wiki summary.

learn more… | top users | synonyms

1
vote
1answer
15 views

Postgres or MapReduce to Match Reciprocal Needle Pairs from a Stack of Needles?

I have a Postgres (latest version) table with about a million lines (a_lat, a_lon, b_lat, b_lon) that each have about a dozen criteria. These lines go together in reciprocal pairs (A - B, B - A), ...
0
votes
0answers
13 views

MySQL 5.6 or 5.7 sharding geospatial data

I have ran into a problem where I need to start to scale horizontally. With my database I am performing only nearest point searches with my data. So from a lat/long inoput I find all the nearest ...
3
votes
2answers
67 views

Why is vertical scaling expensive?

As you can find here Best Practices For Horizontal Application Scaling, vertical scaling is much more expensive than horizontal scaling: vertical scaling cost increase exponentially whereas with ...
1
vote
1answer
57 views

Insert/ Update Scalability (more records, continually longer lookups — solution?)

posted this before but never received a clear answer. I'm working on an ETL project and am basically feeding JSON records of "tickets" into a MySQL database. (actually come to think of it, may be ...
0
votes
1answer
53 views

Scalability question

There is a DB {MySql} with 5 tables A, BIG, C, D, E. Their grow factor is about 1/100/1/1/1. The table BIG has Insert/Read/Update request ratio about 1/10/2. The Inserts and Updates 'cannot fail'. ...
-2
votes
1answer
73 views

How to scale a write intesive application with a SQL Server 2014? [closed]

I am studying about many databases lately and the one thing I can't get the hang of is how to scale a write intensive application in SQL Server, I'm talking about native features, not third party ...
4
votes
3answers
129 views

Scalable query for running counts of events within x previous days

I already posted this question on stackoverflow but I thought that I might get a better answer here. I have a table storing millions of events occurring to users: ...
0
votes
0answers
62 views

Is mysql cluster a viable choice for real time data availability when it comes to data in high volumes (petabytes)?

I've been reading a lot of articles promoting MySql Cluster as a big data storage solution for real time applications, one of which is this one . But as I went through the Mysql Cluster docs, I saw ...
0
votes
1answer
76 views

GROUP BY on two large joined tables

I have two tables: Person ------ PersonID (PK) GroupID DeletedDate PersonData ---------- PersonID (PK) FieldID (PK) Value The fields are custom-created; hence the schema. I am trying to run a ...
3
votes
1answer
512 views

Scaling PostgreSQL TRIGGER(s)

How Postgres triggers mechanism scales ? We have a large PostgreSQL installation and we are trying to implement an event based system using log tables and TRIGGER(s). Basically we would like to ...
0
votes
1answer
81 views

schema for mysql database with 30million rows

We am building a SaaS backend for restaurants. We integrate directly with POS, so each POS keeps sending customer orders that we store in a following table in mysql. CREATE TABLE name ( `id` ...
0
votes
2answers
114 views

Standalone MongoDB to MongoDB Shard Setup

I'm learning MongoDB currently so maybe the title of this question is worded poorly, but I'm looking to understand the scaling of Mongo better. I did some research on what DB would be best for my ...
2
votes
1answer
66 views

How do you generate large OLTP and reporting workloads? [closed]

My current skillset lies towards plain vanilla data warehousing, with < 20k users, our setups are rather simple.. I am looking for ways to enhance my technical skills, however, as much theory as I ...
0
votes
0answers
23 views

How to figure out how to split a table using the proper scheme

If you have a big table that you want to partition and you have 2 candidates as the partitioning key (in the sense that 50% of the queries use one column and 50% of the queries use the other column ...
1
vote
1answer
42 views

Denormalization Strategies: When is it appropriate to create redundancy for faster querying?

So I sometimes see databases purposely add redundant information for faster querying. I saw that this was mentioned here but I was wondering when would this be better than indexing, or creating a ...
0
votes
1answer
516 views

Architecture for low-latency RESTful server on Amazon EC2 with Node.js and a database

This is a question for DBAs with a lot of practical experience with large scale deployments. We are trying to build an app with very low latency. (We really want to create the best user experience, ...
0
votes
1answer
113 views

Optimization for 100 GB Real-time info table

I have a table of device information feed which is filled by data from remote device units. There will be like 1000 units each sending an update per minute which will be stored as a row in that ...
2
votes
1answer
1k views

SQL Server many-to-many relationship, row reuse?

Below is your typical many-to-many relationship. Pretend there are 2 tasks that have the exact same Tags. because of foreign key/primary key relationship constraints, there is no way for the rows in ...
1
vote
1answer
33 views

Are there any inherently non-partitionable datasets?

Are there any practical datasets that, by their nature, cannot be partitioned in any practical way? I am wondering about this because such datasets would be ill-suited to scaling out on commodity ...
0
votes
0answers
84 views

Would an INNER JOIN query in a sharded relational database hit the same shard when using a compound primary key?

In this question I have asked about modulus calculation in sharded tables, this design is used: CREATE TABLE Orders (order_id INT PRIMARY KEY, ...); CREATE TABLE LineItems (order_id INT, lineitem_id ...
1
vote
0answers
58 views

Modulus calculation for compound primary keys in MySQL Cluster (and other sharded databases)

In MySQL Cluster, and generally in sharded databases, how is a compound primary key's modulus calculated? Say, for example, there are two tables that have primary keys like so: CREATE TABLE Orders ...
1
vote
1answer
56 views

Reference implementation of a scalable storage system with strong consistency?

Is there a storage system with these properties? horizontally scalable - can handle arbitrary read and write throughput if you add enough nodes to the cluster consistent, in the sense that all reads ...
2
votes
4answers
178 views

What are the risks of building a database outside IT control?

I've been looking for the best StackExchange community to ask this. I hope it fits well here. As a risk analyst in a trading business, I'm often confronted with large volumes of data, way beyond the ...
3
votes
1answer
2k views

Confused about load balancing and horizontal scaling in postgresql [closed]

Please correct me if I am wrong but I guess handling more requests and load by adding more machines or balancing the load between multiple servers is horizontal scalling. So, if I add more servers, ...
0
votes
1answer
139 views

Scaling out mongodb on the same storage

We are planning to have some tests to predict how mongo db operates and functions with 2 terabyte of inforamtion on a single collection. So the plan is to have 15-20 sharded machines (on the same ...
5
votes
1answer
5k views

PostgreSQL High Availability/Scalability using HAProxy and PGBouncer

I have multiple PostgreSQL servers for a web application. Typically one master and multiple slaves in hot standby mode (asynchronous streaming replication). I use PGBouncer for connection pooling: ...
1
vote
0answers
204 views

PostgreSQL how to create a scalable location-based query

I need suggestions because I am not good enough. I have a DB in PostgreSQL that runs on AWS (Amazon Web Services). I have a table "user_location" where there are stored the locations of each user ...
0
votes
0answers
80 views

MySQL: Optimizing for large but discrete data sets

In brief: I'm developing a database that handles GTFS datasets from multiple transit agencies. Each dataset contains millions of rows in the stop_times.txt file (and thus its corresponding table). ...
1
vote
1answer
416 views

optimizing MySQL for traffic analytics system

background : I've developed a URL shortener system like Bitly with same features , so the system also tracks clickers info and represent as graphs to the person who has shorten the link as ...
3
votes
2answers
216 views

Scalability: Desiging a message table

This question might be deadly simple but I wanted to make sure I am going through the correct path and not making any simple and irrevertible mistakes. My project will be based on messaging and it ...
1
vote
1answer
637 views

Mitigating MySQL replication delay in single master/multiple slave setup

For a particular web application that requires high availability I am thinking of scaling out by setting-up MySQL replication with one "write" master and multiple "read" slaves (as explained here ...
0
votes
0answers
140 views

MySql Scalability, Master Slave replication a good idea? Read description please

I have a database on mysql which is scaling on daily basis and right now its on 5M records and in a month it may double itself. I have a table containing users data which is the largest one. I have ...
4
votes
4answers
912 views

In Memory Database for high transaction and volume spikes on a website [closed]

I am looking at the IA for a hypothetical application scenario, for example: A high demand application such as an online ticketing website where there can be demand for 100,000 tickets to be ...
-3
votes
1answer
81 views

Are database servers less scalable than web servers? [closed]

Isn't it possible to scale database servers through virtualization, like forever? I like to keep most of the workload on database server because it optimizes queries and the architecture decreases the ...
1
vote
2answers
3k views

Stress test MySQL with queries captured with general log in MySQL

Is there any tool available which can do stress testing using the log file created by MySQL general log? After a lot of search on google I found few stress testing tools which only use some benchmarks ...
8
votes
3answers
7k views

Is it better to separate a big query into multiple smaller queries?

There are situations which require having really big query joining several tables together with sub select statements in them to produce the desired results. My question is, should we consider ...
50
votes
5answers
9k views

Why can't RDBM's cluster the way NoSQL does?

One of the big plusses for nosql DBMS is that they can cluster more easily. Supposedly with NoSQL you can create hundreds of cheap machines that store different pieces of data and query it all at ...
0
votes
1answer
104 views

Looking for a scalable relational database [closed]

I'm looking for a database that achieves the following: fully ACID. relational with joins. has transactions. allows the administrator to specify sharding rules (for example: users from Europe should ...
3
votes
1answer
2k views

Very Slow MySQL Queries, even with indexes

I have a relatively large 4-deep relational data setup like this: client_applications: (potentially 1,000's of records)    - ...    - account_id    - ...
6
votes
1answer
3k views

What are the practical limitations on a column family in Cassandra?

In Cassandra, it's not recommended to have more than a few thousand column families, let's say 2,000 for the sake of argument. In cases where more than 2,000 types of data need to be persisted, one ...
2
votes
1answer
292 views

How to scale MySQL transaction wrapped queries?

So I'm working on an app that wraps many of it's SQL queries in a transaction for it's SQL usage and I'm trying to think of some ways to scale the DB layer up.... Up until now we've been throwing ...
4
votes
1answer
1k views

Postgres scalability - What is the impact of connection pooling?

We had a question over on Server Fault that raised an interesting question: In light of the scalability improvements in Postgres 9.2 scalability improvements, is it better to use connection pooling ...
1
vote
2answers
132 views

How to store massive quantities of machine generated data?

I am tasked with developing a system that stores massive quantities of machine generated data (syslogs) from a number of server farms and I am wondering what tools you fine folks use such instances. ...
1
vote
1answer
1k views

What are the pros and cons of using CLIENT_RESULT_CACHE_SIZE and RESULT_CACHE_MODE

I have multiple questions For Oracle Database 11g Release 11.2.0.1.0 what are the benefits of using CLIENT_RESULT_CACHE_SIZE and RESULT_CACHE_MODE. How will the CLIENT side cache be kept sync if ...
5
votes
2answers
3k views

Proper storage size estimation technique for PostgreSQL

We are preparing a PostgreSQL database for production usage and we need to estimate the storage size for this database. We are a team of developers with low expertise on database administration, so we ...
1
vote
4answers
670 views

database sharding on mysql

I need to do database sharding(horizontal) based on MySQL. My database schema is as following: I have tables A,B,C A,B is Global(which need not to be sharding) C have huge data so it need to be ...
3
votes
0answers
274 views

SQL Server 2008 R2 - Merge Replication - Can the Distribution Database size cause replication problems?

Setup: We have a SQL Server 2008 R2 instance running on a central server. The server is a merge-replication publisher. We have 'N' subsribers running SQL Server 2008 Express. 'N' can grow to any ...
13
votes
1answer
10k views

Scalability limitations of PostgreSQL and MySQL

I've heard that performance of non-sharded relational database such as MySQL or PostgreSQL "breaks" beyond 10 TB. I suspect that limits as such do exist as one would not come up with Netezza, ...
1
vote
1answer
389 views

MySQL table locks solution -> InnoDb / Partitions

I'm experiencing many table locks in a MyISAM MySQL table that is pretty big (Constx10xGB and 15M rows). It's mainly caused due many heavy inserts. My current thoughts to optimize: Changing to ...
4
votes
1answer
468 views

NuoDB with Amazon S3

I think about developing my new mobile and web app against NuoDB. I believe in their technology. The thing that I care the most is backup and restore in case of data corruption and read / write speed ...