The tag has no usage guidance.

learn more… | top users | synonyms

1
vote
0answers
22 views

How to design for scalable set operations?

This is the requirement for an educational application to teach words to language learners: Learner can declare that he/she knows a given word Learner can memorize the given word (to be managed by ...
6
votes
2answers
9k views

PostgreSQL High Availability/Scalability using HAProxy and PGBouncer

I have multiple PostgreSQL servers for a web application. Typically one master and multiple slaves in hot standby mode (asynchronous streaming replication). I use PGBouncer for connection pooling: ...
0
votes
0answers
40 views

MongoDB daily document growth rate

Is MongoDB feasibly suited to handle 90,000,000 documents per day? Version 3.2. I did some rough estimates on my data size. There will be about 90,000,000 recorded events per day, each a maximum size ...
24
votes
1answer
15k views

Scalability limitations of PostgreSQL and MySQL

I've heard that performance of non-sharded relational database such as MySQL or PostgreSQL "breaks" beyond 10 TB. I suspect that limits as such do exist as one would not come up with Netezza, ...
0
votes
1answer
116 views

schema for mysql database with 30million rows

We am building a SaaS backend for restaurants. We integrate directly with POS, so each POS keeps sending customer orders that we store in a following table in mysql. CREATE TABLE name ( `id` ...
3
votes
1answer
55 views

Are Availability Groups feasible for 4 TB of data?

Currently we have a database which is 4 TB in size. We would like to put it in a 2014 Availability Group to scale out reads. Data gets changed a lot inside this database. I have following questions: ...
0
votes
1answer
45 views

MySQL - design help for scale

I will be using Amazon Web Services RDS service to host a MySQL 5.7 server (Currently AWS do not support 5.7, but they have announced they are working on implementing this version and it will be out ...
3
votes
0answers
46 views

Design a many-to-many table for scale

I am thinking about scaling issues with many-to-many join tables. I have a constructed example as following: Many users have many wishlists. On every wishlist are many items. If I scale this ...
1
vote
2answers
138 views

Scaling for MariaDB / Percona / Galera Scaling

I ventured into the world of NoSQL database and as great as they are the lack of true ACID compliance has been a nightmare resulting in a lot of bad data so I am looking into relational databases. I ...
0
votes
1answer
917 views

PostgreSQL: remaining connection slots are reserved for non-replication superuser connections, too many clients already

We have web application using Golang, PostgreSQL, and sqlx (adapter, connection pooler), that each request requires 1 to 8 queries, sometimes 1 transaction with 5-8 select and 5-8 insert queries. We ...
0
votes
1answer
55 views

Flexibility vs Performance question

I'm in the middle of designing a database, and have a question for the veterans in regards to scalability. Currently, I have table A, which represents some object, and table B, which contains ...
0
votes
0answers
33 views

Good Book for professional database design [duplicate]

I am currently working on a project with some friends and we are hoping to make a business out of it. We are at the database design phase and as a learning source I am using "An Introduction to ...
1
vote
2answers
304 views

Standalone MongoDB to MongoDB Shard Setup

I'm learning MongoDB currently so maybe the title of this question is worded poorly, but I'm looking to understand the scaling of Mongo better. I did some research on what DB would be best for my ...
5
votes
2answers
55 views

Allocation strategy where tables regularly grow and shrink by large amounts

I have a SQL Server 2014 database with 100+ tables that have a 5% growth rate over the course of a year. Lots of reading, very little writing. This database is going to expand by another set of 20+ ...
1
vote
1answer
133 views

Count childs of parent with sql or save in column?

In postgres I have a threaded setup where parent id has many childs and childs can have childs. What is the most efficient way to get the count with a sql statement similar to this: WITH RECURSIVE ...
1
vote
1answer
47 views

Postgres or MapReduce to Match Reciprocal Needle Pairs from a Stack of Needles?

I have a Postgres (latest version) table with about a million lines (a_lat, a_lon, b_lat, b_lon) that each have about a dozen criteria. These lines go together in reciprocal pairs (A - B, B - A), ...
2
votes
3answers
1k views

Is it OK to create hundreds of databases in SQL Azure versus one big one and run the risk of deadlocking

I need to create highly scalable solution - where field devices in thousands of sites are delivering data in real time to a back end system, and SQL Azure seems to fit the bill nicely in terms of ...
0
votes
0answers
34 views

MySQL 5.6 or 5.7 sharding geospatial data

I have ran into a problem where I need to start to scale horizontally. With my database I am performing only nearest point searches with my data. So from a lat/long inoput I find all the nearest ...
8
votes
3answers
10k views

Is it better to separate a big query into multiple smaller queries?

There are situations which require having really big query joining several tables together with sub select statements in them to produce the desired results. My question is, should we consider ...
3
votes
2answers
128 views

Why is vertical scaling expensive?

As you can find here Best Practices For Horizontal Application Scaling, vertical scaling is much more expensive than horizontal scaling: vertical scaling cost increase exponentially whereas with ...
1
vote
1answer
94 views

Insert/ Update Scalability (more records, continually longer lookups — solution?)

posted this before but never received a clear answer. I'm working on an ETL project and am basically feeding JSON records of "tickets" into a MySQL database. (actually come to think of it, may be ...
1
vote
1answer
509 views

optimizing MySQL for traffic analytics system

background : I've developed a URL shortener system like Bitly with same features , so the system also tracks clickers info and represent as graphs to the person who has shorten the link as ...
0
votes
1answer
62 views

Scalability question

There is a DB {MySql} with 5 tables A, BIG, C, D, E. Their grow factor is about 1/100/1/1/1. The table BIG has Insert/Read/Update request ratio about 1/10/2. The Inserts and Updates 'cannot fail'. ...
-2
votes
1answer
86 views

How to scale a write intesive application with a SQL Server 2014? [closed]

I am studying about many databases lately and the one thing I can't get the hang of is how to scale a write intensive application in SQL Server, I'm talking about native features, not third party ...
4
votes
3answers
369 views

Scalable query for running counts of events within x previous days

I already posted this question on stackoverflow but I thought that I might get a better answer here. I have a table storing millions of events occurring to users: ...
0
votes
0answers
147 views

Is mysql cluster a viable choice for real time data availability when it comes to data in high volumes (petabytes)?

I've been reading a lot of articles promoting MySql Cluster as a big data storage solution for real time applications, one of which is this one . But as I went through the Mysql Cluster docs, I saw ...
5
votes
1answer
2k views

Scaling PostgreSQL TRIGGER(s)

How Postgres triggers mechanism scales ? We have a large PostgreSQL installation and we are trying to implement an event based system using log tables and TRIGGER(s). Basically we would like to ...
0
votes
1answer
143 views

GROUP BY on two large joined tables

I have two tables: Person ------ PersonID (PK) GroupID DeletedDate PersonData ---------- PersonID (PK) FieldID (PK) Value The fields are custom-created; hence the schema. I am trying to run a ...
2
votes
1answer
112 views

How do you generate large OLTP and reporting workloads? [closed]

My current skillset lies towards plain vanilla data warehousing, with < 20k users, our setups are rather simple.. I am looking for ways to enhance my technical skills, however, as much theory as I ...
0
votes
0answers
26 views

How to figure out how to split a table using the proper scheme

If you have a big table that you want to partition and you have 2 candidates as the partitioning key (in the sense that 50% of the queries use one column and 50% of the queries use the other column ...
1
vote
1answer
61 views

Denormalization Strategies: When is it appropriate to create redundancy for faster querying?

So I sometimes see databases purposely add redundant information for faster querying. I saw that this was mentioned here but I was wondering when would this be better than indexing, or creating a ...
0
votes
1answer
763 views

Architecture for low-latency RESTful server on Amazon EC2 with Node.js and a database

This is a question for DBAs with a lot of practical experience with large scale deployments. We are trying to build an app with very low latency. (We really want to create the best user experience, ...
0
votes
1answer
177 views

Optimization for 100 GB Real-time info table

I have a table of device information feed which is filled by data from remote device units. There will be like 1000 units each sending an update per minute which will be stored as a row in that ...
2
votes
1answer
3k views

SQL Server many-to-many relationship, row reuse?

Below is your typical many-to-many relationship. Pretend there are 2 tasks that have the exact same Tags. because of foreign key/primary key relationship constraints, there is no way for the rows in ...
1
vote
1answer
38 views

Are there any inherently non-partitionable datasets?

Are there any practical datasets that, by their nature, cannot be partitioned in any practical way? I am wondering about this because such datasets would be ill-suited to scaling out on commodity ...
0
votes
0answers
144 views

Would an INNER JOIN query in a sharded relational database hit the same shard when using a compound primary key?

In this question I have asked about modulus calculation in sharded tables, this design is used: CREATE TABLE Orders (order_id INT PRIMARY KEY, ...); CREATE TABLE LineItems (order_id INT, lineitem_id ...
2
votes
0answers
78 views

Modulus calculation for compound primary keys in MySQL Cluster (and other sharded databases)

In MySQL Cluster, and generally in sharded databases, how is a compound primary key's modulus calculated? Say, for example, there are two tables that have primary keys like so: CREATE TABLE Orders ...
2
votes
1answer
60 views

Reference implementation of a scalable storage system with strong consistency?

Is there a storage system with these properties? horizontally scalable - can handle arbitrary read and write throughput if you add enough nodes to the cluster consistent, in the sense that all reads ...
2
votes
4answers
287 views

What are the risks of building a database outside IT control?

I've been looking for the best StackExchange community to ask this. I hope it fits well here. As a risk analyst in a trading business, I'm often confronted with large volumes of data, way beyond the ...
60
votes
5answers
12k views

Why can't RDBM's cluster the way NoSQL does?

One of the big plusses for nosql DBMS is that they can cluster more easily. Supposedly with NoSQL you can create hundreds of cheap machines that store different pieces of data and query it all at ...
0
votes
1answer
159 views

Scaling out mongodb on the same storage

We are planning to have some tests to predict how mongo db operates and functions with 2 terabyte of inforamtion on a single collection. So the plan is to have 15-20 sharded machines (on the same ...
4
votes
1answer
3k views

Confused about load balancing and horizontal scaling in postgresql [closed]

Please correct me if I am wrong but I guess handling more requests and load by adding more machines or balancing the load between multiple servers is horizontal scalling. So, if I add more servers, ...
4
votes
3answers
1k views

Are there any in-memory NoSQL databases?

Given that NoSQL databases are designed to support horizontal scalability by throwing in more servers, are there any that are designed to do entirely without persistent disk storage? I'm looking at ...
1
vote
0answers
286 views

PostgreSQL how to create a scalable location-based query

I need suggestions because I am not good enough. I have a DB in PostgreSQL that runs on AWS (Amazon Web Services). I have a table "user_location" where there are stored the locations of each user ...
0
votes
0answers
85 views

MySQL: Optimizing for large but discrete data sets

In brief: I'm developing a database that handles GTFS datasets from multiple transit agencies. Each dataset contains millions of rows in the stop_times.txt file (and thus its corresponding table). ...
1
vote
2answers
161 views

How to store massive quantities of machine generated data?

I am tasked with developing a system that stores massive quantities of machine generated data (syslogs) from a number of server farms and I am wondering what tools you fine folks use such instances. ...
3
votes
2answers
267 views

Scalability: Desiging a message table

This question might be deadly simple but I wanted to make sure I am going through the correct path and not making any simple and irrevertible mistakes. My project will be based on messaging and it ...
1
vote
1answer
815 views

Mitigating MySQL replication delay in single master/multiple slave setup

For a particular web application that requires high availability I am thinking of scaling out by setting-up MySQL replication with one "write" master and multiple "read" slaves (as explained here ...
0
votes
0answers
164 views

MySql Scalability, Master Slave replication a good idea? Read description please

I have a database on mysql which is scaling on daily basis and right now its on 5M records and in a month it may double itself. I have a table containing users data which is the largest one. I have ...
4
votes
4answers
1k views

In Memory Database for high transaction and volume spikes on a website [closed]

I am looking at the IA for a hypothetical application scenario, for example: A high demand application such as an online ticketing website where there can be demand for 100,000 tickets to be ...