Questions tagged [scalability]
The scalability tag has no usage guidance.
152
questions
1
vote
1
answer
67
views
Scalability assessment for a simple auditing system based on an ORM
I was asked to create a simple audit system for a Web application. Due to some constraints I have chosen a "dirt and quick" way. This was possible due to small activity in that particular application (...
- 1,261
0
votes
1
answer
89
views
Need Expert suggestion for choosing best DB scalability
I am looking best opinion on choosing best DB scalability based on below requirement.
Application
App install tracking platform (Ad serving).
It's used to track click, impression, app install.
...
- 160
5
votes
4
answers
12k
views
Is it better to have large tables or many tables (MySQL)
In terms of performance, is it advisable to have only a few tables with a lot of rows or a lot of tables with only a few rows? In my instance, I have groups that I am keeping data separately for. Is ...
Zach Panzarino
0
votes
1
answer
47
views
Why does a query on 4 times more data take 300 times more processing?
To score the price preference of our individual customers in, we run the below query.
Basically it is a weighted average of a product score. The score depends on the product group and the packaging ...
- 85
1
vote
1
answer
148
views
Storing email inbox information
I am working on a webmail client. I want to store the remote mailboxes information in a Postgres database. As a general rule of thumb, most IMAP servers will have a similar default folder structure ...
- 113
1
vote
0
answers
36
views
How to design for scalable set operations?
This is the requirement for an educational application to teach words to language learners:
Learner can declare that he/she knows a given word
Learner can memorize the given word (to be managed by ...
- 1,437
1
vote
2
answers
211
views
scaling options for mysql on linux vm (ubuntu) : how to scale when the RAID 0 is out of space
I was asked to give a scaling road map for our mysql database.
Currently, I have a single mysql instance v5.6.28 running on an ubuntu VM on azure. It has a (software) RAID 0 disk made with two 1TBs.
...
- 11
2
votes
0
answers
1k
views
MongoDB daily document growth rate
Is MongoDB feasibly suited to handle 90,000,000 documents per day? Version 3.2.
I did some rough estimates on my data size. There will be about 90,000,000 recorded events per day, each a maximum size ...
- 123
3
votes
1
answer
360
views
Are Availability Groups feasible for 4 TB of data?
Currently we have a database which is 4 TB in size. We would like to put it in a 2014 Availability Group to scale out reads. Data gets changed a lot inside this database.
I have following questions:
...
- 309
0
votes
1
answer
83
views
MySQL - design help for scale
I will be using Amazon Web Services RDS service to host a MySQL 5.7 server (Currently AWS do not support 5.7, but they have announced they are working on implementing this version and it will be out ...
- 305
2
votes
1
answer
411
views
Design a many-to-many table for scale [closed]
I am thinking about scaling issues with many-to-many join tables.
I have a constructed example as following:
Many users have many wishlists.
On every wishlist are many items.
If I scale this table ...
- 29
9
votes
2
answers
4k
views
Postgres with pgpool architecture
Below is an example pgpool architecture:
This implies that you only need to have pgpool on a single server; is this true? When I look at the configuration I also see that you configure backends ...
user20459
2
votes
0
answers
111
views
PostgreSQL on Cluster
question - I'm working on a project that involves heavy computation and will continually grow in terms of the database it works with. I want to build a small cluster (NUCs I'm thinking) the run a ...
- 21
2
votes
3
answers
2k
views
Scaling for MariaDB / Percona / Galera Scaling
I ventured into the world of NoSQL database and as great as they are the lack of true ACID compliance has been a nightmare resulting in a lot of bad data so I am looking into relational databases.
I ...
- 305
4
votes
1
answer
21k
views
PostgreSQL: remaining connection slots are reserved for non-replication superuser connections, too many clients already
We have web application using Golang, PostgreSQL, and sqlx (adapter, connection pooler), that each request requires 1 to 8 queries, sometimes 1 transaction with 5-8 select and 5-8 insert queries.
We ...
- 1,363
0
votes
1
answer
128
views
Flexibility vs Performance question
I'm in the middle of designing a database, and have a question for the veterans in regards to scalability.
Currently, I have table A, which represents some object, and table B, which contains key-...
- 25
4
votes
1
answer
3k
views
Mongo Scaling - Too Many Connections - Linear Horizontal Scaling with 100s of App Servers
I have a Mongo Cluster. It's sharded on 6 servers, with 6 replica sets, each replica set having one Primary, and two Secondaries. Additionally, there are 3 config servers. Though, I think the Mongo ...
- 303
5
votes
2
answers
81
views
Allocation strategy where tables regularly grow and shrink by large amounts
I have a SQL Server 2014 database with 100+ tables that have a 5% growth rate over the course of a year. Lots of reading, very little writing.
This database is going to expand by another set of 20+ ...
- 153
11
votes
1
answer
14k
views
How do you calculate how much hardware resources you need for a database?
We are in the process of scaling our database servers. I was wondering how should we calculate how much hardware resources we need for our databases?
Here is a little bit information about our ...
- 175
2
votes
2
answers
410
views
MySQL growth 500k records/day
Im not a DBA however am an infrastructure guy used to installing and configuring database servers such as MSSQL, MySQL, MongoDB etc...
I have a MySQL database which is being used for archival of ...
- 163
3
votes
1
answer
672
views
Continuously arriving data with fast filtering on date in PostgreSQL
I have a huge amount of data, which arrives continously. (About 10 million rows/day, ~3000 rows/10 seconds in peak intervals of the day.)
I want to store them in a PostgreSQL database (I use PostGis ...
- 133
1
vote
1
answer
2k
views
Count childs of parent with sql or save in column?
In postgres I have a threaded setup where parent id has many childs and childs can have childs. What is the most efficient way to get the count with a sql statement similar to this:
WITH RECURSIVE ...
- 167
2
votes
1
answer
739
views
Postgres or MapReduce to Match Reciprocal Needle Pairs from a Stack of Needles?
I have a Postgres (latest version) table with about a million lines (a_lat, a_lon, b_lat, b_lon) that each have about a dozen criteria.
These lines go together in reciprocal pairs (A - B, B - A), ...
- 297
7
votes
3
answers
3k
views
Why is vertical scaling expensive?
As you can find here Best Practices For Horizontal Application Scaling, vertical scaling is much more expensive than horizontal scaling:
vertical scaling cost increase exponentially whereas with ...
- 255
1
vote
1
answer
185
views
Insert/ Update Scalability (more records, continually longer lookups --- solution?)
posted this before but never received a clear answer.
I'm working on an ETL project and am basically feeding JSON records of "tickets" into a MySQL database. (actually come to think of it, may be ...
- 1,683
0
votes
1
answer
106
views
Scalability question
There is a DB {MySql} with 5 tables A, BIG, C, D, E. Their grow factor is about 1/100/1/1/1.
The table BIG has Insert/Read/Update request ratio about 1/10/2. The Inserts and Updates 'cannot fail'.
...
2
votes
0
answers
49
views
What is a scaleable database design for querying items and only returning those a particular user has not already seen?
I'm looking at database architectures that will allow the storage of a list of items a user has already seen so that I could subsequently query the items table and only show the user items they had ...
- 21
-2
votes
1
answer
186
views
How to scale a write intesive application with a SQL Server 2014? [closed]
I am studying about many databases lately and the one thing I can't get the hang of is how to scale a write intensive application in SQL Server, I'm talking about native features, not third party ...
- 205
5
votes
3
answers
3k
views
Scalable query for running counts of events within x previous days
I already posted this question on stackoverflow but I thought that I might get a better answer here.
I have a table storing millions of events occurring to users:
...
- 183
1
vote
1
answer
866
views
GROUP BY on two large joined tables
I have two tables:
Person
------
PersonID (PK)
GroupID
DeletedDate
PersonData
----------
PersonID (PK)
FieldID (PK)
Value
The fields are custom-created; hence the schema. I am trying to run a ...
- 111
20
votes
2
answers
13k
views
Scaling PostgreSQL TRIGGER(s)
How Postgres triggers mechanism scales ?
We have a large PostgreSQL installation and we are trying to implement an event based system using log tables and TRIGGER(s).
Basically we would like to ...
- 301
1
vote
0
answers
176
views
schema for mysql database with 30million rows
We am building a SaaS backend for restaurants. We integrate directly with POS, so each POS keeps sending customer orders that we store in a following table in mysql.
CREATE TABLE name (
`id` int(11)...
- 111
2
votes
2
answers
2k
views
Standalone MongoDB to MongoDB Shard Setup
I'm learning MongoDB currently so maybe the title of this question is worded poorly, but I'm looking to understand the scaling of Mongo better. I did some research on what DB would be best for my ...
- 23
1
vote
3
answers
325
views
500 databases or 1 database with 500 tables or just 1 table with all the records? [closed]
I currently have an application that is used by a single end customer. For ease of discussion assume the application only needs a single database table for all records. I now need to support multi-...
- 111
2
votes
1
answer
227
views
How do you generate large OLTP and reporting workloads? [closed]
My current skillset lies towards plain vanilla data warehousing, with < 20k users, our setups are rather simple..
I am looking for ways to enhance my technical skills, however, as much theory as I ...
- 563
1
vote
1
answer
750
views
Denormalization Strategies: When is it appropriate to create redundancy for faster querying?
So I sometimes see databases purposely add redundant information for faster querying. I saw that this was mentioned here but I was wondering when would this be better than indexing, or creating a view?...
- 111
0
votes
1
answer
2k
views
Architecture for low-latency RESTful server on Amazon EC2 with Node.js and a database
This is a question for DBAs with a lot of practical experience with large scale deployments.
We are trying to build an app with very low latency. (We really want to create the best user experience, ...
- 113
1
vote
1
answer
1k
views
Optimization for 100 GB Real-time info table
I have a table of device information feed which is filled by data from remote device units.
There will be like 1000 units each sending an update per minute which will be stored as a row in that table....
- 11
2
votes
1
answer
7k
views
SQL Server many-to-many relationship, row reuse?
Below is your typical many-to-many relationship. Pretend there are 2 tasks that have the exact same Tags. because of foreign key/primary key relationship constraints, there is no way for the rows in ...
- 188
1
vote
0
answers
49
views
Are there any inherently non-partitionable datasets?
Are there any practical datasets that, by their nature, cannot be partitioned in any practical way?
I am wondering about this because such datasets would be ill-suited to scaling out on commodity ...
- 141
0
votes
0
answers
453
views
Would an INNER JOIN query in a sharded relational database hit the same shard when using a compound primary key?
In this question I have asked about modulus calculation in sharded tables, this design is used:
CREATE TABLE Orders (order_id INT PRIMARY KEY, ...);
CREATE TABLE LineItems (order_id INT, lineitem_id ...
- 1,568
2
votes
0
answers
127
views
Modulus calculation for compound primary keys in MySQL Cluster (and other sharded databases)
In MySQL Cluster, and generally in sharded databases, how is a compound primary key's modulus calculated?
Say, for example, there are two tables that have primary keys like so:
CREATE TABLE Orders (...
- 1,568
2
votes
1
answer
92
views
Reference implementation of a scalable storage system with strong consistency?
Is there a storage system with these properties?
horizontally scalable - can handle arbitrary read and write throughput if you add enough nodes to the cluster
consistent, in the sense that all reads ...
- 21
2
votes
4
answers
962
views
What are the risks of building a database outside IT control?
I've been looking for the best StackExchange community to ask this. I hope it fits well here.
As a risk analyst in a trading business, I'm often confronted with large volumes of data, way beyond the ...
- 55
3
votes
1
answer
3k
views
Are TCP loadbalancers an alternative to Postgres 9.X read slaves load balancers like pgPool II or pgBouncer?
To scale read traffic on PostgreSQL streaming replication slaves, I want to be able to load-balance the requests. Postgres documentation suggests using tools like pgPool and pgBouncer, but I am ...
- 275
5
votes
1
answer
5k
views
Sharding Postgres Database
I have a Postgres database that has grown to the size where it is no longer feasible to store everything on a single database node. There is a Customer table in my schema where each row represents a (...
- 773
7
votes
1
answer
5k
views
Confused about load balancing and horizontal scaling in postgresql [closed]
Please correct me if I am wrong but I guess handling more requests and load by adding more machines or balancing the load between multiple servers is horizontal scalling. So, if I add more servers, ...
0
votes
1
answer
526
views
Scaling out mongodb on the same storage
We are planning to have some tests to predict how mongo db operates and functions with 2 terabyte of inforamtion on a single collection.
So the plan is to have 15-20 sharded machines (on the same ...
- 85
22
votes
3
answers
36k
views
PostgreSQL High Availability/Scalability using HAProxy and PGBouncer
I have multiple PostgreSQL servers for a web application.
Typically one master and multiple slaves in hot standby mode (asynchronous streaming replication).
I use PGBouncer for connection pooling: ...
- 2,368
1
vote
0
answers
422
views
PostgreSQL how to create a scalable location-based query
I need suggestions because I am not good enough.
I have a DB in PostgreSQL that runs on AWS (Amazon Web Services).
I have a table "user_location" where there are stored the locations of each user ...
- 21