Questions tagged [scalability]
The scalability tag has no usage guidance.
129
questions
1
vote
0answers
46 views
How to choose database for scalable chat systems? [closed]
I have made a chat application where for now I am storing complete history of chat of all the users.
I am using django as backend and postgres as the database. I am nearing 100k daily active users ...
0
votes
1answer
20 views
mongodb read write locks
I'm currently investigating the possibility of using mongodb. This would contain the a collection of 100million+ documents. I need a process that will be constantly running deleting documents older ...
-3
votes
2answers
154 views
Replication and sharding combinations
In modern relational databases, what combinations of replication and sharding are possible?
Actually, I'm interested in the following strategy. Is it possible to replicate a small table in order to ...
0
votes
0answers
20 views
why is object storage more scalable than file or block storage
I'm starting to learn about object storage, and everywhere I read1 about its advantages, I see the following statement: "Scalability is increased as the physical location of data becomes irrelevant", ...
0
votes
1answer
54 views
is vertical scaling good idea for a mysql database server?
I operate a mariadb/mysql database server and i do not want to get into the problems with managing a multi-cluster database server
i have access to huge CPU and RAM and STORAGE when needed
so my ...
1
vote
0answers
56 views
What is the right way to increment like/comment counters on CosmosDB?
I'm currently working on a new app which will use a NOSQL DB (particularly Cosmos DB) for data scalability. I need to implement a way to update counters every time a post receives a like or comment. ...
1
vote
0answers
235 views
Which is better schema for given discussion forum, MongoDB
Lets have two variants of nosql collection for storing discussion comments.
Assume discussion, where is only one thread/root comment with 100 replies.
A./
collection: discussion_thread (id, ...
1
vote
1answer
222 views
How to deal with unbounded growth in cassandra shards
Let's say we have a website like youtube where we have a comment session per e video. Users want to be able to comment and to see the comments made by other users by comment creation time. This is ...
0
votes
2answers
40 views
Cassandra Cluster storage
I want to try out Cassandra Cluster. My main question is about scalability.
Every node in a Cassandra cluster has the same copy of data. So when I have in total 1 TB and have 5 nodes -> 5TB.
This at ...
0
votes
0answers
107 views
Alternative to Mutli-line Table Valued Functions
I currently have a relatively large MTVF returning a set of data, the code I have provided below is a small subsection of what I have in place. All but 1 part of my function is built from a Cte chain ...
3
votes
1answer
444 views
Database slows down at millions of rows, how to optimize schema and queries?
I am building a database to store Ethereum Transactions that are connected to specific addresses. Once the database table eth_addresses_tx starts hitting 3 million + rows, things start slowing down a ...
0
votes
1answer
43 views
Database for timeseries data [closed]
I work in a fintech company and we are redesigning our data system.
We have around 120 GB of compressed data stored in the filesystem.
The data concerns around 80K different timeseries each one of ...
1
vote
2answers
50 views
Aggregating high volumes of data from many MySQL servers
I have 4 servers of MySQL, each receives around 4 million records a day.
I need to aggregate all the inserted records into another main server after doing some mapping processing to the aggregated ...
1
vote
1answer
80 views
Organizing a comments table: How to justify the choice?
Business domain description
Some of the entity types of interest with respect to the relevant business domain are money, user and progress.
The users may need to post comments about money and ...
0
votes
2answers
119 views
MySQL innodb scaling [closed]
I am using Jamf to manage 25K Macs and 13K iOS devices. Every time I try to update iOS apps or configuration profiles for a large number of devices my database gets overwhelmed. I have MySQL 5.6 ...
43
votes
6answers
7k views
Is there any limit to the number of databases you can put on one SQL server?
I'm setting up a SaaS system, where we're planning to give each customer their own database. The system is already set up so that we can easily scale out to additional servers if the load becomes too ...
8
votes
1answer
957 views
Redesign the storage of large amounts of sensor data
I have been tasked to implement/redesign a solution that will store weather data from a sensor array. The array will consist of ~40 towers, each with about ~10 sensors each that'll sample atmospheric ...
0
votes
1answer
37 views
Searching for scalable storage solution
In one project I save statuses of processing in MySQL table:
`id` `name` `email` `has_apples` `has_bananas`
1 Bob a@b.c 1 0
2 Mike c@d.e 0 1
3 ...
6
votes
1answer
1k views
Scaling Azure SQL database will make my website goes down?
I have a website using Azure SQL databases, and I have some questions:
How does the scaling process work? Is there a flow, diagram that illustrates it?
Will the scaling cause any downtime in my ...
3
votes
2answers
758 views
How really big applications (such as facebook) scale their database [closed]
I am curious of how one will scale huge database for example facebook DB. What I know is the application is deployed somehow to multiple data centers which means that each "node" in each data center ...
2
votes
1answer
46 views
Maintainability of schema changes during runtime of an app [closed]
We have an application which accepts JSON objects from X number of third party services and unfortunately we're locked into using SQL to store it. The objects must be flat (no nesting), but each third ...
0
votes
1answer
41 views
Strategy to scale a comparison table query and load time
given a product table
PRODUCT
id some_column
1 10
2 2
3 8
when comparing all them I get a triangular matrix that I load from a csv file
COMPARISION
id product_left product_right ...
1
vote
1answer
33 views
Is it possible to delegate responsibility for specific indexes for a collection to specific shards in a mongoDB database?
Is it possible to delegate responsibility for specific indexes for a collection to specific shards in a mongoDB database?
For instance, say I have a collection called 'books':
{ title: <string>...
12
votes
3answers
6k views
Should a multi tenant system with SQL Server 2016, Shard or have Tenant isolation via separate database per tenant?
Given the use case:
Tenant data should not cross talk, one tenant does not need another
tenant's data.
Each tenant could potentially have large historical data volume.
SQL Server is hosted in AWS ...
1
vote
1answer
56 views
Scalability assessment for a simple auditing system based on an ORM
I was asked to create a simple audit system for a Web application. Due to some constraints I have chosen a "dirt and quick" way. This was possible due to small activity in that particular application (...
0
votes
1answer
74 views
Need Expert suggestion for choosing best DB scalability
I am looking best opinion on choosing best DB scalability based on below requirement.
Application
App install tracking platform (Ad serving).
It's used to track click, impression, app install.
...
2
votes
4answers
413 views
Is it better to have large tables or many tables (MySQL)
In terms of performance, is it advisable to have only a few tables with a lot of rows or a lot of tables with only a few rows? In my instance, I have groups that I am keeping data separately for. Is ...
0
votes
1answer
38 views
Why does a query on 4 times more data take 300 times more processing?
To score the price preference of our individual customers in, we run the below query.
Basically it is a weighted average of a product score. The score depends on the product group and the packaging ...
1
vote
1answer
58 views
Storing email inbox information
I am working on a webmail client. I want to store the remote mailboxes information in a Postgres database. As a general rule of thumb, most IMAP servers will have a similar default folder structure ...
1
vote
0answers
32 views
How to design for scalable set operations?
This is the requirement for an educational application to teach words to language learners:
Learner can declare that he/she knows a given word
Learner can memorize the given word (to be managed by ...
1
vote
2answers
133 views
scaling options for mysql on linux vm (ubuntu) : how to scale when the RAID 0 is out of space
I was asked to give a scaling road map for our mysql database.
Currently, I have a single mysql instance v5.6.28 running on an ubuntu VM on azure. It has a (software) RAID 0 disk made with two 1TBs.
...
2
votes
0answers
478 views
MongoDB daily document growth rate
Is MongoDB feasibly suited to handle 90,000,000 documents per day? Version 3.2.
I did some rough estimates on my data size. There will be about 90,000,000 recorded events per day, each a maximum size ...
3
votes
1answer
172 views
Are Availability Groups feasible for 4 TB of data?
Currently we have a database which is 4 TB in size. We would like to put it in a 2014 Availability Group to scale out reads. Data gets changed a lot inside this database.
I have following questions:
...
0
votes
1answer
62 views
MySQL - design help for scale
I will be using Amazon Web Services RDS service to host a MySQL 5.7 server (Currently AWS do not support 5.7, but they have announced they are working on implementing this version and it will be out ...
2
votes
1answer
258 views
Design a many-to-many table for scale [closed]
I am thinking about scaling issues with many-to-many join tables.
I have a constructed example as following:
Many users have many wishlists.
On every wishlist are many items.
If I scale this table ...
8
votes
2answers
2k views
Postgres with pgpool architecture
Below is an example pgpool architecture:
This implies that you only need to have pgpool on a single server; is this true? When I look at the configuration I also see that you configure backends ...
2
votes
0answers
102 views
PostgreSQL on Cluster
question - I'm working on a project that involves heavy computation and will continually grow in terms of the database it works with. I want to build a small cluster (NUCs I'm thinking) the run a ...
1
vote
3answers
957 views
Scaling for MariaDB / Percona / Galera Scaling
I ventured into the world of NoSQL database and as great as they are the lack of true ACID compliance has been a nightmare resulting in a lot of bad data so I am looking into relational databases.
I ...
2
votes
1answer
12k views
PostgreSQL: remaining connection slots are reserved for non-replication superuser connections, too many clients already
We have web application using Golang, PostgreSQL, and sqlx (adapter, connection pooler), that each request requires 1 to 8 queries, sometimes 1 transaction with 5-8 select and 5-8 insert queries.
We ...
0
votes
1answer
85 views
Flexibility vs Performance question
I'm in the middle of designing a database, and have a question for the veterans in regards to scalability.
Currently, I have table A, which represents some object, and table B, which contains key-...
3
votes
1answer
2k views
Mongo Scaling - Too Many Connections - Linear Horizontal Scaling with 100s of App Servers
I have a Mongo Cluster. It's sharded on 6 servers, with 6 replica sets, each replica set having one Primary, and two Secondaries. Additionally, there are 3 config servers. Though, I think the Mongo ...
5
votes
2answers
59 views
Allocation strategy where tables regularly grow and shrink by large amounts
I have a SQL Server 2014 database with 100+ tables that have a 5% growth rate over the course of a year. Lots of reading, very little writing.
This database is going to expand by another set of 20+ ...
6
votes
1answer
7k views
How do you calculate how much hardware resources you need for a database?
We are in the process of scaling our database servers. I was wondering how should we calculate how much hardware resources we need for our databases?
Here is a little bit information about our ...
2
votes
2answers
254 views
MySQL growth 500k records/day
Im not a DBA however am an infrastructure guy used to installing and configuring database servers such as MSSQL, MySQL, MongoDB etc...
I have a MySQL database which is being used for archival of ...
3
votes
1answer
322 views
Continuously arriving data with fast filtering on date in PostgreSQL
I have a huge amount of data, which arrives continously. (About 10 million rows/day, ~3000 rows/10 seconds in peak intervals of the day.)
I want to store them in a PostgreSQL database (I use PostGis ...
1
vote
1answer
1k views
Count childs of parent with sql or save in column?
In postgres I have a threaded setup where parent id has many childs and childs can have childs. What is the most efficient way to get the count with a sql statement similar to this:
WITH RECURSIVE ...
2
votes
1answer
359 views
Postgres or MapReduce to Match Reciprocal Needle Pairs from a Stack of Needles?
I have a Postgres (latest version) table with about a million lines (a_lat, a_lon, b_lat, b_lon) that each have about a dozen criteria.
These lines go together in reciprocal pairs (A - B, B - A), ...
6
votes
2answers
1k views
Why is vertical scaling expensive?
As you can find here Best Practices For Horizontal Application Scaling, vertical scaling is much more expensive than horizontal scaling:
vertical scaling cost increase exponentially whereas with ...
1
vote
1answer
156 views
Insert/ Update Scalability (more records, continually longer lookups — solution?)
posted this before but never received a clear answer.
I'm working on an ETL project and am basically feeding JSON records of "tickets" into a MySQL database. (actually come to think of it, may be ...
0
votes
1answer
88 views
Scalability question
There is a DB {MySql} with 5 tables A, BIG, C, D, E. Their grow factor is about 1/100/1/1/1.
The table BIG has Insert/Read/Update request ratio about 1/10/2. The Inserts and Updates 'cannot fail'.
...
