Questions tagged [scalability]
The scalability tag has no usage guidance.
131
questions
3
votes
1answer
68 views
Concatenate a string value from table A to a text field in table B, in a scalable way
On a daily basis, I am adding values from the following table (table1):
Id Name Value
1 'Name 1' 789
2 'Name 2' 789
3 'Name 3' 789
To the text field (Values) in the following ...
-1
votes
1answer
58 views
Postgres 12 Scalability using table Partitioning and Foreign Data Wrappers
I have looked through the archives and I cant find any discussions around the following topic.
I have a fairly in depth question that I would appreciate some guidance with.
Current environment
...
0
votes
0answers
10 views
Cassandara ec2 Number of Nodes and autoscale for increasing 70Gb of Data
I am beginner in Cassandra, I currently have 72 GBs of Data that is growing and i will have approximately 1gbs inserted into database daily. How can i decide number of nodes in the cluster and how ...
0
votes
1answer
38 views
Does a read replica help if the problem are heavy queries?
The application that our team is responsible for is having ever more often DB performance problems. The application querying it doesn't throw lots of concurrent queries, the problem we have are rather ...
2
votes
1answer
287 views
mongodb read write locks [closed]
I'm currently investigating the possibility of using mongodb. This would contain the a collection of 100million+ documents. I need a process that will be constantly running deleting documents older ...
-3
votes
2answers
188 views
Replication and sharding combinations
In modern relational databases, what combinations of replication and sharding are possible?
Actually, I'm interested in the following strategy. Is it possible to replicate a small table in order to ...
0
votes
1answer
109 views
is vertical scaling good idea for a mysql database server?
I operate a mariadb/mysql database server and i do not want to get into the problems with managing a multi-cluster database server
i have access to huge CPU and RAM and STORAGE when needed
so my ...
1
vote
0answers
133 views
What is the right way to increment like/comment counters on CosmosDB?
I'm currently working on a new app which will use a NOSQL DB (particularly Cosmos DB) for data scalability. I need to implement a way to update counters every time a post receives a like or comment. ...
1
vote
0answers
318 views
Which is better schema for given discussion forum, MongoDB
Lets have two variants of nosql collection for storing discussion comments.
Assume discussion, where is only one thread/root comment with 100 replies.
A./
collection: discussion_thread (id, ...
1
vote
1answer
404 views
How to deal with unbounded growth in cassandra shards
Let's say we have a website like youtube where we have a comment session per e video. Users want to be able to comment and to see the comments made by other users by comment creation time. This is ...
0
votes
2answers
51 views
Cassandra Cluster storage
I want to try out Cassandra Cluster. My main question is about scalability.
Every node in a Cassandra cluster has the same copy of data. So when I have in total 1 TB and have 5 nodes -> 5TB.
This at ...
0
votes
0answers
142 views
Alternative to Mutli-line Table Valued Functions
I currently have a relatively large MTVF returning a set of data, the code I have provided below is a small subsection of what I have in place. All but 1 part of my function is built from a Cte chain ...
3
votes
1answer
986 views
Database slows down at millions of rows, how to optimize schema and queries?
I am building a database to store Ethereum Transactions that are connected to specific addresses. Once the database table eth_addresses_tx starts hitting 3 million + rows, things start slowing down a ...
0
votes
1answer
52 views
Database for timeseries data [closed]
I work in a fintech company and we are redesigning our data system.
We have around 120 GB of compressed data stored in the filesystem.
The data concerns around 80K different timeseries each one of ...
1
vote
2answers
62 views
Aggregating high volumes of data from many MySQL servers
I have 4 servers of MySQL, each receives around 4 million records a day.
I need to aggregate all the inserted records into another main server after doing some mapping processing to the aggregated ...
1
vote
1answer
234 views
Organizing a comments table: How to justify the choice?
Business domain description
Some of the entity types of interest with respect to the relevant business domain are money, user and progress.
The users may need to post comments about money and ...
0
votes
2answers
143 views
MySQL innodb scaling [closed]
I am using Jamf to manage 25K Macs and 13K iOS devices. Every time I try to update iOS apps or configuration profiles for a large number of devices my database gets overwhelmed. I have MySQL 5.6 ...
43
votes
6answers
8k views
Is there any limit to the number of databases you can put on one SQL server?
I'm setting up a SaaS system, where we're planning to give each customer their own database. The system is already set up so that we can easily scale out to additional servers if the load becomes too ...
8
votes
1answer
1k views
Redesign the storage of large amounts of sensor data
I have been tasked to implement/redesign a solution that will store weather data from a sensor array. The array will consist of ~40 towers, each with about ~10 sensors each that'll sample atmospheric ...
0
votes
1answer
37 views
Searching for scalable storage solution
In one project I save statuses of processing in MySQL table:
`id` `name` `email` `has_apples` `has_bananas`
1 Bob a@b.c 1 0
2 Mike c@d.e 0 1
3 ...
6
votes
2answers
1k views
Scaling Azure SQL database will make my website goes down?
I have a website using Azure SQL databases, and I have some questions:
How does the scaling process work? Is there a flow, diagram that illustrates it?
Will the scaling cause any downtime in my ...
4
votes
2answers
986 views
How really big applications (such as facebook) scale their database [closed]
I am curious of how one will scale huge database for example facebook DB. What I know is the application is deployed somehow to multiple data centers which means that each "node" in each data center ...
2
votes
1answer
53 views
Maintainability of schema changes during runtime of an app [closed]
We have an application which accepts JSON objects from X number of third party services and unfortunately we're locked into using SQL to store it. The objects must be flat (no nesting), but each third ...
0
votes
1answer
42 views
Strategy to scale a comparison table query and load time
given a product table
PRODUCT
id some_column
1 10
2 2
3 8
when comparing all them I get a triangular matrix that I load from a csv file
COMPARISION
id product_left product_right ...
1
vote
1answer
33 views
Is it possible to delegate responsibility for specific indexes for a collection to specific shards in a mongoDB database?
Is it possible to delegate responsibility for specific indexes for a collection to specific shards in a mongoDB database?
For instance, say I have a collection called 'books':
{ title: <string>...
12
votes
3answers
7k views
Should a multi tenant system with SQL Server 2016, Shard or have Tenant isolation via separate database per tenant?
Given the use case:
Tenant data should not cross talk, one tenant does not need another
tenant's data.
Each tenant could potentially have large historical data volume.
SQL Server is hosted in AWS ...
1
vote
1answer
56 views
Scalability assessment for a simple auditing system based on an ORM
I was asked to create a simple audit system for a Web application. Due to some constraints I have chosen a "dirt and quick" way. This was possible due to small activity in that particular application (...
0
votes
1answer
76 views
Need Expert suggestion for choosing best DB scalability
I am looking best opinion on choosing best DB scalability based on below requirement.
Application
App install tracking platform (Ad serving).
It's used to track click, impression, app install.
...
2
votes
4answers
1k views
Is it better to have large tables or many tables (MySQL)
In terms of performance, is it advisable to have only a few tables with a lot of rows or a lot of tables with only a few rows? In my instance, I have groups that I am keeping data separately for. Is ...
0
votes
1answer
39 views
Why does a query on 4 times more data take 300 times more processing?
To score the price preference of our individual customers in, we run the below query.
Basically it is a weighted average of a product score. The score depends on the product group and the packaging ...
1
vote
1answer
63 views
Storing email inbox information
I am working on a webmail client. I want to store the remote mailboxes information in a Postgres database. As a general rule of thumb, most IMAP servers will have a similar default folder structure ...
1
vote
0answers
32 views
How to design for scalable set operations?
This is the requirement for an educational application to teach words to language learners:
Learner can declare that he/she knows a given word
Learner can memorize the given word (to be managed by ...
1
vote
2answers
134 views
scaling options for mysql on linux vm (ubuntu) : how to scale when the RAID 0 is out of space
I was asked to give a scaling road map for our mysql database.
Currently, I have a single mysql instance v5.6.28 running on an ubuntu VM on azure. It has a (software) RAID 0 disk made with two 1TBs.
...
2
votes
0answers
553 views
MongoDB daily document growth rate
Is MongoDB feasibly suited to handle 90,000,000 documents per day? Version 3.2.
I did some rough estimates on my data size. There will be about 90,000,000 recorded events per day, each a maximum size ...
3
votes
1answer
204 views
Are Availability Groups feasible for 4 TB of data?
Currently we have a database which is 4 TB in size. We would like to put it in a 2014 Availability Group to scale out reads. Data gets changed a lot inside this database.
I have following questions:
...
0
votes
1answer
62 views
MySQL - design help for scale
I will be using Amazon Web Services RDS service to host a MySQL 5.7 server (Currently AWS do not support 5.7, but they have announced they are working on implementing this version and it will be out ...
2
votes
1answer
263 views
Design a many-to-many table for scale [closed]
I am thinking about scaling issues with many-to-many join tables.
I have a constructed example as following:
Many users have many wishlists.
On every wishlist are many items.
If I scale this table ...
9
votes
2answers
2k views
Postgres with pgpool architecture
Below is an example pgpool architecture:
This implies that you only need to have pgpool on a single server; is this true? When I look at the configuration I also see that you configure backends ...
2
votes
0answers
106 views
PostgreSQL on Cluster
question - I'm working on a project that involves heavy computation and will continually grow in terms of the database it works with. I want to build a small cluster (NUCs I'm thinking) the run a ...
1
vote
3answers
1k views
Scaling for MariaDB / Percona / Galera Scaling
I ventured into the world of NoSQL database and as great as they are the lack of true ACID compliance has been a nightmare resulting in a lot of bad data so I am looking into relational databases.
I ...
3
votes
1answer
15k views
PostgreSQL: remaining connection slots are reserved for non-replication superuser connections, too many clients already
We have web application using Golang, PostgreSQL, and sqlx (adapter, connection pooler), that each request requires 1 to 8 queries, sometimes 1 transaction with 5-8 select and 5-8 insert queries.
We ...
0
votes
1answer
87 views
Flexibility vs Performance question
I'm in the middle of designing a database, and have a question for the veterans in regards to scalability.
Currently, I have table A, which represents some object, and table B, which contains key-...
3
votes
1answer
2k views
Mongo Scaling - Too Many Connections - Linear Horizontal Scaling with 100s of App Servers
I have a Mongo Cluster. It's sharded on 6 servers, with 6 replica sets, each replica set having one Primary, and two Secondaries. Additionally, there are 3 config servers. Though, I think the Mongo ...
5
votes
2answers
61 views
Allocation strategy where tables regularly grow and shrink by large amounts
I have a SQL Server 2014 database with 100+ tables that have a 5% growth rate over the course of a year. Lots of reading, very little writing.
This database is going to expand by another set of 20+ ...
6
votes
1answer
9k views
How do you calculate how much hardware resources you need for a database?
We are in the process of scaling our database servers. I was wondering how should we calculate how much hardware resources we need for our databases?
Here is a little bit information about our ...
2
votes
2answers
276 views
MySQL growth 500k records/day
Im not a DBA however am an infrastructure guy used to installing and configuring database servers such as MSSQL, MySQL, MongoDB etc...
I have a MySQL database which is being used for archival of ...
3
votes
1answer
366 views
Continuously arriving data with fast filtering on date in PostgreSQL
I have a huge amount of data, which arrives continously. (About 10 million rows/day, ~3000 rows/10 seconds in peak intervals of the day.)
I want to store them in a PostgreSQL database (I use PostGis ...
1
vote
1answer
1k views
Count childs of parent with sql or save in column?
In postgres I have a threaded setup where parent id has many childs and childs can have childs. What is the most efficient way to get the count with a sql statement similar to this:
WITH RECURSIVE ...
2
votes
1answer
435 views
Postgres or MapReduce to Match Reciprocal Needle Pairs from a Stack of Needles?
I have a Postgres (latest version) table with about a million lines (a_lat, a_lon, b_lat, b_lon) that each have about a dozen criteria.
These lines go together in reciprocal pairs (A - B, B - A), ...
6
votes
2answers
1k views
Why is vertical scaling expensive?
As you can find here Best Practices For Horizontal Application Scaling, vertical scaling is much more expensive than horizontal scaling:
vertical scaling cost increase exponentially whereas with ...
