Questions tagged [scalability]
The scalability tag has no usage guidance.
137
questions
1
vote
2answers
37 views
How to scale with MySQL (when not ready to scale properly)
We are using MySql.
My situation
I have a large number of tables with millions of rows each. Those tables are updated every second and are used both for adding info and retrieving info. Each table ...
0
votes
0answers
12 views
MySQL - Updating 100m+ rows vs truncating and inserting
I've got a table in the following format:
id: primary key
foreign_id: foreign key
code_1: int w index
code_2: int with index
I have a file that comes in every week with 200m records in the format:
...
1
vote
1answer
15 views
Can I use only OrientDB without any RDBMS?
I'm struggling with this problem and I really need an advice. My project mainly needs a Social structure, users can be connected with some friendship relations, can create groups and so on. In this ...
0
votes
0answers
135 views
TiDB vs Vitess - How should I base my decision?
I need a scalable MySQL-compatable solution that can scale to multiple installs across a network, handling millions of records on several tables with tens of thousands of concurrent requests per-...
4
votes
1answer
50 views
Continuously Adding SSDs to an existing PostgreSQL Server
An Ubuntu 18.04 server dedicated to running a PostgreSQL 11.2 server (with TimescaleDB extension) will be running out of disk space soon so new SSD disks will need to be added to the machine to ...
2
votes
3answers
75 views
Is it realistic to expect a database of size over 50 GB (some 300+ million records) to be imported within 3-4 hours on a single server (MySQL)?
I have hundreds of millions of rows in a text/csv file (genomics database btw - each record is less than 255 characters long...).
Ideally I'd like to make them searchable since right now my best bet ...
3
votes
1answer
69 views
Concatenate a string value from table A to a text field in table B, in a scalable way
On a daily basis, I am adding values from the following table (table1):
Id Name Value
1 'Name 1' 789
2 'Name 2' 789
3 'Name 3' 789
To the text field (Values) in the following ...
-1
votes
1answer
150 views
Postgres 12 Scalability using table Partitioning and Foreign Data Wrappers
I have looked through the archives and I cant find any discussions around the following topic.
I have a fairly in depth question that I would appreciate some guidance with.
Current environment
...
0
votes
0answers
11 views
Cassandara ec2 Number of Nodes and autoscale for increasing 70Gb of Data
I am beginner in Cassandra, I currently have 72 GBs of Data that is growing and i will have approximately 1gbs inserted into database daily. How can i decide number of nodes in the cluster and how ...
0
votes
1answer
49 views
Does a read replica help if the problem are heavy queries?
The application that our team is responsible for is having ever more often DB performance problems. The application querying it doesn't throw lots of concurrent queries, the problem we have are rather ...
2
votes
1answer
450 views
mongodb read write locks [closed]
I'm currently investigating the possibility of using mongodb. This would contain the a collection of 100million+ documents. I need a process that will be constantly running deleting documents older ...
-3
votes
2answers
218 views
Replication and sharding combinations
In modern relational databases, what combinations of replication and sharding are possible?
Actually, I'm interested in the following strategy. Is it possible to replicate a small table in order to ...
0
votes
1answer
165 views
is vertical scaling good idea for a mysql database server?
I operate a mariadb/mysql database server and i do not want to get into the problems with managing a multi-cluster database server
i have access to huge CPU and RAM and STORAGE when needed
so my ...
1
vote
0answers
203 views
What is the right way to increment like/comment counters on CosmosDB?
I'm currently working on a new app which will use a NOSQL DB (particularly Cosmos DB) for data scalability. I need to implement a way to update counters every time a post receives a like or comment. ...
1
vote
0answers
386 views
Which is better schema for given discussion forum, MongoDB
Lets have two variants of nosql collection for storing discussion comments.
Assume discussion, where is only one thread/root comment with 100 replies.
A./
collection: discussion_thread (id, ...
1
vote
1answer
483 views
How to deal with unbounded growth in cassandra shards
Let's say we have a website like youtube where we have a comment session per e video. Users want to be able to comment and to see the comments made by other users by comment creation time. This is ...
0
votes
2answers
58 views
Cassandra Cluster storage
I want to try out Cassandra Cluster. My main question is about scalability.
Every node in a Cassandra cluster has the same copy of data. So when I have in total 1 TB and have 5 nodes -> 5TB.
This at ...
0
votes
0answers
177 views
Alternative to Mutli-line Table Valued Functions
I currently have a relatively large MTVF returning a set of data, the code I have provided below is a small subsection of what I have in place. All but 1 part of my function is built from a Cte chain ...
3
votes
1answer
1k views
Database slows down at millions of rows, how to optimize schema and queries?
I am building a database to store Ethereum Transactions that are connected to specific addresses. Once the database table eth_addresses_tx starts hitting 3 million + rows, things start slowing down a ...
0
votes
1answer
55 views
Database for timeseries data [closed]
I work in a fintech company and we are redesigning our data system.
We have around 120 GB of compressed data stored in the filesystem.
The data concerns around 80K different timeseries each one of ...
1
vote
2answers
75 views
Aggregating high volumes of data from many MySQL servers
I have 4 servers of MySQL, each receives around 4 million records a day.
I need to aggregate all the inserted records into another main server after doing some mapping processing to the aggregated ...
1
vote
1answer
331 views
Organizing a comments table: How to justify the choice?
Business domain description
Some of the entity types of interest with respect to the relevant business domain are money, user and progress.
The users may need to post comments about money and ...
0
votes
2answers
161 views
MySQL innodb scaling [closed]
I am using Jamf to manage 25K Macs and 13K iOS devices. Every time I try to update iOS apps or configuration profiles for a large number of devices my database gets overwhelmed. I have MySQL 5.6 ...
43
votes
6answers
9k views
Is there any limit to the number of databases you can put on one SQL server?
I'm setting up a SaaS system, where we're planning to give each customer their own database. The system is already set up so that we can easily scale out to additional servers if the load becomes too ...
8
votes
1answer
1k views
Redesign the storage of large amounts of sensor data
I have been tasked to implement/redesign a solution that will store weather data from a sensor array. The array will consist of ~40 towers, each with about ~10 sensors each that'll sample atmospheric ...
0
votes
1answer
38 views
Searching for scalable storage solution
In one project I save statuses of processing in MySQL table:
`id` `name` `email` `has_apples` `has_bananas`
1 Bob a@b.c 1 0
2 Mike c@d.e 0 1
3 ...
6
votes
2answers
2k views
Scaling Azure SQL database will make my website goes down?
I have a website using Azure SQL databases, and I have some questions:
How does the scaling process work? Is there a flow, diagram that illustrates it?
Will the scaling cause any downtime in my ...
4
votes
2answers
1k views
How really big applications (such as facebook) scale their database [closed]
I am curious of how one will scale huge database for example facebook DB. What I know is the application is deployed somehow to multiple data centers which means that each "node" in each data center ...
2
votes
1answer
57 views
Maintainability of schema changes during runtime of an app [closed]
We have an application which accepts JSON objects from X number of third party services and unfortunately we're locked into using SQL to store it. The objects must be flat (no nesting), but each third ...
0
votes
1answer
42 views
Strategy to scale a comparison table query and load time
given a product table
PRODUCT
id some_column
1 10
2 2
3 8
when comparing all them I get a triangular matrix that I load from a csv file
COMPARISION
id product_left product_right ...
1
vote
1answer
33 views
Is it possible to delegate responsibility for specific indexes for a collection to specific shards in a mongoDB database?
Is it possible to delegate responsibility for specific indexes for a collection to specific shards in a mongoDB database?
For instance, say I have a collection called 'books':
{ title: <string>...
13
votes
3answers
8k views
Should a multi tenant system with SQL Server 2016, Shard or have Tenant isolation via separate database per tenant?
Given the use case:
Tenant data should not cross talk, one tenant does not need another
tenant's data.
Each tenant could potentially have large historical data volume.
SQL Server is hosted in AWS ...
1
vote
1answer
56 views
Scalability assessment for a simple auditing system based on an ORM
I was asked to create a simple audit system for a Web application. Due to some constraints I have chosen a "dirt and quick" way. This was possible due to small activity in that particular application (...
0
votes
1answer
76 views
Need Expert suggestion for choosing best DB scalability
I am looking best opinion on choosing best DB scalability based on below requirement.
Application
App install tracking platform (Ad serving).
It's used to track click, impression, app install.
...
2
votes
4answers
2k views
Is it better to have large tables or many tables (MySQL)
In terms of performance, is it advisable to have only a few tables with a lot of rows or a lot of tables with only a few rows? In my instance, I have groups that I am keeping data separately for. Is ...
0
votes
1answer
41 views
Why does a query on 4 times more data take 300 times more processing?
To score the price preference of our individual customers in, we run the below query.
Basically it is a weighted average of a product score. The score depends on the product group and the packaging ...
1
vote
1answer
64 views
Storing email inbox information
I am working on a webmail client. I want to store the remote mailboxes information in a Postgres database. As a general rule of thumb, most IMAP servers will have a similar default folder structure ...
1
vote
0answers
32 views
How to design for scalable set operations?
This is the requirement for an educational application to teach words to language learners:
Learner can declare that he/she knows a given word
Learner can memorize the given word (to be managed by ...
1
vote
2answers
136 views
scaling options for mysql on linux vm (ubuntu) : how to scale when the RAID 0 is out of space
I was asked to give a scaling road map for our mysql database.
Currently, I have a single mysql instance v5.6.28 running on an ubuntu VM on azure. It has a (software) RAID 0 disk made with two 1TBs.
...
2
votes
0answers
614 views
MongoDB daily document growth rate
Is MongoDB feasibly suited to handle 90,000,000 documents per day? Version 3.2.
I did some rough estimates on my data size. There will be about 90,000,000 recorded events per day, each a maximum size ...
3
votes
1answer
228 views
Are Availability Groups feasible for 4 TB of data?
Currently we have a database which is 4 TB in size. We would like to put it in a 2014 Availability Group to scale out reads. Data gets changed a lot inside this database.
I have following questions:
...
0
votes
1answer
62 views
MySQL - design help for scale
I will be using Amazon Web Services RDS service to host a MySQL 5.7 server (Currently AWS do not support 5.7, but they have announced they are working on implementing this version and it will be out ...
2
votes
1answer
265 views
Design a many-to-many table for scale [closed]
I am thinking about scaling issues with many-to-many join tables.
I have a constructed example as following:
Many users have many wishlists.
On every wishlist are many items.
If I scale this table ...
9
votes
2answers
2k views
Postgres with pgpool architecture
Below is an example pgpool architecture:
This implies that you only need to have pgpool on a single server; is this true? When I look at the configuration I also see that you configure backends ...
2
votes
0answers
107 views
PostgreSQL on Cluster
question - I'm working on a project that involves heavy computation and will continually grow in terms of the database it works with. I want to build a small cluster (NUCs I'm thinking) the run a ...
1
vote
3answers
1k views
Scaling for MariaDB / Percona / Galera Scaling
I ventured into the world of NoSQL database and as great as they are the lack of true ACID compliance has been a nightmare resulting in a lot of bad data so I am looking into relational databases.
I ...
3
votes
1answer
16k views
PostgreSQL: remaining connection slots are reserved for non-replication superuser connections, too many clients already
We have web application using Golang, PostgreSQL, and sqlx (adapter, connection pooler), that each request requires 1 to 8 queries, sometimes 1 transaction with 5-8 select and 5-8 insert queries.
We ...
0
votes
1answer
89 views
Flexibility vs Performance question
I'm in the middle of designing a database, and have a question for the veterans in regards to scalability.
Currently, I have table A, which represents some object, and table B, which contains key-...
3
votes
1answer
2k views
Mongo Scaling - Too Many Connections - Linear Horizontal Scaling with 100s of App Servers
I have a Mongo Cluster. It's sharded on 6 servers, with 6 replica sets, each replica set having one Primary, and two Secondaries. Additionally, there are 3 config servers. Though, I think the Mongo ...
5
votes
2answers
61 views
Allocation strategy where tables regularly grow and shrink by large amounts
I have a SQL Server 2014 database with 100+ tables that have a 5% growth rate over the course of a year. Lots of reading, very little writing.
This database is going to expand by another set of 20+ ...
