Questions tagged [scalability]
The scalability tag has no usage guidance.
140
questions
1
vote
2
answers
618
views
Is SqlServer or NoSql more cost efficient when scaling horizontally
RDBMS are traditionally better for vertical scaling since it's harder to do horizontal scaling. As a result NoSql databases are cheaper since they can more easily scale horizontally.
However in recent ...
- 137
6
votes
1
answer
360
views
Is Postgres ideal for Many to Many data (M:N) which keeps increasing?
I have a use-case where the data is many to many and needs a wide range of querying functionality.
Participants and Events
A User/Participant can register for multiple events. And each event can have ...
- 173
0
votes
2
answers
50
views
In what range can you expect a cloud based Sql Server data reading in mb/s? [closed]
Say everything is optimized, queries are simple, indexes are good, etc.
Should a SQL (Azure) database where you spend about 150 to 300 euros per month easily handle data reading with about 250-500 mb ...
- 333
0
votes
0
answers
38
views
Scale out methods with high availability [duplicate]
We have a two site AlwaysOn Cluster and looking to do a High Available multi-site write solution. We have looked at Replication, Sharding, and SQL Elastic options for having multi-site write for a ...
- 1
1
vote
2
answers
81
views
How to scale with MySQL (when not ready to scale properly)
We are using MySql.
My situation
I have a large number of tables with millions of rows each. Those tables are updated every second and are used both for adding info and retrieving info. Each table ...
- 21
1
vote
0
answers
61
views
MySQL - Updating 100m+ rows vs truncating and inserting
I've got a table in the following format:
id: primary key
foreign_id: foreign key
code_1: int w index
code_2: int with index
I have a file that comes in every week with 200m records in the format:
...
- 11
1
vote
1
answer
26
views
Can I use only OrientDB without any RDBMS?
I'm struggling with this problem and I really need an advice. My project mainly needs a Social structure, users can be connected with some friendship relations, can create groups and so on. In this ...
- 111
0
votes
0
answers
1k
views
TiDB vs Vitess - How should I base my decision?
I need a scalable MySQL-compatable solution that can scale to multiple installs across a network, handling millions of records on several tables with tens of thousands of concurrent requests per-...
- 183
4
votes
1
answer
365
views
Continuously Adding SSDs to an existing PostgreSQL Server
An Ubuntu 18.04 server dedicated to running a PostgreSQL 11.2 server (with TimescaleDB extension) will be running out of disk space soon so new SSD disks will need to be added to the machine to ...
- 1,001
2
votes
3
answers
2k
views
Is it realistic to expect a database of size over 50 GB (some 300+ million records) to be imported within 3-4 hours on a single server (MySQL)?
I have hundreds of millions of rows in a text/csv file (genomics database btw - each record is less than 255 characters long...).
Ideally I'd like to make them searchable since right now my best bet ...
- 23
3
votes
1
answer
143
views
Concatenate a string value from table A to a text field in table B, in a scalable way
On a daily basis, I am adding values from the following table (table1):
Id Name Value
1 'Name 1' 789
2 'Name 2' 789
3 'Name 3' 789
To the text field (Values) in the following ...
- 31
-1
votes
1
answer
267
views
Postgres 12 Scalability using table Partitioning and Foreign Data Wrappers
I have looked through the archives and I cant find any discussions around the following topic.
I have a fairly in depth question that I would appreciate some guidance with.
Current environment
...
- 11
0
votes
1
answer
448
views
Does a read replica help if the problem are heavy queries?
The application that our team is responsible for is having ever more often DB performance problems. The application querying it doesn't throw lots of concurrent queries, the problem we have are rather ...
- 101
2
votes
1
answer
1k
views
mongodb read write locks [closed]
I'm currently investigating the possibility of using mongodb. This would contain the a collection of 100million+ documents. I need a process that will be constantly running deleting documents older ...
-3
votes
2
answers
394
views
Replication and sharding combinations
In modern relational databases, what combinations of replication and sharding are possible?
Actually, I'm interested in the following strategy. Is it possible to replicate a small table in order to ...
user173272
0
votes
1
answer
566
views
is vertical scaling good idea for a mysql database server?
I operate a mariadb/mysql database server and i do not want to get into the problems with managing a multi-cluster database server
i have access to huge CPU and RAM and STORAGE when needed
so my ...
- 335
4
votes
0
answers
651
views
What is the right way to increment like/comment counters on CosmosDB?
I'm currently working on a new app which will use a NOSQL DB (particularly Cosmos DB) for data scalability. I need to implement a way to update counters every time a post receives a like or comment. ...
- 141
1
vote
0
answers
725
views
Which is better schema for given discussion forum, MongoDB
Lets have two variants of nosql collection for storing discussion comments.
Assume discussion, where is only one thread/root comment with 100 replies.
A./
collection: discussion_thread (id, ...
- 111
1
vote
1
answer
662
views
How to deal with unbounded growth in cassandra shards
Let's say we have a website like youtube where we have a comment session per e video. Users want to be able to comment and to see the comments made by other users by comment creation time. This is ...
- 111
0
votes
2
answers
94
views
Cassandra Cluster storage
I want to try out Cassandra Cluster. My main question is about scalability.
Every node in a Cassandra cluster has the same copy of data. So when I have in total 1 TB and have 5 nodes -> 5TB.
This at ...
- 103
0
votes
0
answers
379
views
Alternative to Mutli-line Table Valued Functions
I currently have a relatively large MTVF returning a set of data, the code I have provided below is a small subsection of what I have in place. All but 1 part of my function is built from a Cte chain ...
- 333
3
votes
1
answer
5k
views
Database slows down at millions of rows, how to optimize schema and queries?
I am building a database to store Ethereum Transactions that are connected to specific addresses. Once the database table eth_addresses_tx starts hitting 3 million + rows, things start slowing down a ...
- 31
0
votes
1
answer
75
views
Database for timeseries data [closed]
I work in a fintech company and we are redesigning our data system.
We have around 120 GB of compressed data stored in the filesystem.
The data concerns around 80K different timeseries each one of ...
- 101
1
vote
2
answers
183
views
Aggregating high volumes of data from many MySQL servers
I have 4 servers of MySQL, each receives around 4 million records a day.
I need to aggregate all the inserted records into another main server after doing some mapping processing to the aggregated ...
- 113
1
vote
1
answer
1k
views
Organizing a comments table: How to justify the choice?
Business domain description
Some of the entity types of interest with respect to the relevant business domain are money, user and progress.
The users may need to post comments about money and ...
- 177
0
votes
2
answers
179
views
MySQL innodb scaling [closed]
I am using Jamf to manage 25K Macs and 13K iOS devices. Every time I try to update iOS apps or configuration profiles for a large number of devices my database gets overwhelmed. I have MySQL 5.6 ...
47
votes
6
answers
11k
views
Is there any limit to the number of databases you can put on one SQL server?
I'm setting up a SaaS system, where we're planning to give each customer their own database. The system is already set up so that we can easily scale out to additional servers if the load becomes too ...
- 2,813
8
votes
1
answer
2k
views
Redesign the storage of large amounts of sensor data
I have been tasked to implement/redesign a solution that will store weather data from a sensor array. The array will consist of ~40 towers, each with about ~10 sensors each that'll sample atmospheric ...
- 83
0
votes
1
answer
41
views
Searching for scalable storage solution
In one project I save statuses of processing in MySQL table:
`id` `name` `email` `has_apples` `has_bananas`
1 Bob a@b.c 1 0
2 Mike c@d.e 0 1
3 ...
- 121
7
votes
2
answers
3k
views
Scaling Azure SQL database will make my website goes down?
I have a website using Azure SQL databases, and I have some questions:
How does the scaling process work? Is there a flow, diagram that illustrates it?
Will the scaling cause any downtime in my ...
4
votes
2
answers
1k
views
How really big applications (such as facebook) scale their database [closed]
I am curious of how one will scale huge database for example facebook DB. What I know is the application is deployed somehow to multiple data centers which means that each "node" in each data center ...
- 43
2
votes
1
answer
80
views
Maintainability of schema changes during runtime of an app [closed]
We have an application which accepts JSON objects from X number of third party services and unfortunately we're locked into using SQL to store it. The objects must be flat (no nesting), but each third ...
- 121
0
votes
1
answer
46
views
Strategy to scale a comparison table query and load time
given a product table
PRODUCT
id some_column
1 10
2 2
3 8
when comparing all them I get a triangular matrix that I load from a csv file
COMPARISION
id product_left product_right ...
- 101
1
vote
1
answer
39
views
Is it possible to delegate responsibility for specific indexes for a collection to specific shards in a mongoDB database?
Is it possible to delegate responsibility for specific indexes for a collection to specific shards in a mongoDB database?
For instance, say I have a collection called 'books':
{ title: <string>...
- 119
14
votes
3
answers
9k
views
Should a multi tenant system with SQL Server 2016, Shard or have Tenant isolation via separate database per tenant?
Given the use case:
Tenant data should not cross talk, one tenant does not need another
tenant's data.
Each tenant could potentially have large historical data volume.
SQL Server is hosted in AWS ...
- 355
1
vote
1
answer
64
views
Scalability assessment for a simple auditing system based on an ORM
I was asked to create a simple audit system for a Web application. Due to some constraints I have chosen a "dirt and quick" way. This was possible due to small activity in that particular application (...
- 1,199
0
votes
1
answer
79
views
Need Expert suggestion for choosing best DB scalability
I am looking best opinion on choosing best DB scalability based on below requirement.
Application
App install tracking platform (Ad serving).
It's used to track click, impression, app install.
...
- 121
3
votes
4
answers
6k
views
Is it better to have large tables or many tables (MySQL)
In terms of performance, is it advisable to have only a few tables with a lot of rows or a lot of tables with only a few rows? In my instance, I have groups that I am keeping data separately for. Is ...
Zach Panzarino
0
votes
1
answer
41
views
Why does a query on 4 times more data take 300 times more processing?
To score the price preference of our individual customers in, we run the below query.
Basically it is a weighted average of a product score. The score depends on the product group and the packaging ...
- 85
1
vote
1
answer
72
views
Storing email inbox information
I am working on a webmail client. I want to store the remote mailboxes information in a Postgres database. As a general rule of thumb, most IMAP servers will have a similar default folder structure ...
- 113
1
vote
0
answers
32
views
How to design for scalable set operations?
This is the requirement for an educational application to teach words to language learners:
Learner can declare that he/she knows a given word
Learner can memorize the given word (to be managed by ...
- 1,277
1
vote
2
answers
163
views
scaling options for mysql on linux vm (ubuntu) : how to scale when the RAID 0 is out of space
I was asked to give a scaling road map for our mysql database.
Currently, I have a single mysql instance v5.6.28 running on an ubuntu VM on azure. It has a (software) RAID 0 disk made with two 1TBs.
...
- 11
2
votes
0
answers
888
views
MongoDB daily document growth rate
Is MongoDB feasibly suited to handle 90,000,000 documents per day? Version 3.2.
I did some rough estimates on my data size. There will be about 90,000,000 recorded events per day, each a maximum size ...
- 123
3
votes
1
answer
273
views
Are Availability Groups feasible for 4 TB of data?
Currently we have a database which is 4 TB in size. We would like to put it in a 2014 Availability Group to scale out reads. Data gets changed a lot inside this database.
I have following questions:
...
- 299
0
votes
1
answer
75
views
MySQL - design help for scale
I will be using Amazon Web Services RDS service to host a MySQL 5.7 server (Currently AWS do not support 5.7, but they have announced they are working on implementing this version and it will be out ...
- 295
2
votes
1
answer
327
views
Design a many-to-many table for scale [closed]
I am thinking about scaling issues with many-to-many join tables.
I have a constructed example as following:
Many users have many wishlists.
On every wishlist are many items.
If I scale this table ...
- 29
9
votes
2
answers
3k
views
Postgres with pgpool architecture
Below is an example pgpool architecture:
This implies that you only need to have pgpool on a single server; is this true? When I look at the configuration I also see that you configure backends ...
user20459
2
votes
0
answers
108
views
PostgreSQL on Cluster
question - I'm working on a project that involves heavy computation and will continually grow in terms of the database it works with. I want to build a small cluster (NUCs I'm thinking) the run a ...
- 21
1
vote
3
answers
2k
views
Scaling for MariaDB / Percona / Galera Scaling
I ventured into the world of NoSQL database and as great as they are the lack of true ACID compliance has been a nightmare resulting in a lot of bad data so I am looking into relational databases.
I ...
- 295
4
votes
1
answer
20k
views
PostgreSQL: remaining connection slots are reserved for non-replication superuser connections, too many clients already
We have web application using Golang, PostgreSQL, and sqlx (adapter, connection pooler), that each request requires 1 to 8 queries, sometimes 1 transaction with 5-8 select and 5-8 insert queries.
We ...
- 1,199