Questions tagged [scalability]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
7
votes
1answer
5k views

Confused about load balancing and horizontal scaling in postgresql [closed]

Please correct me if I am wrong but I guess handling more requests and load by adding more machines or balancing the load between multiple servers is horizontal scalling. So, if I add more servers, ...
0
votes
0answers
15 views

Which design could be better for scale a MongoDB query for this usecase

Context Imagine that you have two MongoDB collections, one called nodes and the other called mirrors. Each mirror is replica of a node. So, in case the node is removed (for whatever reason), you ...
0
votes
1answer
97 views

Does a read replica help if the problem are heavy queries?

The application that our team is responsible for is having ever more often DB performance problems. The application querying it doesn't throw lots of concurrent queries, the problem we have are rather ...
4
votes
1answer
198 views

Continuously Adding SSDs to an existing PostgreSQL Server

An Ubuntu 18.04 server dedicated to running a PostgreSQL 11.2 server (with TimescaleDB extension) will be running out of disk space soon so new SSD disks will need to be added to the machine to ...
2
votes
2answers
6k views

Stress test MySQL with queries captured with general log in MySQL

Is there any tool available which can do stress testing using the log file created by MySQL general log? After a lot of search on google I found few stress testing tools which only use some benchmarks ...
23
votes
4answers
15k views

What does horizontal scaling mean?

In database context, I have come across horizontal scalability as one of the advantages of the NOSQL databases. What does the term mean? CouchDB on Wikipedia NoSQL on Wikipedia How would it compare ...
0
votes
2answers
42 views

In what range can you expect a cloud based Sql Server data reading in mb/s? [closed]

Say everything is optimized, queries are simple, indexes are good, etc. Should a SQL (Azure) database where you spend about 150 to 300 euros per month easily handle data reading with about 250-500 mb ...
3
votes
1answer
83 views

Concatenate a string value from table A to a text field in table B, in a scalable way

On a daily basis, I am adding values from the following table (table1): Id Name Value 1 'Name 1' 789 2 'Name 2' 789 3 'Name 3' 789 To the text field (Values) in the following ...
19
votes
3answers
29k views

PostgreSQL High Availability/Scalability using HAProxy and PGBouncer

I have multiple PostgreSQL servers for a web application. Typically one master and multiple slaves in hot standby mode (asynchronous streaming replication). I use PGBouncer for connection pooling: ...
0
votes
0answers
33 views

MySQL - Updating 100m+ rows vs truncating and inserting

I've got a table in the following format: id: primary key foreign_id: foreign key code_1: int w index code_2: int with index I have a file that comes in every week with 200m records in the format: ...
0
votes
0answers
38 views

Scale out methods with high availability [duplicate]

We have a two site AlwaysOn Cluster and looking to do a High Available multi-site write solution. We have looked at Replication, Sharding, and SQL Elastic options for having multi-site write for a ...
2
votes
3answers
668 views

Is it realistic to expect a database of size over 50 GB (some 300+ million records) to be imported within 3-4 hours on a single server (MySQL)?

I have hundreds of millions of rows in a text/csv file (genomics database btw - each record is less than 255 characters long...). Ideally I'd like to make them searchable since right now my best bet ...
1
vote
2answers
51 views

How to scale with MySQL (when not ready to scale properly)

We are using MySql. My situation I have a large number of tables with millions of rows each. Those tables are updated every second and are used both for adding info and retrieving info. Each table ...
1
vote
1answer
15 views

Can I use only OrientDB without any RDBMS?

I'm struggling with this problem and I really need an advice. My project mainly needs a Social structure, users can be connected with some friendship relations, can create groups and so on. In this ...
0
votes
0answers
666 views

TiDB vs Vitess - How should I base my decision?

I need a scalable MySQL-compatable solution that can scale to multiple installs across a network, handling millions of records on several tables with tens of thousands of concurrent requests per-...
6
votes
2answers
2k views

Scaling Azure SQL database will make my website goes down?

I have a website using Azure SQL databases, and I have some questions: How does the scaling process work? Is there a flow, diagram that illustrates it? Will the scaling cause any downtime in my ...
-1
votes
1answer
228 views

Postgres 12 Scalability using table Partitioning and Foreign Data Wrappers

I have looked through the archives and I cant find any discussions around the following topic. I have a fairly in depth question that I would appreciate some guidance with. Current environment ...
8
votes
3answers
3k views

SQL Server - Multiple running totals

I've got a base table with transactions and I need to create a table with running totals. I need them to be per account and and also have a few running totals for each account (depending on the ...
3
votes
1answer
3k views

Database slows down at millions of rows, how to optimize schema and queries?

I am building a database to store Ethereum Transactions that are connected to specific addresses. Once the database table eth_addresses_tx starts hitting 3 million + rows, things start slowing down a ...
2
votes
1answer
869 views

mongodb read write locks [closed]

I'm currently investigating the possibility of using mongodb. This would contain the a collection of 100million+ documents. I need a process that will be constantly running deleting documents older ...
6
votes
2answers
8k views

Very Slow MySQL Queries, even with indexes

I have a relatively large 4-deep relational data setup like this: client_applications: (potentially 1,000's of records)    - ...    - account_id    - ...
17
votes
2answers
9k views

Scaling PostgreSQL TRIGGER(s)

How Postgres triggers mechanism scales ? We have a large PostgreSQL installation and we are trying to implement an event based system using log tables and TRIGGER(s). Basically we would like to ...
45
votes
6answers
10k views

Is there any limit to the number of databases you can put on one SQL server?

I'm setting up a SaaS system, where we're planning to give each customer their own database. The system is already set up so that we can easily scale out to additional servers if the load becomes too ...
-3
votes
2answers
288 views

Replication and sharding combinations

In modern relational databases, what combinations of replication and sharding are possible? Actually, I'm interested in the following strategy. Is it possible to replicate a small table in order to ...
2
votes
4answers
3k views

Is it better to have large tables or many tables (MySQL)

In terms of performance, is it advisable to have only a few tables with a lot of rows or a lot of tables with only a few rows? In my instance, I have groups that I am keeping data separately for. Is ...
0
votes
1answer
347 views

is vertical scaling good idea for a mysql database server?

I operate a mariadb/mysql database server and i do not want to get into the problems with managing a multi-cluster database server i have access to huge CPU and RAM and STORAGE when needed so my ...
9
votes
2answers
3k views

Postgres with pgpool architecture

Below is an example pgpool architecture: This implies that you only need to have pgpool on a single server; is this true? When I look at the configuration I also see that you configure backends ...
4
votes
0answers
364 views

What is the right way to increment like/comment counters on CosmosDB?

I'm currently working on a new app which will use a NOSQL DB (particularly Cosmos DB) for data scalability. I need to implement a way to update counters every time a post receives a like or comment. ...
1
vote
0answers
566 views

Which is better schema for given discussion forum, MongoDB

Lets have two variants of nosql collection for storing discussion comments. Assume discussion, where is only one thread/root comment with 100 replies. A./ collection: discussion_thread (id, ...
0
votes
2answers
77 views

Cassandra Cluster storage

I want to try out Cassandra Cluster. My main question is about scalability. Every node in a Cassandra cluster has the same copy of data. So when I have in total 1 TB and have 5 nodes -> 5TB. This at ...
89
votes
5answers
23k views

Why can't RDBM's cluster the way NoSQL does?

One of the big plusses for nosql DBMS is that they can cluster more easily. Supposedly with NoSQL you can create hundreds of cheap machines that store different pieces of data and query it all at ...
1
vote
1answer
565 views

How to deal with unbounded growth in cassandra shards

Let's say we have a website like youtube where we have a comment session per e video. Users want to be able to comment and to see the comments made by other users by comment creation time. This is ...
43
votes
1answer
35k views

Scalability limitations of PostgreSQL and MySQL

I've heard that performance of non-sharded relational database such as MySQL or PostgreSQL "breaks" beyond 10 TB. I suspect that limits as such do exist as one would not come up with Netezza, ...
1
vote
3answers
1k views

Scaling for MariaDB / Percona / Galera Scaling

I ventured into the world of NoSQL database and as great as they are the lack of true ACID compliance has been a nightmare resulting in a lot of bad data so I am looking into relational databases. I ...
8
votes
1answer
2k views

Redesign the storage of large amounts of sensor data

I have been tasked to implement/redesign a solution that will store weather data from a sensor array. The array will consist of ~40 towers, each with about ~10 sensors each that'll sample atmospheric ...
0
votes
0answers
247 views

Alternative to Mutli-line Table Valued Functions

I currently have a relatively large MTVF returning a set of data, the code I have provided below is a small subsection of what I have in place. All but 1 part of my function is built from a Cte chain ...
1
vote
2answers
105 views

Aggregating high volumes of data from many MySQL servers

I have 4 servers of MySQL, each receives around 4 million records a day. I need to aggregate all the inserted records into another main server after doing some mapping processing to the aggregated ...
0
votes
1answer
64 views

Database for timeseries data [closed]

I work in a fintech company and we are redesigning our data system. We have around 120 GB of compressed data stored in the filesystem. The data concerns around 80K different timeseries each one of ...
1
vote
1answer
646 views

Organizing a comments table: How to justify the choice?

Business domain description Some of the entity types of interest with respect to the relevant business domain are money, user and progress. The users may need to post comments about money and ...
0
votes
2answers
175 views

MySQL innodb scaling [closed]

I am using Jamf to manage 25K Macs and 13K iOS devices. Every time I try to update iOS apps or configuration profiles for a large number of devices my database gets overwhelmed. I have MySQL 5.6 ...
14
votes
4answers
26k views

Is it better to separate a big query into multiple smaller queries?

There are situations which require having really big query joining several tables together with sub select statements in them to produce the desired results. My question is, should we consider ...
2
votes
1answer
284 views

Design a many-to-many table for scale [closed]

I am thinking about scaling issues with many-to-many join tables. I have a constructed example as following: Many users have many wishlists. On every wishlist are many items. If I scale this table ...
3
votes
4answers
3k views

Is it feasible to have MySQL in-memory storage engine utilize 512 GB of RAM?

I am considering scaling up instead of scaling out. Therefore, I am interested to hear whether it is feasible to utilize the MySQL in-memory storage engine for a database that is 500+ GB, given that ...
7
votes
2answers
14k views

Proper storage size estimation technique for PostgreSQL

We are preparing a PostgreSQL database for production usage and we need to estimate the storage size for this database. We are a team of developers with low expertise on database administration, so we ...
1
vote
2answers
913 views

Experience with Scalearc sharding?

Does anyone have experience with Scalearc sharding and it's ability to scale ? Does Scalearc needs to have the whole data in memory or does it behave well with disk storage ? I'm thinking to big ...
2
votes
2answers
1k views

Experience using ScaleArc in test or production?

Has anyone had any experience using ScaleArc? My CTO has asked my thoughts on it, and I have seen no information out there regarding real-world experiences.
5
votes
2answers
2k views

How much data is needed to show MySQL Cluster's performance scaling vs. InnoDB

I am evaluating MySQL Cluster as a possible replacement for an InnoDB schema. So far, I have tested it with 10s of MB of data, and found MySQL Cluster slower than InnoDB; however, I have been told ...
0
votes
1answer
40 views

Searching for scalable storage solution

In one project I save statuses of processing in MySQL table: `id` `name` `email` `has_apples` `has_bananas` 1 Bob a@b.c 1 0 2 Mike c@d.e 0 1 3 ...
2
votes
1answer
59 views

Maintainability of schema changes during runtime of an app [closed]

We have an application which accepts JSON objects from X number of third party services and unfortunately we're locked into using SQL to store it. The objects must be flat (no nesting), but each third ...
4
votes
1answer
528 views

NuoDB with Amazon S3

I think about developing my new mobile and web app against NuoDB. I believe in their technology. The thing that I care the most is backup and restore in case of data corruption and read / write speed ...