Questions tagged [scalability]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
4 answers
89 views

Scalability of Postgres for table with large number of indexed columns

I have a Postgres table with a large number of indexed columns (roughly 100 indexed columns total, and yes, I need them all, and yes, they all need to be separately indexed). Any row update causes all ...
Luke Hutchison's user avatar
0 votes
1 answer
33 views

Why use $group / group_by instead of multiple requests?

Let's suppose that there are 100 types of dogs scattered across the documents of my collection. If I need to group by types of dogs and then compute some aggregate statistic about each type of dog, ...
Bear Bile Farming is Torture's user avatar
1 vote
2 answers
89 views

Scaling from Multiple Database to Single Database Architecture in SQL Server

My application is centered around self-contianed "workspaces". For many really good reasons (everything from management to security), we have always had a one-database-per-workspace ...
Peter Moore's user avatar
1 vote
1 answer
191 views

What makes many NoSQL databases non-ACID compliant when both SQL and NoSQL can scale horizontally

A lot of the difference between SQL and NoSQL that is talked about focuses on the fact that SQL support ACID properties while in many NoSQL dbs ACID is compromised. I'm unsure of the reason but ...
asn's user avatar
  • 127
0 votes
1 answer
144 views

NoSQL vs SQL during sharding when a single read operation only hits one shard

I've a use case where I'm stuck between figuring out whether to use SQL or NoSQL db. The db has 2 fields - A (PK), B A is 8 char-long, while B is 100-400 char long. The operations are simply: Write: ...
asn's user avatar
  • 127
2 votes
0 answers
332 views

Postgres: One-to-many relationship with WHERE, ORDER BY and LIMIT not scalable?

I'm currently using Postgres 12. A few years ago, I set up the following model: a lot table with a one-to-many relationship to a line table. My idea was to mutualize data that are common for multiple ...
K.Banks's user avatar
  • 21
-1 votes
1 answer
86 views

Real Time Analytics: Which database?

We are currently using MongoDB, and it is performing well for our needs. However, we are looking to support better real-time analytics and aggregations, which MongoDB doesn't handle effectively. ...
Nedim's user avatar
  • 99
0 votes
1 answer
300 views

Galera cluster performance, node count, and where to write?

As far I understand Galera does not scale write performance, best case it does not degrade it, compared to one single server, so we use Galera for HA. (read performance may benefit and it can be load ...
g.pickardou's user avatar
0 votes
0 answers
147 views

Is Postgres appropriate for this application? (1 billion inserts a year)

I am currently working on drug record management system which needs to record and retain insurance billing information for patients. So far, all the typical information (patient profile, prescriber ...
d124y's user avatar
  • 1
8 votes
3 answers
5k views

Database design to handle millions of rows in MySQL

We are running an application that is collecting data much faster than we anticipated. Trying to addapt to that, we are doing a redesig of the database. After reading this, this and this, I am not ...
Javi M's user avatar
  • 61
0 votes
0 answers
16 views

Keeping up-to-date values aggregated based on some time period

I'm considering a few solutions for a problem involving some sort of a time series data. Imagine: you have a system that keeps track of financial transactions in multiple currencies. To keep it simple,...
Psycho Punch's user avatar
1 vote
0 answers
438 views

Multiple databases vs partitioning in PostgreSQL

Consider following database schema for PostgreSQL (I'm using 13.x.x) Users table - just place to store users and do authentication Projects - projects which are created by specific user Driver/Event/...
DocC's user avatar
  • 111
0 votes
0 answers
1k views

Handling multiple users/ databases in postgres

Whats the best way to organize seperate companies/users in postgres? I've developed a multi user platform, where users have access to only their companies data. I'm no DB engineer and initially the ...
Lewis Morris's user avatar
1 vote
2 answers
1k views

Is SqlServer or NoSql more cost efficient when scaling horizontally

RDBMS are traditionally better for vertical scaling since it's harder to do horizontal scaling. As a result NoSql databases are cheaper since they can more easily scale horizontally. However in recent ...
CorrieJanse's user avatar
9 votes
1 answer
2k views

Is Postgres ideal for Many to Many data (M:N) which keeps increasing?

I have a use-case where the data is many to many and needs a wide range of querying functionality. Participants and Events A User/Participant can register for multiple events. And each event can have ...
Ayyappa's user avatar
  • 203
0 votes
2 answers
191 views

In what range can you expect a cloud based Sql Server data reading in mb/s? [closed]

Say everything is optimized, queries are simple, indexes are good, etc. Should a SQL (Azure) database where you spend about 150 to 300 euros per month easily handle data reading with about 250-500 mb ...
Dirk Boer's user avatar
  • 365
0 votes
0 answers
40 views

Scale out methods with high availability [duplicate]

We have a two site AlwaysOn Cluster and looking to do a High Available multi-site write solution. We have looked at Replication, Sharding, and SQL Elastic options for having multi-site write for a ...
courth's user avatar
  • 1
1 vote
2 answers
141 views

How to scale with MySQL (when not ready to scale properly)

We are using MySql. My situation I have a large number of tables with millions of rows each. Those tables are updated every second and are used both for adding info and retrieving info. Each table ...
Vicky Dallas's user avatar
1 vote
0 answers
87 views

MySQL - Updating 100m+ rows vs truncating and inserting

I've got a table in the following format: id: primary key foreign_id: foreign key code_1: int w index code_2: int with index I have a file that comes in every week with 200m records in the format: ...
user2311710's user avatar
1 vote
1 answer
32 views

Can I use only OrientDB without any RDBMS?

I'm struggling with this problem and I really need an advice. My project mainly needs a Social structure, users can be connected with some friendship relations, can create groups and so on. In this ...
Philip's user avatar
  • 111
1 vote
0 answers
1k views

TiDB vs Vitess - How should I base my decision?

I need a scalable MySQL-compatable solution that can scale to multiple installs across a network, handling millions of records on several tables with tens of thousands of concurrent requests per-...
xendi's user avatar
  • 203
5 votes
1 answer
656 views

Continuously Adding SSDs to an existing PostgreSQL Server

An Ubuntu 18.04 server dedicated to running a PostgreSQL 11.2 server (with TimescaleDB extension) will be running out of disk space soon so new SSD disks will need to be added to the machine to ...
Nyxynyx's user avatar
  • 1,121
2 votes
3 answers
4k views

Is it realistic to expect a database of size over 50 GB (some 300+ million records) to be imported within 3-4 hours on a single server (MySQL)?

I have hundreds of millions of rows in a text/csv file (genomics database btw - each record is less than 255 characters long...). Ideally I'd like to make them searchable since right now my best bet ...
kevinjordan's user avatar
3 votes
1 answer
676 views

Concatenate a string value from table A to a text field in table B, in a scalable way

On a daily basis, I am adding values from the following table (table1): Id Name Value 1 'Name 1' 789 2 'Name 2' 789 3 'Name 3' 789 To the text field (Values) in the following ...
Scarabas's user avatar
-1 votes
1 answer
385 views

Postgres 12 Scalability using table Partitioning and Foreign Data Wrappers

I have looked through the archives and I cant find any discussions around the following topic. I have a fairly in depth question that I would appreciate some guidance with. Current environment ...
Francois's user avatar
0 votes
1 answer
2k views

Does a read replica help if the problem are heavy queries?

The application that our team is responsible for is having ever more often DB performance problems. The application querying it doesn't throw lots of concurrent queries, the problem we have are rather ...
antonro's user avatar
  • 101
2 votes
1 answer
2k views

mongodb read write locks [closed]

I'm currently investigating the possibility of using mongodb. This would contain the a collection of 100million+ documents. I need a process that will be constantly running deleting documents older ...
BeginnerDBADan's user avatar
-3 votes
2 answers
565 views

Replication and sharding combinations

In modern relational databases, what combinations of replication and sharding are possible? Actually, I'm interested in the following strategy. Is it possible to replicate a small table in order to ...
user avatar
0 votes
1 answer
994 views

is vertical scaling good idea for a mysql database server?

I operate a mariadb/mysql database server and i do not want to get into the problems with managing a multi-cluster database server i have access to huge CPU and RAM and STORAGE when needed so my ...
uberrebu's user avatar
  • 441
4 votes
0 answers
850 views

What is the right way to increment like/comment counters on CosmosDB?

I'm currently working on a new app which will use a NOSQL DB (particularly Cosmos DB) for data scalability. I need to implement a way to update counters every time a post receives a like or comment. ...
user246392's user avatar
1 vote
0 answers
877 views

Which is better schema for given discussion forum, MongoDB

Lets have two variants of nosql collection for storing discussion comments. Assume discussion, where is only one thread/root comment with 100 replies. A./ collection: discussion_thread (id, ...
ooouuiii's user avatar
  • 111
1 vote
1 answer
862 views

How to deal with unbounded growth in cassandra shards

Let's say we have a website like youtube where we have a comment session per e video. Users want to be able to comment and to see the comments made by other users by comment creation time. This is ...
Lucas Piske's user avatar
0 votes
2 answers
118 views

Cassandra Cluster storage

I want to try out Cassandra Cluster. My main question is about scalability. Every node in a Cassandra cluster has the same copy of data. So when I have in total 1 TB and have 5 nodes -> 5TB. This at ...
Fabian Lurz's user avatar
0 votes
0 answers
487 views

Alternative to Mutli-line Table Valued Functions

I currently have a relatively large MTVF returning a set of data, the code I have provided below is a small subsection of what I have in place. All but 1 part of my function is built from a Cte chain ...
Krishn's user avatar
  • 373
3 votes
1 answer
8k views

Database slows down at millions of rows, how to optimize schema and queries?

I am building a database to store Ethereum Transactions that are connected to specific addresses. Once the database table eth_addresses_tx starts hitting 3 million + rows, things start slowing down a ...
cyberkali's user avatar
0 votes
1 answer
81 views

Database for timeseries data [closed]

I work in a fintech company and we are redesigning our data system. We have around 120 GB of compressed data stored in the filesystem. The data concerns around 80K different timeseries each one of ...
fusion's user avatar
  • 101
1 vote
2 answers
315 views

Aggregating high volumes of data from many MySQL servers

I have 4 servers of MySQL, each receives around 4 million records a day. I need to aggregate all the inserted records into another main server after doing some mapping processing to the aggregated ...
Ahmad Tawila's user avatar
1 vote
1 answer
2k views

Organizing a comments table: How to justify the choice?

Business domain description Some of the entity types of interest with respect to the relevant business domain are money, user and progress. The users may need to post comments about money and ...
Nikita's user avatar
  • 187
0 votes
2 answers
200 views

MySQL innodb scaling [closed]

I am using Jamf to manage 25K Macs and 13K iOS devices. Every time I try to update iOS apps or configuration profiles for a large number of devices my database gets overwhelmed. I have MySQL 5.6 ...
Mike Donovan's user avatar
47 votes
6 answers
13k views

Is there any limit to the number of databases you can put on one SQL server?

I'm setting up a SaaS system, where we're planning to give each customer their own database. The system is already set up so that we can easily scale out to additional servers if the load becomes too ...
Shaul Behr's user avatar
  • 2,933
8 votes
1 answer
2k views

Redesign the storage of large amounts of sensor data

I have been tasked to implement/redesign a solution that will store weather data from a sensor array. The array will consist of ~40 towers, each with about ~10 sensors each that'll sample atmospheric ...
Julia's user avatar
  • 83
0 votes
1 answer
45 views

Searching for scalable storage solution

In one project I save statuses of processing in MySQL table: `id` `name` `email` `has_apples` `has_bananas` 1 Bob [email protected] 1 0 2 Mike [email protected] 0 1 3 ...
Igor's user avatar
  • 121
8 votes
2 answers
4k views

Scaling Azure SQL database will make my website goes down?

I have a website using Azure SQL databases, and I have some questions: How does the scaling process work? Is there a flow, diagram that illustrates it? Will the scaling cause any downtime in my ...
Vinicius Deschamps's user avatar
5 votes
2 answers
2k views

How really big applications (such as facebook) scale their database [closed]

I am curious of how one will scale huge database for example facebook DB. What I know is the application is deployed somehow to multiple data centers which means that each "node" in each data center ...
daffr32's user avatar
  • 53
2 votes
1 answer
149 views

Maintainability of schema changes during runtime of an app [closed]

We have an application which accepts JSON objects from X number of third party services and unfortunately we're locked into using SQL to store it. The objects must be flat (no nesting), but each third ...
caesay's user avatar
  • 121
0 votes
1 answer
49 views

Strategy to scale a comparison table query and load time

given a product table PRODUCT id some_column 1 10 2 2 3 8 when comparing all them I get a triangular matrix that I load from a csv file COMPARISION id product_left product_right ...
Ruben Trancoso's user avatar
1 vote
1 answer
46 views

Is it possible to delegate responsibility for specific indexes for a collection to specific shards in a mongoDB database?

Is it possible to delegate responsibility for specific indexes for a collection to specific shards in a mongoDB database? For instance, say I have a collection called 'books': { title: <string>...
Adé's user avatar
  • 119
15 votes
3 answers
10k views

Should a multi tenant system with SQL Server 2016, Shard or have Tenant isolation via separate database per tenant?

Given the use case: Tenant data should not cross talk, one tenant does not need another tenant's data. Each tenant could potentially have large historical data volume. SQL Server is hosted in AWS ...
D.S.'s user avatar
  • 365
1 vote
1 answer
67 views

Scalability assessment for a simple auditing system based on an ORM

I was asked to create a simple audit system for a Web application. Due to some constraints I have chosen a "dirt and quick" way. This was possible due to small activity in that particular application (...
Alexei's user avatar
  • 1,261
0 votes
1 answer
89 views

Need Expert suggestion for choosing best DB scalability

I am looking best opinion on choosing best DB scalability based on below requirement. Application App install tracking platform (Ad serving). It's used to track click, impression, app install. ...
Ranjeet Ranjan's user avatar