Skip to main content

Questions tagged [big-data]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1 vote
0 answers
67 views

Efficient way to store and retrieve big amount of data

I need to store a big amount of data (about 2.5 billion of new data rows per month), but also I need a very fast way to retrieve the latest value per group on specific time point. The data looking ...
the_it_guy's user avatar
0 votes
0 answers
31 views

Data management pattern for temporarily storing/retrieving excessive quantities of data

I have been tasked with creating an application to ingest raw binary data from a software-defined radio (SDR) and display it in a web frontend. One of the purposes of this app is to allow a user to ...
moltarze's user avatar
  • 101
0 votes
1 answer
86 views

Optimizing Performance for Large PostgreSQL Database with Parent and Child Tables

Description I am currently facing performance challenges in a PostgreSQL database scenario involving a parent table and 30 child tables. Notably, none of these tables are partitioned, and some of the ...
Marco Cesarato's user avatar
2 votes
1 answer
103 views

Should I use this partitioning/indexing strategy for a table that grows by about a billion rows per year?

I'm a software engineer with about 10 years of experience building Laravel web applications for small businesses. I'm pretty comfortable with database design, but it's not my greatest strength. One of ...
Ken Schnetz's user avatar
0 votes
1 answer
200 views

What Databases would be recommended for historical data with over 500M rows and data ingested daily? [closed]

I'm wondering what Databases would best be suited for storing historical data for multiple clients with over 500M rows and incremental data ingested daily per client? Since data would be used for ...
izuu's user avatar
  • 1
1 vote
2 answers
371 views

SQL Server database with 100 terabyte of data

We have been assigned a new application where the vendor is expecting the database to grow to around 100 TB. They have mentioned SQL Server Standard Edition is compatible. My question is whether SQL ...
SQL_NoExpert's user avatar
  • 1,061
0 votes
0 answers
60 views

Hive Mysql Metastore Row lock increases

Below is the query causing row locks in Mariadb Galera Cluster 10.2.32 (3Node Cluster). select MT_COMMENT from AUX_TABLE where MT_KEY1='Cleaner' and MT_KEY2=0 for update Hive is a framework, we ...
Mannoj's user avatar
  • 1,561
0 votes
1 answer
749 views

How should I configure my postgresql to create an Index for a big table (655GB) as fast as possible?

I'm importing a big dataset of medication usage (14 years of public health system data from Brazil) for a data science project. When I first started, I created one of the tables with all the indexes (...
Jaejatae's user avatar
  • 103
0 votes
0 answers
106 views

Mysql UNION ALL query - Too slow using longtext

I have this query select t.field1 from ( select t1.field1, t1.field2 from db1.table t1 UNION ALL select t2.field1, t2.field2 from db2.table t2 UNION ALL select t3.field1, t3.field2 from db3.table t3 ...
Eric H's user avatar
  • 1
8 votes
3 answers
5k views

Database design to handle millions of rows in MySQL

We are running an application that is collecting data much faster than we anticipated. Trying to addapt to that, we are doing a redesig of the database. After reading this, this and this, I am not ...
Javi M's user avatar
  • 61
0 votes
1 answer
235 views

Indexing first and last item of a series only

Does PostgreSQL support any kind of sparse index that would be useful for indexing only the first and last value in the table, based on a certain key? Consider the following data: CREATE TABLE samples ...
Mikko Ohtamaa's user avatar
2 votes
1 answer
647 views

Data lake or data warehouse first?

I have been confused whether to create a data lake or a data warehouse and hope some experienced real-world professional can give me some enlightenment. I will like to store, visualise and perform ...
SunnyBoiz's user avatar
  • 143
0 votes
0 answers
2k views

PostgreSQL too slow selecting 1 billion records in a table

I created a PostgreSQL db in Google Cloud SQL and have a single table that stores 1 billion records CREATE TABLE a ( user_id INT NOT NULL, dt DATE NOT NULL DEFAULT CURRENT_DATE, id VARCHAR(...
Eric Lee's user avatar
  • 121
0 votes
1 answer
133 views

Can I try SQL Server BIG Data Cluster for free?

I can not find information about test SQL Server Big Data Cluster in a development environment. We have some AKSes and would like to try installing SQL Server Big Data Cluster. Is it possible to do it ...
grochu's user avatar
  • 45