Questions tagged [big-data]
The big-data tag has no usage guidance.
11
questions
2
votes
1
answer
76
views
Should I use this partitioning/indexing strategy for a table that grows by about a billion rows per year?
I'm a software engineer with about 10 years of experience building Laravel web applications for small businesses. I'm pretty comfortable with database design, but it's not my greatest strength.
One of ...
- 23
0
votes
1
answer
54
views
What Databases would be recommended for historical data with over 500M rows and data ingested daily? [closed]
I'm wondering what Databases would best be suited for storing historical data for multiple clients with over 500M rows and incremental data ingested daily per client? Since data would be used for ...
- 1
1
vote
2
answers
133
views
SQL Server database with 100 terabyte of data
We have been assigned a new application where the vendor is expecting the database to grow to around 100 TB. They have mentioned SQL Server Standard Edition is compatible.
My question is whether SQL ...
- 1,039
0
votes
0
answers
30
views
Hive Mysql Metastore Row lock increases
Below is the query causing row locks in Mariadb Galera Cluster 10.2.32 (3Node Cluster).
select MT_COMMENT from AUX_TABLE where MT_KEY1='Cleaner' and MT_KEY2=0 for update
Hive is a framework, we ...
- 1,521
0
votes
1
answer
512
views
How should I configure my postgresql to create an Index for a big table (655GB) as fast as possible?
I'm importing a big dataset of medication usage (14 years of public health system data from Brazil) for a data science project.
When I first started, I created one of the tables with all the indexes (...
- 103
0
votes
0
answers
93
views
Mysql UNION ALL query - Too slow using longtext
I have this query
select t.field1 from (
select t1.field1, t1.field2 from db1.table t1
UNION ALL
select t2.field1, t2.field2 from db2.table t2
UNION ALL
select t3.field1, t3.field2 from db3.table t3
...
- 1
8
votes
3
answers
5k
views
Database design to handle millions of rows in MySQL
We are running an application that is collecting data much faster than we anticipated. Trying to addapt to that, we are doing a redesig of the database.
After reading this, this and this, I am not ...
- 51
0
votes
1
answer
198
views
Indexing first and last item of a series only
Does PostgreSQL support any kind of sparse index that would be useful for indexing only the first and last value in the table, based on a certain key?
Consider the following data:
CREATE TABLE samples ...
- 299
2
votes
1
answer
477
views
Data lake or data warehouse first?
I have been confused whether to create a data lake or a data warehouse and hope some experienced real-world professional can give me some enlightenment.
I will like to store, visualise and perform ...
- 143
0
votes
0
answers
2k
views
PostgreSQL too slow selecting 1 billion records in a table
I created a PostgreSQL db in Google Cloud SQL and have a single table that stores 1 billion records
CREATE TABLE a (
user_id INT NOT NULL,
dt DATE NOT NULL DEFAULT CURRENT_DATE,
id VARCHAR(...
- 121
0
votes
1
answer
125
views
Can I try SQL Server BIG Data Cluster for free?
I can not find information about test SQL Server Big Data Cluster in a development environment. We have some AKSes and would like to try installing SQL Server Big Data Cluster.
Is it possible to do it ...
- 45