Skip to main content
Database Administrators

Return to Question

Post Closed as "Needs more focus" by mustaccio, user212533, John Eisbrener, McNets, HandyD
no such thing as dumb questions; formatted question;
Source Link
John K. N.
John K. N.
  • 18.3k
  • 14
  • 54
  • 115

So for many days, I had a question in mind.

howHow do modern data warehouses tackle frequent small writes? esp. when streaming data is one of the sources? esp. when streaming data is one of the sources?
Kafkae.g. Kafka/Kinesis => DW(Snowflake, Teradata, Oracle ADW, etc)

I was under the impression that Since since data warehouse tables are typically highly denormalized and columnar  (for quick performance for reporting queries to avoid joins); they they are slow for frequent small writes But, but have good performance for reporting style SELECTSELECT statements. Hence the concept of bulk nightly uploads from OLTP data sources to OLAP data warehouses.

  • What has changed in the modern DW internal architecture?
  • Is there a staging area within DW itself, where data lands and then it is aggregated, stats are collected and then denormalized before it finally rests into actual DW tables powering the reporting?

what has changed in the modern DW internal architecture? Is there a staging area within DW itself, where data lands and then it is aggregated, stats are collected and then denormalized before it finally rests into actual DW tables powering the reporting? II am interested in knowing how does it internally works at a high level. I

I know this is a dumbbasic question, but this is my understanding from my school days hence I am pretty sure it is out of date.

So for many days, I had a question in mind.

how do modern data warehouses tackle frequent small writes? esp. when streaming data is one of the sources? Kafka/Kinesis => DW(Snowflake, Teradata, Oracle ADW, etc)

I was under the impression that Since data warehouse tables are typically highly denormalized and columnar(for quick performance for reporting queries to avoid joins); they are slow for frequent small writes But have good performance for reporting style SELECT statements. Hence the concept of bulk nightly uploads from OLTP data sources to OLAP data warehouses.

what has changed in the modern DW internal architecture? Is there a staging area within DW itself, where data lands and then it is aggregated, stats are collected and then denormalized before it finally rests into actual DW tables powering the reporting? I am interested in knowing how does it internally works at a high level. I know this is a dumb question, this is my understanding from my school days hence I am pretty sure it is out of date.

So for many days, I had a question in mind.

How do modern data warehouses tackle frequent small writes? esp. when streaming data is one of the sources?
e.g. Kafka/Kinesis => DW(Snowflake, Teradata, Oracle ADW, etc)

I was under the impression that since data warehouse tables are typically highly denormalized and columnar  (for quick performance for reporting queries to avoid joins) they are slow for frequent small writes, but have good performance for reporting style SELECT statements. Hence the concept of bulk nightly uploads from OLTP data sources to OLAP data warehouses.

  • What has changed in the modern DW internal architecture?
  • Is there a staging area within DW itself, where data lands and then it is aggregated, stats are collected and then denormalized before it finally rests into actual DW tables powering the reporting?

I am interested in knowing how does it internally works at a high level.

I know this is a basic question, but this is my understanding from my school days hence I am pretty sure it is out of date.

Source Link
Libertarian
Libertarian
  • 3
  • 1

how do modern data warehouses tackle frequent small writes? esp. when streaming data is one of the sources?

So for many days, I had a question in mind.

how do modern data warehouses tackle frequent small writes? esp. when streaming data is one of the sources? Kafka/Kinesis => DW(Snowflake, Teradata, Oracle ADW, etc)

I was under the impression that Since data warehouse tables are typically highly denormalized and columnar(for quick performance for reporting queries to avoid joins); they are slow for frequent small writes But have good performance for reporting style SELECT statements. Hence the concept of bulk nightly uploads from OLTP data sources to OLAP data warehouses.

what has changed in the modern DW internal architecture? Is there a staging area within DW itself, where data lands and then it is aggregated, stats are collected and then denormalized before it finally rests into actual DW tables powering the reporting? I am interested in knowing how does it internally works at a high level. I know this is a dumb question, this is my understanding from my school days hence I am pretty sure it is out of date.