Questions tagged [data-warehouse]

A database system optimised for reporting, particularly in aggregate. Often, but not always implemented using a star schema.

80 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
6
votes
0answers
218 views

Using the Schema Generation Wizard to regenerate a previously-generated schema

Lately, I've been trying to clean up a group of projects that my company received from an outside contractor. Among them is an SSAS cube, with an associated data warehouse. The contractors generated ...
4
votes
1answer
582 views

Revision Tracking & Source Control for Azure SQL Data Warehouse

What is a good approaching for tracking incremental changes to database tables, stored procedures, etc for Azure SQL Data Warehouse? I am in the process of moving a large database over to Azure SQL ...
4
votes
2answers
891 views

Should I snowflake or duplicate it across my facts?

I'm building up a data warehouse to be used by SSAS to create cubes on, and I'm debating between two possible schemas. In my warehouse, I've got two different fact tables that tracking daily changes ...
3
votes
1answer
648 views

Dimensional Modeling of a Loan Approval Process and Fact Grain(s)

Context: I'm modeling for a lending company and I'm modeling our loan approval process. I have some basic dimensions relative to our business figured out: Time(day) Account Opportunity ...
3
votes
1answer
1k views

speed up PostgreSQL calculations via OLAP warehouse?

So I have asked these three questions: Percentile rank that takes sorted argument (or same functionality) in PostgreSQL 9.3.5 https://quant.stackexchange.com/questions/29572/building-financial-data-...
3
votes
0answers
45 views

If a relationship is (logically) deleted, is this marked in the link table or just the satellite table(s)?

I'm learning about the Data Vault data warehousing approach, and so far the examples I've seen have a DateLoaded or similar field in all tables (hub, link, and satellite), but a DateExpired or EndDate ...
3
votes
1answer
77 views

Are there best practices or standardized syntax for writing transformation logic?

This may not be the appropriate forum, but its the closest I could find. I write a lot of source to target mappings for work. These documents just describe the operations/logic/transformations/etc. ...
3
votes
0answers
1k views

An advice to design a fact table with a scd as a dimension

I have a problem of conception and I would like to get your ideas on how I'll be able to do what I am aiming. My goal is to create a datawarehouse, with three dimensions and one fact table so far. ...
2
votes
0answers
69 views

Store email in a separate Dimension or in a Degenerate Dimension?

I just started learning about dimensional modeling and I am creating a star to analyse email newsletter signups for an online business. I have a fact table that records the signups and links to a ...
2
votes
1answer
53 views

How to implement updates from mySQL operational DB to Azure SQL DB for reporting

We have an operational mySQL DB running on AWS for a transactional system and an Azure SQL DB for reporting with PowerBI. Now I'd like to regularly (e.g. every night) do an update of certain tables ...
2
votes
0answers
68 views

Data Warehouse Kimball Interest Rate and Interest Payment Amount

I am modelling a bank account. Should interest rate be in the DimCheckingAccountId (which shows ProductDimensions, or should it be in fact table, or both). Same with another LoanAccount, should ...
2
votes
3answers
3k views

Should dates in dimensional tables use dimDate?

Assuming my dimDate has a surrogate key. Should all the date columns in the dimensional tables (not the fact tables) store the surrogate key of date dimension? Or just plain date? For example, in ...
2
votes
0answers
96 views

When to Converge Dimensions in a Data Warehouse When There Are Few Common Attributes

I have two StudentDetails dimension tables in my data warehouse and I want to know whether I should converge them, considering that they have limited attributes in common (keeping in mind Kimball DW ...
2
votes
0answers
216 views

Stage database in SQL Azure

I am designing a data warehouse using Azure Databases (not Azure Data Warehouse though), I have 2 databases: the main data warehouse and a staging database. As in a traditional ETL process, the raw ...
2
votes
0answers
1k views

PostgreSQL: Deadlock on insert trigger that also updates same table.

I am trying to implement a type 2 slowly changing dimension in a PG data warehouse using a trigger function. The basic idea is that when a new row is inserted, we should check whether NEW is a ...
2
votes
0answers
234 views

How to gracefully handle a source database restore from backup in a data warehouse

We are facing a challenging situation with our data warehouse source databases. Frequently these sources databases will be restored from backups. There is a high turnover of data entry persons who ...
1
vote
0answers
23 views

History data in CUBE

Not sure if this is the right place to ask. I am working on an ETL procedure that gathers the data about medication consumption and I have something like this: Dim_Patient, the dimension for all the ...
1
vote
2answers
47 views

best approach to model relation between customers - dimension dim_customer (SCD2) which can be a HEAD/SUB customer

Hi there we have the concept of having department. each department can either be its own head or it can have a head department. (similar to each person in a organisation is a employee but some ...
1
vote
0answers
24 views

How are the relations from Relational Database mapped to DataWarehouse?

this year one of our collage courses was related to creating and managing data warehouses. There is one thing that I really struggle to understand and I would appreciate any kind of advice that you ...
1
vote
0answers
75 views

Flags Cloud SQL for Postgresql

Context: I have a process that runs daily, with different amounts of data, if the process is on the weekday (process last 30 days), for weekends (process last year). The input is of 10 tables of ...
1
vote
0answers
177 views

Star schema with both many-to-many relations and inheritance

The problem I am trying to make a star schema of the following situation, concerning transactions on a stock market. I have summarized the scenario here: Each transaction traded on the market has 1 ...
1
vote
1answer
350 views

MySQL data warehouse calculate difference over years in percentage

I'm trying to create a query to calculate the percentage difference based on a time range (e.g. WHERE dim_date.year BETWEEN 2009 AND 2011) spreaded on months. My question is how to add the new ...
1
vote
0answers
219 views

Increase frequency of updates (ETL process) to Data Warehouse

Our data warehouse is updated only once every 24 hours (at night). The ETL process takes a little over an hour to complete. I am wondering how to increase the frequency of the ETL process - so running ...
1
vote
0answers
90 views

Best Distkey and Sortkey for the tables in Redshift

I have the following tables: orders: id => primary key customer_id => store_id => order_date => datetime order_items: id => primary key order_id => FK The query I usually run ...
1
vote
0answers
802 views

Kimball Cross Reference Table/Bridge Table

I'm currently struggling to understand how to apply Kimball methodology to a certain scenario and talk of Outriggers/Cross Reference/Bridge tables are starting to get confusing. Consider the scenario:...
1
vote
0answers
162 views

Are there differences between a Fact Constellation Schema and a Galaxy Schema?

I am currently working on the two schemes and I'm just not sure if they are the same or if there are differences. In the literature both are often synonymous. Now I've read that there are differences (...
1
vote
0answers
200 views

Where to start for DW/BI project?

I work for a small technology MSP. Having just completed my Bachelor's in Information Science, with a focus in data science, my boss has asked me to build a data warehouse (I think he really means a ...
1
vote
0answers
64 views

Data transformation

so, I'm facing this issue in storing my file in my database. I have an excel file that I need to load in database. The file looks like this: The issue is that I need to store all this in columns: ...
1
vote
1answer
59 views

Data Warehouse: Remove legacy data after conversion into new system

I have a Data Warehouse that pulls data from two locations, the first being our modern system, the second being our legacy system. The legacy system consists of 200 databases with identical schemas ...
1
vote
0answers
72 views

Reversing many-to-many relationships when transforming data into a galaxy schema

I'm building a galaxy schema for our analytics and sales data from our normalized data tables. I'm unsure how to treat many-to-many relationships in the facts tables. For example, for each sale, we ...
1
vote
2answers
150 views

Cleanup / Prune Unreferenced Data in Dimension Tables

We have a star schema data warehouse running on MySQL 5.6. We keep a rolling 18 months of data in our fact tables using partitions by month. We have a number of dynamic dimension tables that are ...
1
vote
0answers
55 views

Efficient methods/tools for managing a data warehouse pulling data from multiple database applications

Until now I have used an on premise SQL Server 2012 and performed manual ETL operations to create/manage a data warehouse that contains data from multiple database application from different ...
1
vote
0answers
169 views

Star Schema Design - Multiple Sources

I am planning the design of a star schema for a data warehouse. The data will be loaded from 2 separate staging tables (Client and System). The end user will need to be able to compare any ...
1
vote
0answers
1k views

Oracle 12c fast refreshing nested materialized view problems

I have a couple of nested materialized views that fast refresh. They worked fine in Oracle 11g but since upgrading to Oracle 12c (12.1.0.2.0 on Solaris) I am occasionally getting duplicate rows (the ...
1
vote
1answer
351 views

Robust way to build a DWH without downtime

I have a process that build a data warehouse using Postgres. The process runs every day at midnight and works in three steps: Copy the production database (prod) to a temporary database (dwh_tmp) ...
1
vote
0answers
52 views

Which Azure Storage / Database Technology to make use of for large scale logging

I am looking into making use of Azure hosting services in order to develop a logging service for our company. Azure has a number of storage options available (I have listed links below). However I am ...
1
vote
0answers
387 views

ETL flow in Fact tables

I've got a simple theoretical question... If we discuss the principles, elements, design and implementation of a typical ETL flow for a dimension table then, in my understanding, (in short) I'd say ...
1
vote
0answers
31 views

Process dimension task is querying the wrong column!

I'm in the process of building a new data warehouse and I am starting out by building a few dimensions. I have created a fairly typical "Date" dimension and that seems ok. Now I am trying to create ...
1
vote
0answers
97 views

Best practices to segment people and attributes for advertising database

we have a database where we store people some attributes per person in the following tables: people (id, name, email, phone) people_attributes (id, person_id, attribute_name, attribute_value) ...
1
vote
1answer
24 views

Table for actual values

I have a database where logging happens and is processed in a data-warehouse styled environment. This means there are staging cycles where the data comes in and then is processed. For a reporting ...
1
vote
0answers
749 views

how many fact tables do I need given I want to build an OLAP for Quotation Line Item and Purchase Order Line Item report?

My client wants a report that looks something like this on a daily basis. +---------------+-------------------------+-----------------+---------------------+-------------+ | ProductNumber | ...
1
vote
2answers
568 views

Dimension for Product Properties based on Product Type - Sparse Dimension?

I am creating dimension of product properties for sales facts. Property of product depend of product type. For example: - Type = smartphone. Properties = model, OS, size - Type = book. Properties = ...
1
vote
0answers
644 views

500GB Data Warehouse - how to split it into FileGroups?

OK, so I have: 500 GB DW 5 GB in smaller DBs 220 GB TempDB 350 GB in Log files. My machine is Fujitsu Primergy 64 cores (with HT) and 192 GB RAM. I have several IO locations: 540 GB in-server HDD ...
1
vote
0answers
355 views

Should I use conventional or computed columns for trivial metrics in a fact table?

A fact table can often have measures which are basically trivial. For example Orders.NumOrders will probably be "1" for every real record (synthetic records representing "no order" may populate this ...
0
votes
0answers
25 views

Key attributes in junk dimension table

I have a fact tables with a composite primary key of 10 key columns. 6 out of 10 of the key columns are short, and with low cardinality, and also "mostly" unrelated to each other. I am ...
0
votes
0answers
14 views

What are the table naming conventions in tabular models?

With SSAS multi-dimensional models we use star schemas and cubes as our data source. With tabular models we use relational databases. Therefore, with tabular models naming our tables Dim_* and Fact_* ...
0
votes
0answers
63 views

Would Star Schema be useful for a Lakehouse table accessed from PowerBI?

In a Lakehouse, the data is stored in a datalake (deltalake in to be precise in my case) instead of a data warehouse. The data there is stored in parquet format, which is a columnar format. I am ...
0
votes
1answer
36 views

Data modeling for Warehouse

I am learning about database warehouses and data modeling. I came across this blog post. In the section, "DIFFERENT MODELS HAVE DIFFERENT PURPOSES" the author mentions that depending upon ...
0
votes
0answers
63 views

Schema and/or reporting SQL for customer acquisition cost?

This a design question around creating fact/dimension tables to query customer acquisition cost and how to properly model the tables and query them. Business Rule We spend some money per day per ...
0
votes
1answer
25 views

Leaf Joins in Data Warehouse Design

Upon modeling my very first data warehouse model (for historical stock market data), I stumbled across a data relationship that I don't know how to best account for. The fact tables include multiple ...