-1

We have a production application which gathers data for certain type of incidents, based on a collection of fields (aprox. 1000) of different data types - string, int, double, date, time, GUID (id of selected option for dropdown lists), list of GUID's (for checkbox lists).

The current design is like this:

  1. Occurrence has a collection of TermResults;
  2. Each TermResult has a collection StringResults, IntResults, NumberResults, DateResults, TimeResults, GuidResults, ListGuidsResults;

We have a distinct table for each type of result in the database. So, reading / updating an occurrence will require joining data from about 12 tables (we have additional data stored in other tables, besides the Results ones).

The operations we need to do with the data are:

  1. Read / update an occurrence;
  2. Query database to retrieve occurrences based on very complex dynamic queries, which can be built based on those 1000 fields. For example, retrieve occurrence where field1.Value >=3 and field2.value <= '01/01/2017' or field3.value == GUID1 etc, etc.
  3. Export results of queries (occurrences data) to Excel / JSON;

Up until now, we had the following performance limits, for about 30000 occurrences:

  1. Read / update -> 5-20 s, depending on the size of the occurrence;
  2. querying -> very much depending on the size of the query (number of fields queried), but still fast enough 1-20 s.

The performance of Read / Write and also the time needed to extract data for export is very bad already, so we decided to re-engineer the database design.

Solutions we are thinking to adopt:

  1. Store all results into a single table InputResult, with a column for Value of type nvarchar(max), so we cannot add an index on Value. Made some tests with over 1M records. Read / write would be fast. Querying is disastrous. More than 5 min to query by a single field.
  2. Store all results into a single table InputResult, with columns for all data types - ValueAsString, ValueAsInt, ValueAsNumber, ValueAsDate, etc. We can index each column appropriately, and query them individualy. Downsides- will have lots of nulls on each line. The table would grow pretty big.

I'm thinking the second solution combines the best of both ideas. What do you think? I am wondering if someone has encountered this kind of problem and has some experience to share.

So, what design would be better for this kind of problem?

EDIT: Redesigned the table and posted new question here: How to improve execution time for queries by multiple fields of different data types on a single table?

1

So, what design would be better for this kind of problem?

Why not have a relational design where each attribute has its own column with a useful name, and an appropriate data type? If you want to optimize the storage of a large number of columns, that's a storage concern, not a design concern. See, eg Sparse Columns and Row Compression.

Or consider storing the details in a JSON or XML document.

  • You're suggesting having a table with 1000 columns, one column per attribute? – MFA Sep 25 '18 at 7:16
  • Yes. I'm suggesting having a table with 1000 columns, one column per attribute. – David Browne - Microsoft Sep 25 '18 at 11:20
  • Thanks for your answer, but I don't think that solution is appropriate in my case. I've redesigned the table and posted a new question here: dba.stackexchange.com/questions/218720/… – MFA Sep 27 '18 at 11:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.