How to store short stories for access to individual sentences?

Question

I am designing a database for the first time ever (using PostgreSQL), and am wondering about the most efficient/logical way of storing a body of text (aka, a story).

The conflict stems from the fact that the user will access text bodies in two ways:

1. access the entire body of the story on click of the story name.

2. the user can input a word or phrase into a search bar, which will return all sentences (not the whole stories) in which the word/phrase is found (meaning that it could potentially return many sentences from many stories).

There will be a great ("infinite") number of stories, and about 40 sentences per story, although it is free text so some stories will contain a few hundred sentences.

My initial DB design was to have a Story model (I'm using Ruby on Rails) with a story_id, story_title, and author_id_fk, and then to have a Storyline model with storyline_id, storyline, and story_id_fk.

However, I'm now doubting myself and think that maybe the best way to do it is to slap the body of the story onto a 4th column in the Story model called story_text, where I will store an array of strings (aka, the original text parsed into its corresponding sentences), and then the Storyline model can either simply not exist (in which case the appropriate item from the array would be called when needed - less normal, but also perhaps more efficient..?), or to keep the Storyline model, but have it contain a reference to the appropriate storyline as opposed to the actual text itself.

Any thoughts or suggestions would be much appreciated!

Answer 1

The best solution heavily depends on the definition of "best". You may be looking for simplicity, reliability, performance or disk space optimization or other things.

Specifications

infinite amount of stories, and about ~40 sentences per story

Few sentences, possibly many stories.

user can input a word or phrase

We need phrase search. (Maybe across sentence boundaries?)

Also assuming:

• "Sentences" are substrings simply separated by dot and space('. ' ).
• Resources (RAM, disk space) may be a limiting factor.
• Retrieving the whole body and searching for individual sentences are equally frequent tasks.

Solution

Postgres 9.6 introduced exactly what you need. The release notes:

Full-text search can now search for phrases (multiple adjacent words)

The latest Postgres release and a text search index will go a long way for performance in either case.

You already considered to:

slap the body of the story onto a 4th column in the Story model

But an array adds 24 bytes + 1 or more bytes per sentence and makes indexing much more complicated. (As well as pretty much all other operations.) I would exclude the array solution.

• Indexing an array for full text search

An additional storyline table with one row per sentence makes indexing simple again, but adds ~ 40 bytes per sentence. Finding and retrieving sentences is simple. But all write operations are more complicated.

About storage size in Postgres:

• Configuring PostgreSQL for read performance

Consider a single table with body as plain text column. Splitting the string into sentences on the fly is fast (with a simple definition of "sentence"). Smaller total storage size may be more important for performance with big tables (fewer pages to fetch). And we only need to index one column. In short:

• dead simple storage and handling
• minimum disk space
• still a hot contender for best performance

Table:

CREATE TABLE story (
  story_id  serial PRIMARY KEY
, author_id int REFERENCES author  -- NOT NULL?
, title text                       -- NOT NULL?
, body  text                       -- NOT NULL?
);

FTS index, assuming English language:

CREATE INDEX story_body_english_fts_idx ON story USING GIN (to_tsvector('english', body));

Query to return whole story:

SELECT *
FROM   story
WHERE  to_tsvector('english', body) @@ phraseto_tsquery('english', 'Lewis Carroll');

Query to return sentences:

SELECT story_id, sentence_nr, sentence
FROM   story, unnest(string_to_array(body, '. ')) WITH ORDINALITY x(sentence, sentence_nr)
WHERE  to_tsvector('english', body)     @@ phraseto_tsquery('english', 'Lewis Carroll')
AND    to_tsvector('english', sentence) @@ phraseto_tsquery('english', 'Lewis Carroll');

The query demonstrates a phrase example. But it works for single words as well.

The repeated WHERE condition is logically redundant but necessary to use the index on body. This retrieves only stories with qualifying sentences, then the recheck on unnested sentences (repeating the same condition) is cheap.

About unnest(string_to_array(body, '. ')) WITH ORDINALITY:

• How to preserve the original order of elements in an unnested array?
• What is the difference between LATERAL and a subquery in PostgreSQL?

Note: The check on the whole text body can find phrases across sentence boundaries - unlike searching individual sentences. But the recheck on sentence excludes such matches. You can fine-tune one or the other way ...

There are many more options with FTS: prefix matching, other languages, no language (no stemming or thesaurus), phrases with words in between ...

• Get partial match from GIN indexed TSVECTOR column

Answer 2

I think you'll get the maximum flexibility if you model your data with two tables. Whenever you need the whole story, you can just join them and aggregate the story lines using string_agg. I'm afraid I cannot help with the specifics of performing it within Rails (and its ActiveRecord model), but I think that find_by_sql will allow you to do it by writing the SQL sentence. If you prefer, it can also be done within ActiveRecord, although performance might be a bit worse. However, having the database do the aggregation is normally much more efficient.

When you just need to find specific sentences, your query will be much easier (and most probably more efficient), specially because you'll retrieve much less information.

Answer 3

I think the answer depends upon how you intend this application to be used.

If you intended it to be a collaborative editing tool. Or there was some significant advantage in searching for sentences then storing individual sentences would be beneficial. However that would add complications elsewhere. I would imagine the formatting and presentation of paragraphs would be difficult to get right. And the arrangement of sentences in a paragraph or chapters influence how a book is read and interpreted.

Alternatively you could store the entire text together along with its formatting. If a user searched for a keyword, you could substring the text and just display from the preceding and following full stops. In effect just filtering sentences that contain hits.