I’m working on an application similar to the Google Ngram viewer, tracking the occurrence of terms in articles from different news publications over time.
In the past I’ve worked primarily on front-end development, so I have very little experience with storing and retrieving large amounts of data. I am seeking advice on what schema and server back-end would be most stable, efficient, and cost-effective. Currently our data exists as a collection of nested JSONs structured like this:
one file for each publication (e.g. The Washington Post)
keys for monthly date intervals from 1975 to 2000
keys for all ngrams of n=1, n=2, n=3, n=4, n=5 within that month
keys for each ngram with that n number (e.g.
"it was"as a key within n=2)the integer count of occurrences for that ngram (within that month, for that publication)
The application should allow you to search this data and generate a graph tracking the occurrence of specific terms from 1975-2000. Whatever back-end we use needs to be able to return searches filtered by publication, as well as counts summed from multiple publications (there are also certain groupings of publications, e.g. “west coast publications”, which we would like to optimize for).
As I said, I don’t really know what I’m doing — please let me know if there is anything I can explain better or more information/tags I can provide.
