2

In the future, we will have millions ofs record in our web product. So we had used BigQuery for data storage and analysis. We have to build filters on our search page like Flipkart and Amazon provide product filters.

Basically, we need different filters on left side of our search page. With help of these filters, we can see our desired results. Every filters have their count, i.e., record counts in that category/term. So we have to build queries in such way that they give result and count of every category(term). Here, "category" means different types filter on different columns as we see on e-commerce sites, e.g., Flipkart, Amazon etc.

Faceted search (also called faceted navigation, guided navigation, or parametric search) breaks up search results into multiple categories, typically showing counts for each, and allows the user to "drill down" or further restrict their search results based on those facets.

Is there any framework/plugin available like Solr which can be used with BigQuery to provide above desired functionality?

Is BigQuery not suitable for above purpose? Do we need to stick with any RDBMS database (e.g., PostgreSQL, MySQL, etc.) and search engine (e.g., Solr, Elasticsearch, etc.) for this purpose?

4

Is BigQuery not suitable for above purpose? Do we need to stick with any RDBMS database (e.g., PostgreSQL, MySQL, etc.) and search engine (e.g., Solr, Elasticsearch, etc.) for this purpose?

Correct, BigQuery does not provide search or faceting capabilities. BigQuery is a data warehouse, so it's good for running analytics over large amounts of data, but it does not provide indexing, search, or facets.

You may be able to use Google Cloud Datastore (document database) for this, since it provides secondary indexes, or you may want to do as you suggested, and use a combination of a SQL database such as MySQL or PostgreSQL together with a search product such as Elasticsearch or Solr.

1

BigQuery support a collection of RegEx and String query functions, making it suitable for text search queries across STRING fields. However, there is a 64k per row (and field) limit for each BigQuery record, so it may not possible to support a totally unstructured, unlimited size, document text search case.

https://developers.google.com/bigquery/docs/query-reference#stringfunctions

https://developers.google.com/bigquery/docs/query-reference#regularexpressionfunctions

For a full text search capabilities I would suggest looking at the Search API:

https://cloud.google.com/appengine/docs/standard/python/search/

If you want to choose the option of using a RDBMS database that supports full text search, you will have to connect it to BiqQuery with some external tool like Alooma.

  • Thanks, I have ended with Mongodb (mongoid) + Elasticsearch (searchkick) in my Rails application. It is working great at my end. – Lalit Kumar Maurya Aug 18 '17 at 6:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.