I need to run BigQuery on top of Bigtable live, not as an export. I have found the information stating it was in beta but only as an export function. I would like to run BigQuery against Bigtable data without exporting. Has anyone done this or is it possible? The importance is that running it realtime no data is lost by doing a point in time export.
Yes you can. Link to Bigtable using it as an external table. It is not then created in BigQuery itself. Per the documentation:
To query Cloud Bigtable data using a permanent external table, you:
- Create a table definition file (for the CLI or API)
- Create a table in BigQuery linked to the external data source
- Query the data using the permanent table
It is in Beta, but this is standard for Google and usually only lasts 6 months or so. In my experience the features and products and alot more stable in Beta than you find on other providers.