0

I'm exploring sql polybase using Azure blob storage and other data sources.

For blob storage sql server brings the entire blob onto the server and then performs where/join.

What is the behaviour when using another database like mysql instead of Azure blob storage? Does polybase bring entire mysql table data into the sql server for each query?

1 Answer 1

0

It depends. My rough understanding of Polybase is it's a little more built out and granular Linked Server equivalent that is object specific. Instead of setting up a connection to an entire server, it virtualizes that connection to specific Tables on the remote server. I believe, similarly to a Linked Server, it tries to execute that query remotely when possible but there are use cases when it may need to remotely scan all of the data then apply the actual query logic locally.

More information can be found in the books online for Introducing data virtualization with PolyBase:

A key use case for data virtualization with the PolyBase feature is to allow the data to stay in its original location and format. You can virtualize the external data through the SQL Server instance, so that it can be queried in place like any other table in SQL Server. This process minimizes the need for ETL processes for data movement. This data virtualization scenario is possible with the use of PolyBase connectors.

You query external data by using the same T-SQL syntax used to query a database table.

An example the docs discuss regarding execution on the remote server in a scenario where Hadoop is the remote server:

PolyBase pushes some computations to the external source to optimize the overall query. The query optimizer makes a cost-based decision to push computation to Hadoop, if that will improve query performance. The query optimizer uses statistics on external tables to make the cost-based decision. Pushing computation creates MapReduce jobs and leverages Hadoop's distributed computational resources.

So it's going to depend on the database system of the external source but also like Linked Servers, likely depends on the type of query you're executing too.

2
  • Linked server is able to pass criteria to the linked server or brings the whole table across?
    – variable
    Sep 23 at 11:52
  • @variable It can do both, just like Polybase, it depends on the query. That's why I compared Polybase to it.
    – J.D.
    Sep 23 at 11:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.