6

Polybase is powerful but not really that sophisticated so will not be able to deal with this weird format. You have three options as I see it: Correct the file format at source. Instead of having weird mix of fixed-width and delimited file format, use a standard file format, such as .csv. This format you have where the column delimiter is either carriage ...


4

I'll work from the assumption that all of the tables listed in the query above are external tables pointing to the same external data source. With that in mind, here are a few considerations. Distributed Requests There are two DMVs which will provide a bit more information than what you have currently: sys.dm_exec_distributed_requests and sys....


4

PolyBase needs TCP enabled to work. By default TCP is disabled in SQL Network Configuration.


3

According to a hex editor, your sample file has single line feed (0A) for some line endings and two line feeds as the delimiter between rows: A U-SQL custom extractor might be able to handle this file, but I wonder if we will run into issues with the full 30GB file. Instructions Set up an Azure Data Lake Analytics (ADLA) account if you don't already have ...


2

Creation and maintenance of PolyBase statistics is in the hands of the DBA. The "getting started" guide provides several steps to establish PolyBase, the last of which is CREATE STATISTICS referencing the external table. The documentation on CREATE STATISTICS states: Statistics for external tables When creating external table statistics, SQL ...


2

It turned out to be a permissions issue with the DWConfiguration/ DWDiagnostics and DWQueue databases that I just reattached after the reinstall of the OS, for some reason the [NT AUTHORITY\NETWORK SERVICE] user did not have the correct permissions on these databases and thus the service kept on failing. I managed to debug this with an extended events ...


2

I ended up using sed to cleanse the file zcat myfile.txt.gz | sed -r 's/[ ]+/vin/g'|tr -d '\n'|tr 'vinvin' '\n'|grep -v '^$' > myfile.txt This took care of formatting issues with the source files. Once these files were uploaded to Azure blob storage, the rest was easy. I created external table pointing to files on blob via Polybase and then created ...


1

You can check the SQL Server Error Log to get more information as to why dump files were generated. Microsoft also provides the sys.dm_server_memory_dumps dmv for troubleshooting, but it isn't clear to me how that dmv would be helpful.


1

Polybase does not have this ability. The source file and external table definition must match for it to work. Simply add or remove additional column(s) in your external table definition. Then when you are physically creating your table in the database eg via CTAS, add/remove the additional column(s) from your CTAS statement. Alternately, consider the new ...


1

SQL 2017 is not good with Polybase. Currently, I am using SQL 2019 for this and all is good.


1

MSDN has updated the documentation Option 6: Cloudera 5.1, 5.2, 5.3, 5.4, 5.5, and 5.9 on Linux https://msdn.microsoft.com/en-us/library/mt143174.aspx


Only top voted, non community-wiki answers of a minimum length are eligible