1

I'm using polybase to import a parquet file.

Over time, it is likely we may add or remove named columns in the file.

When I add an additional column, I get the below error:

External file access failed due to internal error: 'File test.parquet: HdfsBridge::CreateRecordReader - Unexpected error encountered creating the record reader: HadoopExecutionException: Column count mismatch. Source file has 16 columns, external table definition has 15 columns.'

This is because I added an additional column that wasn't in the external table definition.

As parquet contains a file schema and the external table knows the name of each column, is there a way it can be set to ignore the extra unused column?

1

Polybase does not have this ability. The source file and external table definition must match for it to work. Simply add or remove additional column(s) in your external table definition. Then when you are physically creating your table in the database eg via CTAS, add/remove the additional column(s) from your CTAS statement.

Alternately, consider the new COPY INTO statement. This does have the ability to specify a column list, eg

COPY INTO test_parquet ( col1, col2, col3, col4 )
FROM 'https://myaccount.blob.core.windows.net/myblobcontainer/folder1/*.parquet'
WITH (
    FILE_FORMAT = myFileFormat
    CREDENTIAL = ( IDENTITY = 'Shared Access Signature', SECRET = '<Your_SAS_Token>' )
)

However please note this feature is i) currently in preview, ii) for copying into physical tables (ie not external tables) and iii) I haven't tested the column list specifically with Parquet.

2
  • This is really helpful! Do you know if copy into can use a named source/credential? – Neil P Mar 24 '20 at 16:57
  • Yes, the documentation shows two paths in the CREDENTIAL section for Parquet: 1) if you are importing from Azure Blob storage you can use SAS or Azure account key, or 2) SAS / MSI / Service Principal / Key or Azure Active Directory (AAD). – wBob Mar 24 '20 at 18:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.