I'm using polybase to import a parquet file.
Over time, it is likely we may add or remove named columns in the file.
When I add an additional column, I get the below error:
External file access failed due to internal error: 'File test.parquet: HdfsBridge::CreateRecordReader - Unexpected error encountered creating the record reader: HadoopExecutionException: Column count mismatch. Source file has 16 columns, external table definition has 15 columns.'
This is because I added an additional column that wasn't in the external table definition.
As parquet contains a file schema and the external table knows the name of each column, is there a way it can be set to ignore the extra unused column?