I have one partitioned table with more than 2500 partitioned tables. So when I try to update schema for such table I get following error, Too many partitions produced by query, allowed 2000, query produces at least XXXX partitions.

So I have divided this copy operation into few chunks like,

INSERT INTO {dest_table} (columns) SELECT (columns) from {src_table} where {partitioned_column} > {timestamp or date} AND {partitioned_column} < {timestamp or date}.

So copy operation is divided into let's say 4 iteration.

But i can not use JobConfigurationQuery with WriteDisposition = "WRITE_TRUNCATE" OR "WRITE_APPEND", because the newer table will have updated schema.

Is there anyway I can create these query jobs atomically?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.