Using Azure Synapse Analytics, I have configured a pipeline and a Mapping Data Flow that were working great until I recently added a Derived Column to my Mapping Data Flow, which created a column that was not originally present when I created the table in the Database Designer. The column just adds a timestamp using currentTimestamp() to help me track when rows have been updated.

I assumed that the Mapping Data Flow would create a new column, as Schema Drift is enabled. It didn't. So I created the new column in the Database Designer, but that didn't help either, I receive the same error, as in the post title:

cannot resolve target.new_column in UPDATE clause given columns [Column List].

It then repeats the same error, and provides this which I have included in hopes that it would be helpful.

\n\tat org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)\n\tat org.apache.spark.sql.catalyst.plans.logical.DeltaMergeInto$.$anonfun$resolveReferences$3(deltaMerge.scala:303)\n\tat scala.collection.immutable.List.foreach(List.scala:392)\n\tat org.apache.spark.sql.catalyst.plans.logical.DeltaMergeInto$.resolveExprsOrFail$1(deltaMerge.scala:299)\n\tat org.apache.spark.sql.catalyst.plans.logical.DeltaMergeInto$.resolveOrFail$1(deltaMerge.scala:312)\n\tat org.apache.spark.sql.catalyst.plans.l","failureType":"UserError","target":"lhApplicationUsers to Delta DF","errorCode":"DFExecutorUserError"}

Some hopefully helpful facts:

  • Sink is a Delta table in an ADLS2 container
  • Source is a different ADLS2 container with CSV
  • Confirmed that Database Designer column format is correct

What I have determined is that, when I import the schema from the connection, the new column is not present. However, in the database view and in a SELECT statement, the new column does appear and it is null (as expected since the Mapping Data Flow has not run).

I have tried recreating the table in the Database Designer, I have tried deleting and re-adding the Sink in mapping data flow. I have tried changing the format to Date and using currentDate() (to eliminate column formatting errors). What I have not done is deleted the underlying data, as I really don't want to do that unless I absolutely must.

Any suggestions on how to address this problem?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.