Using Azure Synapse Analytics, I have configured a pipeline and a Mapping Data Flow that were working great until I recently added a Derived Column to my Mapping Data Flow, which created a column that was not originally present when I created the table in the Database Designer. The column just adds a timestamp using
currentTimestamp() to help me track when rows have been updated.
I assumed that the Mapping Data Flow would create a new column, as
Schema Drift is enabled. It didn't. So I created the new column in the Database Designer, but that didn't help either, I receive the same error, as in the post title:
target.new_columnin UPDATE clause given columns [Column List].
It then repeats the same error, and provides this which I have included in hopes that it would be helpful.
\n\tat org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)\n\tat org.apache.spark.sql.catalyst.plans.logical.DeltaMergeInto$.$anonfun$resolveReferences$3(deltaMerge.scala:303)\n\tat scala.collection.immutable.List.foreach(List.scala:392)\n\tat org.apache.spark.sql.catalyst.plans.logical.DeltaMergeInto$.resolveExprsOrFail$1(deltaMerge.scala:299)\n\tat org.apache.spark.sql.catalyst.plans.logical.DeltaMergeInto$.resolveOrFail$1(deltaMerge.scala:312)\n\tat org.apache.spark.sql.catalyst.plans.l","failureType":"UserError","target":"lhApplicationUsers to Delta DF","errorCode":"DFExecutorUserError"}
Some hopefully helpful facts:
- Sink is a Delta table in an ADLS2 container
- Source is a different ADLS2 container with CSV
- Confirmed that Database Designer column format is correct
What I have determined is that, when I import the schema from the connection, the new column is not present. However, in the database view and in a SELECT statement, the new column does appear and it is null (as expected since the Mapping Data Flow has not run).
I have tried recreating the table in the Database Designer, I have tried deleting and re-adding the Sink in mapping data flow. I have tried changing the format to Date and using
currentDate() (to eliminate column formatting errors). What I have not done is deleted the underlying data, as I really don't want to do that unless I absolutely must.
Any suggestions on how to address this problem?