I have an SSIS process that is built to push records to a web service that has some sporadic auth timeouts. My design is working well but we are running large batches of records 250,000 per batch over Millions of rows. What I'm looking to do is something similar to this post https://stackoverflow.com/questions/7076018/how-to-fail-ssis-component-on-invalid-rows but instead of failing the data flow task I want to to just complete successfully without processing more rows so it iterates on to my next loop iteration. Essentially once I get an error I know the rest for that batch will fail and I want to iterate my loop which I already have logic built to re-run failed records. Any thoughts on how to accomplish?
It is possible to process individual rows and redirect the records that failed. You could even validate records before you try to insert them.
How do you want it to work? Unless they specifically need to be processed together as a batch I would be processing individual records and storing failed records for correction and resubmission.