Yes it’s redundant, but it’s a psychological thing - I somehow can’t stop doing it. Like a nervous twitch.

Joel - “for what time frame?”

Well, it was a data migration for a telecom billing system - there were no errors encountered during the ETL process run itself. There were some bugs in the data populated on the target system - but by and large the rating and billing ran correctly. Given the development timeframe, it was a pretty decent job done. And I think it was good because I used SQL - seeing the logic in one page of SQL is valuable - it is very easy to lose sight of the picture if the logic is spread over some ‘mapping tool’. Or maybe I am just plain too comfortable with SQL.

It is pretty amazing how far off users can be between reality and perceived requirements.

I started to write an answer, but it turned into something rather longer so I’ll start writing up something as a separate posting on ETL error handling.

I have asked it before and will ask again - how does your “set oriented SQL” powered stand up to error conditions? What method are you adopting for this?

I used SQL some years back for a data migration project (this was when I was left alone by the “competency centre” and did not have to follow their “methodology”). The code was small - every piece of logic was within a screenful - it was fast - it worked (meaning very few bugs). I just loved it (well, I did not really think about error handling - but then, no error showed up! ;)

And then the methodology guys came along and people do something else that I don’t want to talk about. It takes longer - leads to more bugs and yet they do it… so I just moved out of the ETL part of it all.