Correct. An interesting part of the product’s history is that it “originally” had to mostly solve for what Siebel calls “remote” users. Before version 7 (released in 2000 or 2001) the primary usage of Siebel was for mobile field sales users who worked disconnected on their laptops on a local database (Sybase SQL Anywhere). Siebel had a neat little “store and forward” replication scheme and had to solve concurrency problems in this kind of architecture.

It’s “connected client” architecture was 2-tier client server in which the SQL statements came directly from the client application on the user’s computer. Of course, the modification_num solution works for this architecture as well, but it really wasn’t an option for the product to use a server database feature.

The mobile client architecture is still in use today - but Siebel’s application server architecture (as well as the ubiquity of broadband connections compared to end of the 20th century) makes it much less common in my experience.

There are some decisions that appear to have been made in the pursuit of “database agnosticism” (although none terribly limiting in practice), but this one I think they got pretty much right.

Perhaps. I’m not knowledgable enough about the SCN and when it increments to confirm. But here’s the scoop on MODIFICATION_NUM and why it is really there.

Each update statement is of the form:
UPDATE table set
field1 = :bv1,
field2 = :bv2,
…
modification_num = :bvN
where row_id = :bvX
and modification_num = :bvY

Why do such a thing? Concurency control.

Siebel creates a data object in its application server that it calls a “Business Component” (or BC for short). It creates a BC from a select query against the database. The user can then work with the data in the BC - and even update it. The updates are staged in the BC object on the application server - and only written back to the database on certain run time events (WriteRecord I believe - perhaps a few others).

Well, Siebel doesn’t take out a FOR UPDATE lock on the database. What it does do is store the MODIFICATION_NUM and ROW_ID for each record in each table in the BC. When the update statement is executed, Siebel sets the MODIFICATION_NUM to CURR_MOD_NUM + 1 (bvN in my example above). It also includes CURR_MOD_NUM in the update statement (bvY in my example).

So - what happens if another BC object has updated that record on that table in the meantime? As you can see, the update statement will not update any rows, since the filter in the where clause “fails”. Siebel traps this error, rolls back the user’s change and complains.

Note that this can happen between sessions (which is the more logical case) or within a user session (which is the more common one). It is possible that a user session creates two different BC objects that contain the same record in the database - and then try to update both of them in turn. This is almost always a “bug”, but one that happens from time to time.

So - as you can see, MODIFICATION_NUM is not really intended to be a counter of the number of times a record has been updated in the server database. But you can also see that it effectively is.

MODIFICATION_NUM sounds to me like SCN in the Siebel world - is it so?

Yes, I think you might well be right on point 1. That certainly makes it more simple, though some of the basic challenges remain.

2 and 3 are very interesting also. I wish I could get that kind of insight from our own Siebel team :D

Great blog. I tend to be more of a lurker on this and other blogs for a while. I figured I’d offer you some thoughts on this subject though.

I see you have a solution - but one that sounds like you may revisit. A few thoughts:

1. It sounds like this is Siebel, so I assume that your update_dt field is really LAST_UPD. Are you certain that this field is nullable in your implementation. Every version I’ve ever inspected this on (including the 8.0 I just tested) populates both CREATED and LAST_UPD on the initial insert of the record.
2. Again, assuming this is Siebel, I’d point you to MODIFICATION_NUM being compared to an image table (as someone else suggested) as a reliable source of an answer to “What records have been updated since I last ran?” The date fields are just dates - and have issues others have commented on. The MODIFICATION_NUM field is updated on each insert or update statement that goes through Siebel processes.
3. Please verify that your filter is really based on updates to the table you are working with - and not on one or more of what Siebel calls a “Business Component.” For example, if you want to extract “all orders that have been updated”, your solution is not likely to drive strictly off of the S_ORDER table, but also off of the S_ORDER_ITEM table (and possibly the variety of attribute and extended attribute tables for each.) Similarly, Siebel models the “primary address” as a part of the Account - but the S_ORG_EXT table won’t be updated just becuase someone updates the zipcode on the primary address.

I think there’s a flaw in your method though, as it relies on two kep points:

i) A max_id that is growing sequentially.
ii) Having an index on the max_id.

Which is rather the same as the problem of having an index on the update_dt itself, I feel, with the same remediations being required.

Did you implement a solution already?

I’m a bit late with my proposal. But anyway - my idea comes down to: are there any other ways apart from create_dt to recognize the newly inserted records? Using create_dt for this seems to me too much work because we are normally interested only in the new records (hence we are using a very limited infomation of the column and the eventual index).

For example, if the table has a surrogate increasing ID column (the new reacords get the highest ID), you might take advantage of it. What about a small log table where the latest “captured” ID ls logged?

Theen the logic would look this way (you will still need an index on update_dt):

log the current MAX_ID;

select …
from …
where (ID > MAX_ID_PREVIOUS_RUN and ID <= MAX_ID)
or (update_dt >= trunc(sysdate)-1 and update_dt < trunc(sysdate))