Data Consistency
Data consistency with pagination
Introduction
When extracting data with pagination techniques while ongoing updates are occurring, it is important to realise that we can only achieve eventual consistency.
Common issues
Here are some use cases where applying pagination can cause issues:
This shows the case when a sort order in a non-deterministic field (like lastUpdated) is forced to combine with pagination. If, during or between consecutive page reads, data is being updated in a manner that affects the order of the returned data, some items might be skipped.
This shows the case when changes are made to pages that are already read. An update might be missing on ED 9 due to this timing issue.
When applying pagination techniques, it might not be possible to avoid the second scenario, but there are techniques to compensate for this. It can be done by combining techniques of pagination and incremental load.
It is important to understand that the same data can be supplied, and the destination data source might already have this data. That is why it is recommended to use upsert operations, based on the unique key information. In the case of GeneralLedgerTransaction, this would be companyId,transactionNumber,additionalInformation/sequenceNumber.
Example for compensating eventual changes during the data loading process, in the case that eventual consistency is not good enough, and at the end of loading, we should have achieved full consistency.
This scenario is applied for a full load example, which immediately continues with incremental load cycles, until an incremental load cycle no longer provides new data.