Processing REDO

Overview

There are times the Senzing engine determines additional work needs to be performed on an entity. In some cases it will decide automatically that this work should be done at a different time, for instance:

• Cleaning up decisions made based on attributes that are determined to no longer be important (most common)
• Records being loaded in parallel around the same cluster of entities causing conflicts
• Automatic corrections

When this happens a special record is written to the SYS_EVAL_QUEUE table for future processing. These entries and known as REDOs. In the case of the G2Loader.py python script, the script automatically processes these.

Details

The table SYS_EVAL_QUEUE is in the core Senzing repository and is comprised of:

• LENS_CODE - A key for an advanced feature which is not in use at the moment
• ETYPE_CODE - Entity type code, typically something like PERSON or COMPANY
• DSRC_CODE - The user provided data source identifying code
• ENT_SRC_KEY - A key, often internally generated, to identify a specific record
• MSG - The internally formatted message to be processed by the engine

The first 4 field make up the unique key to identify the record.

Processing

Logically the data is processed as follows:

1. Select records from the SYS_EVAL_QUEUE table
   • select LENS_CODE, ETYPE_CODE, DSRC_CODE, ENT_SRC_KEY, MSG from SYS_EVAL_QUEUE
   • It is best to do this in blocks of records (e.g. 100) in case the table has a lot of records in it
   • It is recommended to do this periodically during process (e.g. every 1000 records) and to limit to a maximum number of blocks processed at a time (e.g. max 10 blocks) to balance new data and redo records
     
     
2. For each row selected
   • Call G2Engine.process(redo entry)
   • Handle errors as you would with G2Engine.addRecord(...) or other such functions
   • On successful processing, delete the record
     • delete from SYS_EVAL_QUEUE where LENS_CODE = ? and ETYPE_CODE = ? and DSRC_CODE = ? and ENT_SRC_KEY = ?

Example

The G2Loader.py script that ships with Senzing API implements redo processing in the processRedo(...) function.