[Advanced] Selective Hashing and SoftHSM How To

Overview

This article describes how to setup Selective Hashing on the Senzing API. Keep in mind this is an advanced topic and requires manually making changes to the g2config.json configuration file. If you require Selective Hashing and are not comfortable making JSON edits, please submit a feature request for a tool to support this to [e-mail].

This article is focused on using G2Loader.py and setting it up to self-anonymize but the pattern is the same for using the APIs directly.

Prerequisites

• softhsm
  • Red Hat / CentOS
    • sudo yum install softhsm
    • sudo usermod -aG ods <your_userid>
    • Logout and login
  • Debian
    • sudo apt install softhsm2
    • sudo usermod -aG softhsm <your_userid>
    • sudo mkdir -p /var/lib/softhsm/tokens/
    • sudo chown :softhsm /var/lib/softhsm/tokens/
    • sudo chmod 770 /var/lib/softhsm/tokens/
    • Logout and login
    • export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/x86_64-linux-gnu/softhsm/
      
      • You can add this to ~/.bachrc (or similar) or modify the project setupEnv 

Generate secure store and SALT

1. Edit your G2Module.ini and add:

[PIPELINE]
...
SECURE_STORE_URL=pkcs11://token/?slotID=0
SECURE_STORE_LIB=softhsm2
SECURE_STORE_PIN=
ENABLE_SECURE_STORE=TRUE

2. Initialize the secure store by running these commands from the project directory, specify a value with more than 4 characters for the SO PIN:

• ./bin/g2ssadm -tokinit -label G2_STORE -c ./etc/G2Module.ini
• ./bin/g2ssadm -ssinit -c ./etc/G2Module.ini

3. Create the SALT by running this command from the python directory:

• ./bin/g2saltadm -new -name G2SALT -c ./etc/G2Module.ini

4. The g2saltadm command will return 4 lines that need to be added to the ./etc/G2Module.ini:

[SALT]
NAME=G2SALT
CHECKSUM=*******************
ANONYMIZATION=SHA2

Editing the configuration to enable feature hashing

Open the g2config.json in the python directory and search for CFG_FTYPE

...
"CFG_FTYPE": [
{
 "FTYPE_STAB": 0,
 "FCLASS_ID": 1,
 "FTYPE_FREQ": "NAME",
 "RTYPE_ID": 0,
 "ANONYMIZE": 1,
 "DERIVED": 0,
 "USED_FOR_CAND": 0,
 "PERSIST_HISTORY": 1,
 "VERSION": 1,
 "FTYPE_ID": 1,
 "FTYPE_CODE": "NAME",
 "FTYPE_EXCL": 0
 },
...

The important things to note are the ANONYMIZE and FTYPE_CODE fields. The FTYPE_CODE tells you which feature is being referenced and the ANONYMIZE flag tells you whether or not to hash the field.

NOTE: Only change the fields you must hash to ANONYMIZE as it will impact the quality of scoring. It is highly recommend not to ANONYMIZE NAME or ADDRESS as that will most adversely impact match quality.

 

Run it

Once the above steps are complete, you can run G2Loader.py as usual.

NOTE: Keep in mind you can't change a loaded feature to/from being ANONYMIZED without reloading those records. Senzing can't 'unhash' a feature and does not crawl through loaded data to hash it automatically.