Experian Match

Each match between two records can have one of four match levels. The default matching rules define the match levels as follows:

In order for Experian Match to start work on your data, you need to configure your data sources. Experian Match will need to know how to connect to your data source (including authentication information) and which data from the source to match on.

Experian Match creates blocks of similar records to assist with the generation of suitable candidate record pairs for scoring. Blocks are created from records that have the same blocking key values. Blocking keys are created for each input record from combinations of the record’s elements that have been keyed. Keying is the process of encoding individual elements to the same representation so that they can be matched despite minor differences in spelling.

A default set of best practice blocking keys are provided with the software for name and address data. These can be acquired by a call to the REST API. To use these the user must modify them to match their input data and requirements then submit them via the REST API.

Before setting up your match session, a rule set must be configured. A rule set is a set of logical expressions (rules), written in our own Domain Specific Language (DSL), which control how records are compared and how match levels are decided. We have designed the rule DSL to give you complete control over how records match.

A default best practice rule set is provided with the software for name and address data. This can be adjusted for optimal matching depending on your data and requirements.

A cluster is a collection of records that have been identified as representing the same entity using the rules that you have provided.

Prior to installing or starting the service, data should be installed.

Experian.Gdq.Standardize.Standalone.exe can be executed simply by running the executable. However, it is preferable to install it as a Windows Service.

The following parameters within the configuration file Experian.Gdq.Standardize.Standalone.config can be modified:

It is possible to override the file containing the alias to rootname mappings using the property "standardisation.rootname.file.path".

The Matching product is configured to use GdqStandardizeServer on the default port 127.0.0.1:5000. To change this, edit the application.properties file in the deployment directory, and add two properties: standardisation.host, and standardisation.port set to the required value.

When standardising records, GdqStandardize needs to know what country the data in the record is referring to, in order to derive more information. Proper, country-specific standardisation affects the rest of the Matching process, as it changes how potential matches are found.

The defaultCountry configuration setting influences which country the standardisation system will assume the record is from. However, this influence can be overridden on a per-record basis by specifying an ISO 3166-1 alpha-3 code in the input data. The data should be mapped to the COUNTRY data type.

Common ISO codes:

The Experian Match .war file contains support for HSQL database match stores. HSQL is a simple, disk backed, in-memory relational database management system.

Experian Match will generate a HSQL database on a local file path when configured as a JDBC connection as follows:

A relative database path such as match\\hsql.db will store the database relative to your web server. Alternatively an absolute path such as c:\\temp\\experianmatch\\hsql.db can be used.

On shutdown of Experian Match, the database will be persisted to disk at the location defined.

Experian Match utilises HikariCP connection pooling which can be configured external to Experian Match through the addition of a properties file named hikari.properties in the web applications resource directory. Experian Match is configured to use the default settings and additional tuning may be necessary based on the JDBC data source used. See Hikari configuration settings for full details and suggestions for configuring connectionTimeout, maximumPoolSize, and dataSource.*.

In order to create your own rule set or modify the default rule set it is important to understand the concepts.

Rules take the following form: <rule reference>=<expression>

A rule reference consists of a rule name followed by a ‘.’ followed by a match level.

An expression may take multiple forms. It is either a low-level expression operating on the elements within a record, or a higher-level expression composed of references to other rules.

A rule set is made up of a combination of three rule types, the three rule types increase in specificity from Match to Theme to Element. Match and Theme rules are comprised of references to rules from the level below. Element rules are comprised of rules set on specific data elements, for example a postcode or building number.

We can visualise this using a tree diagram:

Rule sets are supplied as a JSON escaped string when making a rules configuration request. JSON has a number of reserved characters, such as line breaks that need to be escaped when supplying a string as part of a request. This involves replacing these characters with the relevant escape characters e.g. \n for a line break.

Long JSON escaped strings, such as a matching rule set, are not very human readable. We recommend that you make changes to your rules before escaping the string. There are a number of free online tools that will escape JSON for you.

The default rule sets for each supported country can be found here:

There are three rule types that can be defined within the rules: Match, Theme and Element (single element, multi element, generic string and date).

The following comparators can be used. The results available for the default comparator (ExactString) will also be available for all other comparators.

The Match REST API is able to identify and correct many known terms that may appear in the input record. A selection of element modifier keywords can be used to retrieve modified versions of the input elements within rule definition.

Example:

Download text file containing the JSON escaped default rules string for the United Kingdom.

Download text file containing the JSON escaped default rules string for Australia.

The keying of each element is defined by a BlockingKeyElementAlgorithm object. If no BlockingKeyElementAlgorithm is specified the SIMPLIFIED_STRING algorithm is used.

The CONSONANT and SOUNDEX key types support the following character sets: Basic Latin (ASCII), Latin-1 Supplement, Latin Extended-A, Latin Extended Additional.

All other key types have been tested with the following Latin character sets: Basic Latin (ASCII), Latin-1 Supplement, Latin Extended-A, Latin Extended-B, Latin Extended-C, Latin Extended-D, Latin Extended-E, Latin Extended Additional, IPA Extensions, Phonetic Extensions, Phonetic Extensions Supplement.

Blocking Keys are hashed when written to the match store. This obfuscates them for security; in cases where the match store is encrypted, the blocking key value must remain unencrypted. Hashing also improves performance of blocking; because hashing produces values of the same length, the database is able to create a more efficient index over the blocking keys.

Note: When using a relational database as a match store, hashing blocking keys will affect the type of the blocking key value column. With hashing enabled the column will be CHAR(40), when hashing is disabled it will be VARCHAR(255). When running a job with hashing enabled, it is not possible to run a job with hashing disabled against the same match store. The match store tables will have to be dropped before running a job with hashing disabled.

Hashing has no impact on Search or Transactional operations.

The log level is specified for each major component of Matching within its own section of the log4j2 configuration under the XML section '<Loggers>'. For example:

This specifies that the Scoring component would have a log level of 'WARNING' - this is the recommended default for all components. Each component can have the logging level increased or decreased to change the granularity in the log file.

The components of Experian Match that may be individually configured are:

The log levels in log4j follow the hierarchy in the table below. Therefore if you set the log level to DEBUG, you would get all the levels below DEBUG as well.

This facility is configured in the system 'application.properties' file via a collection of parameters. These are:

Typical output comprises:

For the system log file and Web page these are all reported together, for the csv based metrics each is written to its own individual file.

A reporting operation is available that provides statistics about how blocking keys were used for a matching job. An example response is given below.

The response contains two sets of blocking key statistics for each key configured, and details of the block size threshold.

blockSizeThreshold is the maximum value for which records, with the same blocking key, will be considered as candidate matching pairs. Experian Match records a Threshold Exception when the number of records in a block exceeds this value. The default value of 200 can be overidden with the maxBlockSize configuration setting when creating a blocking key.

numThresholdExceptions is the number of exceptions recorded for each blocking key type.

Each set of statistics has the number of blocks generated and the minimum, maximum, mean, median, standard deviation of block size.

statisticsExcludingExceptions are calculated with only the blocks that were included for matching as they were within the threshold.

statisticsIncludingExceptions are calculated for all the blocks that could be generated including block sizes above threshold.

For this tutorial we will be running the API under Tomcat using the default port - in this case port 8080.

For the input we will use the example CSV file provided, connect to a disk-backed HSQL database as the match store and output the results to a new CSV file.

Our input file contains mocked up name, address and email data from the United Kingdom. By performing a match and output job we will be able to see where our data matches, ultimately providing an output file demonstrating where similar records have been clustered together. This would allow us to assess, remove or combine our duplicate records.

You can see a 6 row sample below:

As you can see there’s a lot of data that looks very similar. Once our job is complete, we should expect to see many of these clustered together. Whether or not these match will be down to how strict our ruleset is.

We can make our requests using a few different methods, you can find more information in the making requests with the API section.

We have provided a Postman collection (FF-HSQL-FF) for this tutorial, you should be able to make each request in the collection sequentially to perform a match and output job. Please ensure that you have imported the Postman collections and environments correctly. The parameters provided when importing the environments will work, however the sample requests below demonstrate the explicit values.

The simplest way to explore the API and make your requests is with the Swagger interactive documentation hosted at http://localhost:8080/matching-rest-api-2.7.1/swagger-ui.html. You should be able to copy and paste the objects from the tutorial into the relevant requests, this will provide you with a better understanding of each request and the workflow.

The first part of configuring your job is to configure the data source. Our data source will be the provided CSV file - DEMO_100.csv.

For simplicity we will store our input and subsequent output files in C:\temp

To make a data source configuration request we need to make a POST request to /v2/configuration/datasource. As we are running Experian Match on Tomcat, with the webapp deployed at the ROOT, the full url for this request is http://localhost:8080/v2/configuration/datasource. When using the Postman collection we want to use the "Post connection settings for INPUT FLATFILE" request.

We need to post an object where we specify how we connect to our input, in this case our CSV file. We also need to map the input fields so that the API knows what type of data we have in our input file.

Here is what our request body will look like:

As you can see we have set our connectionType as FLATFILE and we have set the path to that of our CSV file.

We have also set our field mappings to match that of our input file. For this example we want to match on name and address and so we have mapped the relevant fields.

The response body we get from the API is:

The response confirms our connection type and field mappings as well as providing us with a data source ID. Make a note of this ID, it is important and we will need it later on.

Next we need to configure our output. As we are outputting to a flat file we will configure an output CSV.

We need to post another object to our API, this time to http://localhost:8080/v2/configuration/output. When using the Postman collection we want to use the "Post connection settings for OUTPUT FLATFILE" request.

This object needs to include connection settings for our output file as well as mapping our input fields to our desired output fields.

Our request body will look like the below:

As you can see, some of our output fields actually differ from our data source fields. Experian Match allows us to output whichever fields we want from our input file, even if they haven’t been used for matching. For example, we are outputting the EMAIL and DOB fields, this is information we want within our output file but we don’t want the API to use it for matching on this occasion, hence omitting it from our input mapping.

We have also included a Cluster_ID, Match_Status and Record_ID output field. These fields take $CLUSTER_ID, $MATCH_STATUS and $RECORD_ID as input fields respectively. Fields that start with $ are calculated or system fields, as such, both IDs and the match status are returned by the API and not derived from the original data source. We therefore set the source as 0. For all other input fields we set the source as 1 to reflect the datasourceID returned to us when configuring our data source.

The response body we get from the API is:

Once again our response confirms our output connection settings and field mappings. We also get an output ID - as with the data source ID we will use this for configuring our session.

We should expect to see these fields in our output file:

We now need to configure the rules we will be using to control the stringency of our matching job. We do this by making a post request to http://localhost:8080/v2/configuration/rule. When using the Postman collection we want to use the "Post rules" request.

We need to provide a JSON escaped string containing our rules. We have used the default ruleset for the United Kingdom, which you can find more information about in our rules section. We can also give our rules a description, this is useful if we plan to use different rulesets for subsequent jobs.

Our request object will look like the below:

The response body we will get from the API is:

Our response will return the description, rules and version we posted along with a ruleSet ID which will be used when we configure our session.

We will use the default United Kingdom blocking keys from the GET request to http://localhost:8080/v2/configuration/blockingKey/default/GBR

To use these blocking keys with our session we need to make a POST request to http://localhost:8080/v2/configuration/blockingKey including the keys. Copy the response from the GET request and POST this to create the keys. The newly generated keys will each have a blockingKeyId for use in the session configuration.

Now that we have configured our data source, output, rules and blocking keys we can configure our session.

We need to post the IDs returned to us in our previous requests as well as providing connection settings for our SQL database to be used as the match store. We make the post request to http://localhost:8080/v2/session When using the Postman collection we want to use the "Post session" request.

For the blocking keys we want to use IDs 1 to 9 which we set earlier. As we haven’t configured any other data sources, outputs or rules, our IDs should all be 1. We can also make use of the HSQL functionality provided by the API. This allows us to specify a disk-backed store.

Our request body will look like the below:

As we are making use of HSQL we have set the connection type to JDBC and our JdbcUrl to a file in our c:\\temp folder. We also need to provide the right driver. The HSQL driver is packaged with the API.

The response body we get from the API is:

Our response will return us the same JSON as we posted along with a session ID. We will use this sessionID when performing a match job.

Our final step is to run the match and output job. We need to post the sessionID to http://localhost:8080/v2/matching/job/matchAndOutput. When using the Postman collection we want to use the "Match and Output" request.

We can also give our a description and specify a callback URI, we will leave the callback blank for this tutorial.

Our request body will look like the below:

The initial response body we get from the API is:

The response shows:

To check the status of our job we need to make a GET request to http://localhost:8080/v2/matching/job/1. The 1 refers to the jobID returned to us when we first scheduled the job. You could also make a GET request to http://localhost:8080/v2/matching/job to return the status of all jobs.

We will get the following response if our job was successful:

We will now be able to navigate to our C:\temp folder and view our E2E_100_output.csv output file. If you sort your output file by Cluster_ID you will be able to see which records have been clustered together.

A 6 row sample of the output is shown below:

Experian Match has clustered a number of our records together, clearly showing that, as we suspected, we had duplicate data in our data source.

Searches can be performed against a built Match Store with very little configuration. Using the example on the Overview page, a search can be made as such:

The session ID is passed as a parameter in the path. The contents of search is a schema-less JSON object. The keys correspond to the input mappings specified in the Data Source field mappings. Here, Address3 was not specified, and so it can be omitted, and will be treated as blank when Searching.

The response object will look like this:

Without any additional configuration, the response will be a list of objects containing a record and it’s corresponding match status with respect to the search term. The record contains fields from the Output Mapping which could potentially match the provided search term. If no output mapping has been configured (i.e. because you have run a MATCH job), then the output will contain fields recordID, sourceID, and clusterID.

The Search result order is in descending order of matching status. So EXACT matches are returned first in the collection.

Potential matches are determined using the rule set configured in the Session.

If you wish to override the fields which are use for searching, or the rules use for matching, and output fields this will require additional configuration.

After a match job has run, you may wish to find clusters from a single Record ID or a Cluster ID.

When searching using a Cluster ID, the result will be all records in the specified cluster. The path parameters are the Session ID and the Cluster ID being searched for.

The response object will look like this:

When searching using a Record ID, the result will be all records in the same cluster, including the specified record ID. The path parameters are the Session ID, and the Source ID and Record ID being searched for.

The response object will look like this:

A result JSON object is produced upon successful completion of the maintenance request. This contains the identifier of the affected Record in the Match Store along with the clustering outcome (see below), the Cluster Id to which the Record has been allocated and a collection containing the full set of Record ids in the same Cluster. In addition, a changes collection contains the Clusters and their member identifiers that have been affected by the operation.

When a maintenance operation is performed, the final stage is to regenerate the Clusters for all Records affected by the operation, i.e. scored together or previously clustered with the Record in question. The outcome of re-clustering may be one of several cases.

A transactional request will be rejected in any of the following circumstances:

Note that for a Delete no check is made that the Record Id has been deleted from the data source: this is the responsibility of the client workflow.

Configuration for input data source, output sink and rule set.

Run a full matching job or individual stages.

Search records in the system.

Configure a session to use with matching operations.

Retrieve status and license information.

Perform transactional maintenance of records in the system.