Text Reference Evaluation
Introduction
Sintelix provides a mechanism to create an evaluation report by comparing text references. This is called Text Reference Evaluation.
- Only document collections A collection is a container for storing and organising ingested files and documents. Only the textual content is stored in collections, not the original files and documents. containing the same documents (that is, identical source) are eligible for evaluation.
- One document collection is elected as the Gold Standard, while another is used as the Test collection.
- The Gold Standard collection acts as the base for comparison (which determines the terminology used in the evaluation report)
- For example, if a text reference is found in the Gold document but not in the Test document; evaluation will report it as a MISS.
- The evaluation result is summarised by a F1 score. If the Gold and Test documents are identical, the maximum F1 of 100% is achieved.
- F1 score degrades when dissimilarity between the Gold and Test documents increases.
- You can compare a collection with itself to extract the Tag Counts.
Navigating to Text Reference Evaluation
1. Select Documents on the Sintelix Menu Bar.
2. Select the Text Reference Evaluation button.
This opens the Evaluation Page as shown in the next section.
The Evaluation Page
On the left of the Evaluation Page is the Existing Evaluations list pane at the top, and the Create Evaluation Pane at the bottom.
Existing Evaluations
Click on any existing Evaluation in the list to see the Summary Report associated with that evaluation.
We recommend deleting evaluations as soon as you are finished with them.
Renaming either of the Collections involved in the evaluation will orphan the evaluation, making it unusable. Undoing the rename will restore the evaluation.
Create New Evaluation
Review your settings and click the Save & Evaluate button to begin the evaluation process.
The evaluation process may take a while, and a spinner will be displayed during the process. You can continue using Sintelix while waiting, as the evaluation will continue in background.
View the Evaluation Report
-
After the evaluation process is done, locate the name of your evaluation on the listing. Select it to see the evaluation report.
For a shortcut, you'll find a document collection starting with [Evaluation] on the collection list. Click on the Report icon to show it as an evaluation report.
-
A detailed summary table will show the overall F1 score and other components. Select the Blue Numbers to drill down to documents.
-
To delete the Evaluation Report when you finish, click the Trash button. The document collection created for evaluation will be removed as well.
Both your original Gold and Test documents are safe and are not modified.
Compare a Collection with itself
To extract tag counts from a Collection.
Enter a unique name for the evaluation, then select Create.
From the Evaluation Settings pane, select the same Collection for the Gold Standard Documents and the Test Documents.
Click Save & Evaluate to start the evaluation running.
The message Please wait, processing in background will display.
At the end of the process, the Summary Table displays.
The two columns display in the Summary Table showing the Gold Tag categories, and the Correct counts.
The Summary Table can be cut-and-pasted into an Excel spreadsheet.
Selecting any number in the Correct column will display an Evaluation Tags pane with details of the tags that were in this Tag category. In this scenario, the number outlined in red represents the count for category Ethnicity.
