Evaluation using FrameBase TBox
Overview
We evaluate the performances of PIKES as an ontology population approach for the FrameBase ontological schema, reporting precision and recall against a manually annotated gold standard based on the text used in [1] . We experiment with three different configurations of the linguistic feature extraction phase that differ for the Semantic Role Labeling (SRL) tools used:
- Semafor only, performing SRL w.r.t. FrameNet;
- Mate-tools only, performing SRL w.r.t. PropBank and NomBank; and,
- both Semafor and Mate-tools, relying on the automatic combinations of the respective annotations in the mention graph.
Sentences and graphs
The following table lists the sentences of the gold standard used for the evaluation, each one associated to four knowledge graphs containing type and role triples using the FrameBase vocabulary: a gold graph manually built by two annotators, and three graphs produced by PIKES for the three configurations considered.
In order to simplify the manual construction of gold graphs, the link between an instance in a gold graph and the corresponding mention is implicit and given by the instance URI, whose local name corresponds to the head token of the mention in the text. In case of ambiguities, i.e., if there are multiple occurrences of a word in the sentence, a sequential index is added (e.g., in sentence S7, :syria_1 and :syria_2 refer respectively to the first and second occurrences of Syria).
PIKES graphs were obtained using the public demo of PIKES, converting the output from TriG to Turtle to get rid of provenance information not needed for this evaluation.
| Sentence | Text | Gold graph | PIKES graphs | ||
|---|---|---|---|---|---|
| Semafor | Mate | Both | |||
| S1 | The lone Syrian rebel group with an explicit stamp of approval from Al Qaeda has become one of the uprising most effective fighting forces, posing a stark challenge to the United States and other countries that want to support the rebels but not Islamic extremists. | .ttl | .ttl | .ttl | .ttl |
| S2 | Money flows to the group, the Nusra Front, from like-minded donors abroad. | .ttl | .ttl | .ttl | .ttl |
| S3 | Its fighters, a small minority of the rebels, have the boldness and skill to storm fortified positions and lead other battalions to capture military bases and oil fields. | .ttl | .ttl | .ttl | .ttl |
| S4 | As their successes mount, they gather more weapons and attract more fighters. | .ttl | .ttl | .ttl | .ttl |
| S5 | The group is a direct offshoot of Al Qaeda in Iraq, Iraqi officials and former Iraqi insurgents say, which has contributed veteran fighters and weapons. | .ttl | .ttl | .ttl | .ttl |
| S6 | This is just a simple way of returning the favor to our Syrian brothers that fought with us on the lands of Iraq, said a veteran of Al Qaeda in Iraq, who said he helped lead the Nusra Front's efforts in Syria. | .ttl | .ttl | .ttl | .ttl |
| S7 | The United States, sensing that time may be running out for Syria president Bashar al-Assad, hopes to isolate the group to prevent it from inheriting Syria. | .ttl | .ttl | .ttl | .ttl |
| S8 | As the United States pushes the Syrian opposition to organize a viable alternative government, it plans to blacklist the Nusra Front as a terrorist organization, making it illegal for Americans to have financial dealings with the group and prompting similar sanctions from Europe. | .ttl | .ttl | .ttl | .ttl |
Evaluation results
The table below reports the results of the evaluation for the three configuration, both against FrameBase type triples for frame instances (first three columns), FrameBase role triples (next three columns), and all triples undifferentiated (next three columns).
As one could expect, F1 scores using Mate-tools are lower than the ones obtained using Semafor, reflecting the fact that the latter, being specifically designed for FrameNet SRL, is more suitable for use with FrameBase. However, the combination of both tools in PIKES leads to an increase of recall for role triples with respect to Semafor (or Mate-tools) alone. Note that precision scores for Mate-tools are on par with the ones for Semafor, with a gap in terms of recall that could be potentially addressed with further work on PropBank/NomBank to FrameBase mapping resources.
| Configuration | Type triples | Role triples | All triples | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Precision | Recall | F1 | Precision | Recall | F1 | Precision | Recall | F1 | |
| Semafor | .617 | .698 | .655 | .594 | .352 | .442 | .605 | .466 | .526 |
| Mate | .792 | .358 | .494 | .633 | .176 | .275 | .704 | .236 | .353 |
| Both | .603 | .717 | .655 | .595 | .435 | .503 | .599 | .528 | .561 |
References
- A Comparison of Knowledge Extraction Tools for the Semantic Web.
By Aldo Gangemi.
In ESWC 2013 Proceedings, Springer Berlin Heidelberg, volume 7882, pages 351-366, 2013.
[online version]
