JPARK logo

This page provides additional details on JPARK, an ontological knowledge powered probabilistic approach for jointly revising multiple NLP entity annotations.

The proposed approach is fully implemented and evaluated in the following paper:

  • Joint Posterior Revision of NLP Annotations via Ontological Knowledge
    By Marco Rospocher and Francesco Corcoglioniti.
    In Proceedings of the 27th International Joint Conference on Artificial Intelligence and the 23rd European Conference on Artificial Intelligence, IJCAI-ECAI 2018, Stockholm, Sweden, July 13-19, 2018
    [bib] [pre-print/mirror]

JPARK has been evaluated on three reference datasets for Named Entity Recognition and Classification (NERC) and Entity Linking (EL):

  • AIDA CoNLL-YAGO: This dataset consists of 1,393 English news wire articles from Reuters, with 34,999 mentions hand-annotated with named entity types (PER, ORG, LOC, MISC) for the CONLL2003 shared task on named entity recognition, and later hand-annotated with the YAGO2 entities and corresponding Wikipedia page URLs. It is split in three parts: eng.train (946 docs), eng.testa (216 docs), eng.testb (231 docs).
  • MEANTIME: The NewsReader MEANTIME corpus consists of 480 news articles from Wikinews, in four languages. In our evaluation, we used only the English section and its 120 articles. The dataset, used as part of the SemEval 2015 task on TimeLine extraction, includes manual annotations for named entity types (only PER, ORG, LOC) and DBpedia entity links.
  • TAC-KBP: Developed for the TAC KBP 2011 Knowledge Base Population Track, this dataset consists of 2,231 English documents, including newswire articles and posts to blogs, newsgroups, and discussion fora. For each document, it is known that all the mentions of one or a few query entities can be linked to a certain Wikipedia page and to a specific NERC type (only PER, ORG, LOC), giving rise to a (partially) annotated gold standard for NERC and EL.

The following JPARK resources used in the IJCAI-18 paper are made available:

  • TSV (~39MB) containing the model (used in the IJCAI-18 experiments) built with YAGO as background knowledge, and trained on AIDA CoNLL-YAGO (eng.train). Its columns contain:
    1. a YAGO Class Set (classes in the set are space separated)
    2. the conditional probability — cf. eq. (6) in the paper — of having that class set given a NERC PER annotation
    3. the conditional probability — cf. eq. (6) in the paper — of having that class set given a NERC ORG annotation
    4. the conditional probability — cf. eq. (6) in the paper — of having that class set given a NERC LOC annotation
    5. the conditional probability — cf. eq. (6) in the paper — of having that class set given a NERC MISC annotation
    6. the prior probability of that class set estimated from the ontological background knowledge — cf. eq. (7) in the paper
    7. all the entities (space separated) having as types exactly the classes in that class set
  • PDF (~83KB) file containing all evaluation metrics computed for all measures of IJCAI-18 paper, with and without using JPARK, by
    • micro-averaging, considering only mentions in the gold standard;
    • micro-averaging, considering all mentions returned by the system;
    • macro-averaging by document;
    • macro-averaging by NERC type.
  • FOLDER (~985KB) package of the IJCAI-18 evaluation folder, containing:

    • the official TAC scorer;
    • commands for computing scores (and statistical significance) for all metrics and measures considered (cf. the paper for details on interpreting the values);
    • gold, standard, and JPARK annotations for all datasets (excluding TAC-KBP, under LDC copyright).


Additional evaluation material (manuscript describing the new developments and experiments currently under review):

  • JPARK models:
    • TSV (~37MB) containing the model (the same used in the IJCAI-18 experiments) built with YAGO as background knowledge, and trained on AIDA CoNLL-YAGO (eng.train).
    • TSV (~29MB) containing the model (NOT used in the IJCAI-18 experiments) built with DBpedia Ontology as background knowledge, and trained on AIDA CoNLL-YAGO (eng.train). It contains the same information as the YAGO model, but with DBpedia Ontology class sets instead.
    • TSV (~20MB) containing the model (NOT used in the IJCAI-18 experiments) built with Wikidata as background knowledge, and trained on AIDA CoNLL-YAGO (eng.train). It contains the same information as the YAGO model, but with Wikidata class sets instead.
  • TSV (~94B) NIL Priors, trained on AIDA CoNLL-YAGO (eng.train), for the different NERC categories
  • LINK to download the PIKES binary and models
  • NAF files annotated (also with NIL confidences) with PIKES (as the PIKES annotated files contain the whole text, due to copyright restrictions, only the MEANTIME annotated files can be made available)
    • FOLDER (~10MB), with Stanford NER and DBpedia Spotlight independently spotting named entities
    • FOLDER (~7MB), using the Stanford NER spotter also for DBpedia Spotlight
    • FOLDER (~7MB), with Flair and End-to-End Neural Entity Linking independently spotting named entities
  • JAR (~64MB) JAR binary of JPARK
  • FOLDER (~3MB) package of the evaluation folder, containing:
    • the official TAC scorer;
    • commands for computing scores (and statistical significance) for all metrics and measures considered;
    • gold standard, baselins, and JPARK (with and without NIL extension) annotations for all datasets (excluding TAC-KBP, under LDC copyright).
  • Full evaluation results (with baseline and JPARK scores, as well as p-values):
    • TXT: using separate named entity spotters for NERC (Stanford NER) and EL (DBpedia Spotlight);
    • TXT: using same named entity spotter (Stanford NER) for NERC (Stanford NER) and EL (DBpedia Spotlight);
    • TXT: performance upper bounds for the posterior revision of the annotations NERC (Stanford NER) and EL (DBpedia Spotlight);
    • TXT: using separate named entity spotters for NERC (Flair) and EL (End-to-End Neural Entity Linking);
    • TXT: performance upper bounds for the posterior revision of the annotations NERC (Flair) and EL (End-to-End Neural Entity Linking).

Back to top

Last Published: 2020/07/15.

Reflow Maven skin by Andrius Velykis.

Data and Knowledge Management tools