Getting PIKES

text2naf

Given some text, this returns a NAF file containing the linguistic annotations produced by the annotators used in PIKES. Call example (server is the name of the server hosting PIKES - e.g. localhost):

http://server:8011/text2naf?text=G.%20W.%20Bush%20and%20Bono%20are%20very%20strong%20supporters%20of%20the%20fight%20of%20HIV%20in%20Africa.%20Their%20March%202002%20meeting%20resulted%20in%20a%205%20billion%20dollar%20aid.

text2rdf

Given some text, this returns the RDF content (in TriG format) extracted by PIKES from the given text. Call example:

http://server:8011/text2rdf?text=G.%20W.%20Bush%20and%20Bono%20are%20very%20strong%20supporters%20of%20the%20fight%20of%20HIV%20in%20Africa.%20Their%20March%202002%20meeting%20resulted%20in%20a%205%20billion%20dollar%20aid.

webdemo

PIKES comes with a web interface (such as the web demo available on the PIKES web site); you need graphviz to be installed on the server to run it. With Debian/Ubuntu, just run apt-get install graphviz and restart PIKES. The demo interface (with input textbox for text) is written in php and available under the src/webdemo/ folder in the project. To access it, just surf to

http://server:8011/webdemo

PIKES execution and processing (e.g., which annotators are used) is configurable via a property file. See e.g. the config-pikes.prop file included in the PIKES package. If you want to pass Stanford CoreNLP configurations, just prepend stanford. to the name of the preference (see e.g. stanford.dcoref.maxdist in the provided configuration file). To select the annotators to be applied, just change the value of stanford.annotators. For example, with stanford.annotators = tokenize, ssplit you’ll have only tokenizer and sentence splitter.

The maximum length of text (in characters) processed by PIKES is limited by the max_text_len property in the configuration file. Set this value according to you needs.

If you experience problems or very slow performances, try increasing the RAM value in the run.sh script.

STEP 0 - Creating the input NAF files

The documents to be processed by PIKES have to be provided as “input NAF” files. An “input NAF” file is a NAF file which contains the text to be processed within the <raw> tags, and with a minimal set of header attributes.

Example of minimal input NAF file:

<?xml version="1.0" encoding="UTF-8"?>
<NAF xml:lang="en" version="v3">
  <nafHeader>
    <fileDesc title="Juan Bautista de Anza and the Route to San Francisco Bay" />
    <public publicId="d331" uri="http://blog.yovisto.com/juan-bautista-de-anza-and-the-route-to-san-francisco-bay/" />
  </nafHeader>
  <raw><![CDATA[Juan Bautista de Anza and the Route to San Francisco Bay.
 
Juan Bautista de Anza, from a portrait in oil by Fray Orsi in 1774.  On March 28, 1776, Basque New-Spanish...]]></raw>
</NAF>

To automatically build the input NAF files for your document collection, have a look at some of the converters we have developed. A generic java code for converting plain text files to input NAFs can be found here.

STEP 1 - Processing the input NAF files with PIKES

To process the input NAF files with PIKES you can adapt the script process.sh provided in the PIKES full package:

#!/bin/bash
CLASSPATH=pikes-tintop-1.0-SNAPSHOT-jar-with-dependencies.jar:Semafor-3.0-alpha-04.jar:models/stanford-corenlp-3.7.0-models.jar
CONF=config-pikes.prop
INPUT_FOLDER=path-to-input-naf-folder
OUTPUT_FOLDER=path-to-processed-naf-folder
PARALLEL_INSTANCES=1
MAXLENGTH=4500000
RAM=-Xmx12G

java $RAM -cp $CLASSPATH eu.fbk.dkm.pikes.tintop.FolderOrchestrator -c $CONF -i $INPUT_FOLDER -o $OUTPUT_FOLDER -s $PARALLEL_INSTANCES -z $MAXLENGTH

Adapt the script with the actual (full path) containing the input NAF files (INPUT_FOLDER) and where you want the processed files to be placed (OUTPUT_FOLDER). Input NAF files may be organized in subfolders, and PIKES will preserve in the the output folder the subfolder structure. PARALLEL_INSTANCES tells PIKES how many instances to be run in parallel. MAXLENGTH tells PIKES to skip (i.e. not process) files with more than MAXLENGTH characters (very large files may take long time to be processed). RAM tells pikes how much memory to use (you may have to increase it, based on the size of the files to be processed).

PIKES is configured to process only those input NAF files for which no corresponding processed NAF file is available in the OUTPUT_FOLDER. That is, re-running process.sh on the same INPUT_FOLDER will not overwrite any file already in the OUTPUT_FOLDER. At the same time, if processing of the INPUT_FOLDER stops for any reason, by re-running process.sh PIKES will process only the not-yet processed input NAF files.

You can test the the processing in this step with the input NAF files available here.

STEP 2 - Generating the RDF file(s)

To generate the RDF content from the processed NAF files you can adapt the script generateRDF.sh provided in the PIKES full package:

#!/bin/bash
CLASSPATH=pikes-tintop-1.0-SNAPSHOT-jar-with-dependencies.jar
NAF=path-to-processed-naf-folder
RDF=path-to-rdf-folder
OUTPUT_FILE=$RDF/output.tql.gz

java -cp $CLASSPATH eu.fbk.dkm.pikes.rdf.Main rdfgen $NAF -o $OUTPUT_FILE -n -m -r

Adapt the script with the actual (full path) containing the input NAF files (NAF) and where you want the RDF files to be placed (RDF). MERGED defines the name of the file containing the RDF content from all the NAFs considered (unless the -i flag is used). It is also used to decide, via the file extension provided, the RDF serialization used (e.g., tql in the example). The flags at the end of the java call affect the behaviour of PIKES and the generated RDF content:

• [-r,--recursive] : convert also files recursively nested in specified directories
• [-i,--intermediate] : produce single RDF files (one for each input NAF) instead of a single output file; output path and format are extracted from MERGED
• [-m,--merge] : merge instances (smushing plus filtering of group instances)
• [-n,--normalize] : normalize/compact output so to use less metadata statements

You can test the the generation of RDF content with the processed NAF files available here.