Comparative evaluation - Roles (VN)

* irrelevant element returned by evaluated system, not considered as false positive

SentenceGoldFREDPIKES
S1:posing vn:agent :group:pose_1 vn:agent :group_1:pose vn:agent :group
:posing vn:recipient :countries:pose_1 vn:recipient :united_states_1_countries
:posing vn:recipient :states:pose_1 vn:recipient :united_states_1_States
:posing vn:topic :challenge:pose vn:topic :challenge
:support vn:agent :countries
:support vn:agent :states
:support vn:beneficiary :rebels
:support_1 vn:agent :countries:support_1 vn:agent :united_states_1_countries
:support_1 vn:agent :states:support_1 vn:agent :united_states_1_States
:support_1 vn:beneficiary :extremists
:want vn:pivot :countries
:want vn:pivot :states
:want vn:theme :support
:want vn:theme :support_1:want_1 vn:theme :support_1
:become_1 vn:agent :group_1
:become_1 vn:patient :uprising_1
:pose_1 vn:theme :challenge_1
:want_1 vn:experiencer :united_states_1_countries
:want_1 vn:experiencer :united_states_1_States
:support vn:experiencer :country
:support vn:stimulus :extremist
:support vn:stimulus :rebel_2
:want vn:agent :country
:want vn:result :support
S2:flows vn:theme :money:flow_1 vn:theme :money_1:flow vn:theme <entity:money>
:flow_1 vn:location :Nusra_front
:flow_1 vn:location :group_1
S3:capture vn:agent :battalions:capture_1 vn:agent :battalion_1:capture vn:agent :battalion
:capture vn:theme :bases:capture_1 vn:theme :military_bases_1:capture vn:theme :basis
:capture vn:theme :fields:capture_1 vn:theme :oil_fields_1:capture vn:theme :field
:have vn:pivot :fighters:have vn:pivot :fighter
:have vn:theme :boldness:have vn:theme :boldness
:have vn:theme :skill:have vn:theme :skill
:lead vn:agent :fighters
:lead vn:patient :battalions:lead vn:patient :battalion
:lead vn:result :capture:lead vn:result :capture
:fortify_1 vn:patient :storm_1_positions
:have_1 vn:agent :fighter_1
:have_1 vn:patient :boldness_1_boldness
:have_1 vn:patient :boldness_1_skill
S4:gather vn:agent :they:gather_1 vn:agent :thing_2:gather vn:agent :they_2
:gather vn:theme :weapons:gather_1 vn:theme :weapon_1:gather vn:theme :weapon
:mount vn:theme :successes:mount vn:theme :success
:attract_1 vn:agent :thing_2
:attract_1 vn:patient :fighter_1
:mount_1 vn:agent :success_1
S5:contributed vn:agent :which
:contributed vn:theme :fighters:contribute_1 vn:theme :fighter_1_fighters
:contributed vn:theme :weapons:contribute_1 vn:theme :fighter_1_weapons
:is vn:attribute :offshoot
:is vn:theme :group
:say vn:agent :insurgents:say_1 vn:agent :iraqi_insurgents_1:say vn:agent :insurgent
:say vn:agent :officials:say vn:agent :official
:say vn:topic :contributed
:say vn:topic :is
* :have vn:pivot :Al_Qaeda
* :have vn:theme :fighter
* :have vn:theme :weapon
:say vn:theme :be
S6:fought vn:agent :brothers:fight vn:agent :brother
:fought vn:agent :us
:helped vn:agent :he:help vn:agent :he
:helped vn:theme :lead:help vn:theme :lead
:is vn:attribute :way
:is vn:theme :this
:lead vn:agent :he:lead_1 vn:agent :Al_qaeda_in_iraq_he:lead vn:agent :he
:lead vn:theme :efforts:lead_1 vn:theme :effort_1:lead vn:theme :effort
:returning vn:recipient :brothers:return_1 vn:recipient :brother_1
:returning vn:theme :favor:return_1 vn:theme :favor_1:return vn:theme :favor
:said_1 vn:agent :veteran:say_1 vn:agent :veteran_1:say_2 vn:agent :veteran
:said_1 vn:topic :is
:said_2 vn:agent :who
:said_2 vn:topic :helped
:fight_1 vn:actor1 :brother_1
:fight_1 vn:actor2 :person_2
* :return_1 vn:agent :thing_1
* :say_2 vn:agent :Al_qaeda_in_iraq_he
* :say_2 vn:topic :lead_1
:say vn:agent :Al_Qaeda
:say vn:theme :help
S7:hopes vn:experiencer :states:hope_1 vn:experiencer :united_states_1:hope vn:experiencer :United_States
:hopes vn:theme :isolate:hope vn:theme :isolate
:inheriting vn:agent :it:inherit vn:agent :it
:inheriting vn:theme :syria_2:inherit_1 vn:theme :Syria_Syria_2:inherit vn:theme :Syria_2
:isolate vn:agent :states:isolate_1 vn:agent :united_states_1
:isolate vn:patient :group:isolate vn:patient :group
:prevent vn:agent :states:prevent vn:agent :United_States
:prevent vn:theme :inheriting:prevent vn:theme :inherit
:prevent vn:theme :it:prevent_1 vn:theme :time_1_it:prevent vn:theme :it
:sensing vn:experiencer :states:sense_1 vn:experiencer :united_states_1:sense vn:experiencer :United_States
:sensing vn:stimulus :running:sense vn:stimulus :run
* :be_1 vn:agent :time_1_time
* :be_1 vn:theme :run_1
:isolate_1 vn:theme :group_1
* :prevent_1 vn:agent :group_1
:run_1 vn:location :Syria_president
:run_1 vn:theme :time_1_time
* :sense_1 vn:stimulus :be_1
* :sense vn:stimulus :may
S8:have vn:pivot :americans:have vn:pivot :Americans
:have vn:theme :dealings:have vn:theme :dealings
:making vn:agent :it_1:make vn:agent :it
:making vn:result :illegal
:making vn:theme :have
:organize vn:agent :opposition:organize_1 vn:agent :syrian_opposition_1
:organize vn:theme :government:organize_1 vn:theme :government_1
:plans vn:experiencer :it_1:plan vn:experiencer :it
:plans vn:theme :blacklist:plan_1 vn:theme :blacklist_1:plan vn:theme :blacklist
:prompting vn:agent :it_1
:prompting vn:patient :europe
:prompting vn:result :sanctions
:pushes vn:agent :states:push_1 vn:agent :United_states:push vn:agent :United_States
:pushes vn:patient :opposition:push vn:patient :opposition
:pushes vn:result :organize
* :blacklist_1 vn:agent :United_states
:blacklist_1 vn:patient :Nusra_front
:have_1 vn:agent :Americans
:have_1 vn:patient :dealings_1
* :make_1 vn:agent :United_states
* :plan_1 vn:experiencer :United_states
* :prompt_1 vn:agent :United_states
:prompt_1 vn:patient :sanction_1
:push_1 vn:destination :organize_1
:dealings vn:agent :Americans
:dealings vn:theme <attr:fiscal-1a>
:dealings vn:theme :group
:make vn:result :have
* :make vn:result :it_2
:organize vn:result :government
:push vn:result :opposition
Results w.r.t. gold standardf1=0.450 p=0.547 r=0.382 tp=29 fp=24 fn=47f1=0.615 p=0.741 r=0.526 tp=40 fp=14 fn=36
Results w.r.t. union of correct answersf1=0.558 p=0.547 r=0.569 tp=29 fp=24 fn=22f1=0.762 p=0.741 r=0.784 tp=40 fp=14 fn=11