Evaluating ChatGPT’s Information Extraction Capabilities: An

Assessment of Performance, Explainability, Calibration, and Faithfulness

Bo Li1,2 , Gexiang Fang1,2 , Yang Yang1,2 , Quansen Wang3 ,
Wei Ye1 , Wen Zhao1 , and Shikun Zhang1
1
National Engineering Research Center for Software Engineering, Peking University
2
School of Software and Microelectronics, Peking University
3
Boston University
{deepblue.lb, fanggx, yangy}@stu.pku.edu.cn, quansenw@bu.edu
{wye, zhaowen, zhangsk}@pku.edu.cn
Abstract ChatGPT 2 is currently the most popular LLM,
known for its impressive ability to understand user
The capability of Large Language Models intent and generate human-like responses. Chat-
(LLMs) like ChatGPT to comprehend user GPT is trained on the GPT family (Brown et al.,
arXiv:2304.11633v1 [cs.CL] 23 Apr 2023

intent and provide reasonable responses has
made them extremely popular lately. In this
paper, we focus on assessing the overall abil-
ity of ChatGPT using 7 fine-grained infor-
mation extraction (IE) tasks. Specially, we
present the systematically analysis by mea-
suring ChatGPT’s performance, explainabil-
ity, calibration, and faithfulness, and resulting
in 15 keys from either the ChatGPT or do-
main experts. Our findings reveal that Chat-
GPT’s performance in Standard-IE setting is
poor, but it surprisingly exhibits excellent per-
formance in the OpenIE setting, as evidenced
by human evaluation. In addition, our research
indicates that ChatGPT provides high-quality
and trustworthy explanations for its decisions.
However, there is an issue of ChatGPT being
overconfident in its predictions, which result-
ing in low calibration. Furthermore, ChatGPT
demonstrates a high level of faithfulness to the
original text in the majority of cases. We man-
ually annotate and release the test sets of 7 fine-
grained IE tasks contains 14 datasets to further
promote the research. The datasets and code
are available at this url. 1
1 Introduction
Large Language Models (LLMs) (e.g., GPT-
3 (Brown et al., 2020), LaMDA (Thoppilan et al.,
2022) and PaLM (Chowdhery et al., 2022), etc.)
have greatly promoted the development of the
Natural Language Processing (NLP) community.
With a proper instruction (often the task defini-
tion) (Ouyang et al., 2022; Kojima et al., 2022;
Chung et al., 2022; Wang et al., 2022) and the chain-
of-thought (CoT) prompting (Wei et al., 2022b),
LLMs achieve surprisingly good performances
when dealing with unseen tasks.
1 2
https://github.com/pkuserc/ChatGPT_for_IE
2020; Artetxe et al., 2022; Ouyang et al., 2022)
using reinforcement learning from human feed-
back (RLHF) (Christiano et al., 2017) and high-
quality conversational-style datasets. Apart from
its surprising dialogue ability, ChatGPT has many
other aspects that attract researchers to explore.
Some researchers have delved into the potential
impacts of ChatGPT on human life (Haque et al.,
2022; Zhuo et al., 2023; Susnjak, 2022; Basic et al.,
2023). Other researchers are interested in explor-
ing the capabilities of ChatGPT for various NLP
tasks (Zhang et al., 2022a; Qin et al., 2023; Mitro-
vic et al., 2023; Guo et al., 2023). The capabili-
ties of ChatGPT have been preliminarily explored
through the above research and valuable conclu-
sions have been drawn.
Given ChatGPT is a closed model that does not
provide information about its training details, and
any response from the model encodes an opin-
ion. The response can significantly impact the
user’s experience and shape their beliefs going for-
ward (Aiyappa et al., 2023; Santurkar et al., 2023;
Deshpande et al., 2023; Huang et al., 2023). Con-
sequently, evaluating ChatGPT should involve not
only assessing its ability to achieve high perfor-
mance but also measuring the reliability of the
answers it provides. To help users better under-
stand the overall quality of ChatGPT’s responses
and enable systematic measurement of its capabil-
ities, we design the following four metric dimen-
sions: The first dimension we consider is Perfor-
mance, which reflects ChatGPT’s overall perfor-
mance on various IE tasks from multiple perspec-
tives. The second metric dimension, Explainabil-
ity (Rajani et al., 2019; Aghajanyan et al., 2021;
Zini and Awad, 2023), evaluates whether Chat-
https://chat.openai.com/
GPT could give a justified reason for its predic-
tion, thereby providing insights into ChatGPT’s
decision-making process. The third one is Cal-
ibration (Guo et al., 2017; Kumar et al., 2019;
Thulasidasan et al., 2019; Minderer et al., 2021),
which measures the predictive uncertainty of a
model, and we use this metric to assess if Chat-
GPT is overconfidence on its prediction. The last
dimension is Faithfulness (Maynez et al., 2020;
Koto et al., 2022; Creswell and Shanahan, 2022;
He et al., 2023), it is frequently employed in the
summarization task to determine whether the sum-
mary accurately reflects the input. In our research,
we adopt faithfulness as a measure of whether the
explanations given by ChatGPT are truthful to the
input, or if they are spurious. In summary, accord-
ing to the above four dimensions, we collect 15
keys from either the ChatGPT or domain experts
for the evaluation (§ 3).
In this research, we aim to perform a compre-
hensive study and detailed analysis of ChatGPT’s
capabilities through various information extraction
(IE) tasks. IE involves heterogeneous structure ex-
traction, factual knowledge usage, and diversified
targets(Yamada et al., 2020; Paolini et al., 2021; Lu
et al., 2022), making it an ideal scenario for evalu-
ating ChatGPT’s capabilities. Overall, we conduct
our experiments and analysis based on 14 datasets
belonging to 7 fine-grained IE tasks(§ 4). Addition-
ally, we assess the explainability, calibration, and
faithfulness of ChatGPT’s responses through both
self-check and human-check (§ 5). To sum up, our
main contributions are summarized as follows:
• To assess the overall ability of ChatGPT, we
employ a comprehensive and systematic eval-
uation from four dimensions: 1) performance,
2) explainability, 3) calibration, and 4) faith-
fulness. We then collected 15 keys belong-
ing to above dimensions from either the Chat-
GPT or domain experts for the research. All
the manually annotated datasets and code are
made public available for future research.
• We comprehensively evaluate the overall per-
formance of ChatGPT on various tasks in both
Standard-IE and OpenIE settings and compare
it with other popular models. Our research in-
dicates that ChatGPT’s performance is not sat-
isfactory in the Standard-IE setting. However,
we show that it provides surprisingly good
results in the OpenIE setting, as confirmed
by human evaluation. Furthermore, we also
discover that ChatGPT provides high-quality
and trustworthy explanations for its decisions.
Although, it displays overconfidence in its pre-
dictions, leading to low calibration. Besides,
ChatGPT is largely faithful to the original text
in most cases.
2 Related Work
2.1 Large Language Models
Large Language Models (LLMs) typically con-
tain more than a hundred billion parameters, such
as GPT-3 (Brown et al., 2020), Gopher (Rae
et al., 2021), LaMDA (Thoppilan et al., 2022),
Megatron-turing-NLG (Smith et al., 2022), and
PaLM (Chowdhery et al., 2022), among others.
Scaling up the model size brings impressive abili-
ties on few-shot and zero-shot learning scenarios,
such as producing reasonable results with very few
samples or task descriptions (Brown et al., 2020;
Chowdhery et al., 2022). Moreover, scaling up the
model size unlocks emergent abilities that were not
observed in smaller models, enabling LLMs to ex-
hibit strong generalizability on unseen tasks (Wei
et al., 2022a; Fu et al., 2022; Mahowald et al.,
2023).
With its impressive ability to understand user in-
tent and generate human-like responses, ChatGPT
has become the most popular language model cur-
rently. It is trained on the GPT family (Brown et al.,
2020; Artetxe et al., 2022; Ouyang et al., 2022)
and high-quality conversational-style datasets us-
ing reinforcement learning from human feedback
(RLHF) (Christiano et al., 2017).
Along with its dialogue ability, ChatGPT has
other aspects that researchers are exploring. Some
researchers show potential impacts of ChatGPT
on human life, such as ethical risks (Haque et al.,
2022; Zhuo et al., 2023; Krügel et al., 2023), the
education sector (Susnjak, 2022; Basic et al., 2023;
Kortemeyer, 2023) and the medical scenario (Tu
et al., 2023; Nov et al., 2023; Jeblick et al., 2022).
Additionally, some researchers are keen to examine
the potential of ChatGPT in addressing various nat-
ural language processing tasks. For instance, some
works have examined ChatGPT’s performance in
stance detection (Zhang et al., 2022a), linguistic
and sentiment analysis (Susnjak, 2023; Ortega-
Martín et al., 2023), general NLP tasks (Qin et al.,
2023; Bian et al., 2023; Zhong et al., 2023; Wang
et al., 2023a,b), and machine translation (Jiao et al.,
2023). (Frieder et al., 2023) explores the mathe-
matical capabilities of ChatGPT, while (Bang et al.,
2023) proposes an evaluation of ChatGPT on rea-
soning and other aspects. Additionally, (Mitro-
vic et al., 2023; Guo et al., 2023) investigate the
differences between human-written and ChatGPT-
generated.
2.2 Information Extraction
Information Extraction (IE) is a long-standing re-
search topic that aims to extract structured fac-
tual information from unstructured texts (Andersen
et al., 1992; Crowe, 1995; Chieu et al., 2003; Wu
and Weld, 2010; Khot et al., 2017; Lu et al., 2022).
Typically, IE involves a wide range of tasks, such
as named entity recognition(NER) (Gregoric et al.,
2018; Martins et al., 2019; Li et al., 2020; Das
et al., 2022), entity typing(ET) (Choi et al., 2018;
Dai et al., 2021; Pang et al., 2022; Chen et al.,
2022), relation extraction(RE) (Li et al., 2019; Fu
et al., 2019; Bian et al., 2021; Ye et al., 2022),
relation classification(RC) (Zeng et al., 2015; Ye
et al., 2019; Zhou and Chen, 2021; Li et al., 2022b),
event detection(ED) (Veyseh et al., 2021; Lou et al.,
2021; Liu et al., 2022a; Zhao et al., 2022), event
argument extraction(EAE) (Zhang et al., 2022b;
Du and Ji, 2022; Ma et al., 2022), and event ex-
traction(EE) (Wadden et al., 2019; Du and Cardie,
2020; Liu et al., 2022b; Hsu et al., 2022), among
others. These tasks automatically generate struc-
tured factual outputs related to entity, and relation,
and event, and greatly boost the development of
NLP community.
3 ChatGPT for Information Extraction
In this section, we first briefly introduce 7 fine-
grained IE tasks, then we present how to collect 15
keys from the ChatGPT and domain experts.
3.1 Information Extraction
IE involves a wide range of tasks which need to
extract structured factual information from unstruc-
tured texts, such as entity, and relation, and event.
In this research, we conduct our analysis on the
following 7 fine-grained IE tasks:3 1) Entity Typ-
ing(ET) (Choi et al., 2018; Dai et al., 2021; Pang
et al., 2022; Chen et al., 2022) aims to classify
the type of a target entity under a given input;
2) Named Entity Recognition(NER) (Gregoric
3
We introduce these tasks briefly due to the space limita-
tion. Please refer to the task-specific papers for more details.
et al., 2018; Martins et al., 2019; Li et al., 2020; Das
et al., 2022) aims to first identify the candidate enti-
ties, and then classify their types; 3) Relation Clas-
sification(RC) (Zeng et al., 2015; Ye et al., 2019;
Zhou and Chen, 2021; Li et al., 2022b) requires to
classify the relation between two target entities; 4)
Relation Extraction(RE) (Li et al., 2019; Fu et al.,
2019; Bian et al., 2021; Ye et al., 2022) is a task to
identify the target entities and the relation jointly;
5) Event Detection(ED) (Veyseh et al., 2021; Lou
et al., 2021; Liu et al., 2022a; Zhao et al., 2022)
identifies event triggers and their types; 6) Event
Argument Extraction(EAE) (Zhang et al., 2022b;
Du and Ji, 2022; Ma et al., 2022) distinguishes argu-
ments and categorizes their roles with respect to the
targe event; and 7) Event Extraction(EE) (Wad-
den et al., 2019; Du and Cardie, 2020; Liu et al.,
2022b; Hsu et al., 2022) performs event detection
and argument extraction jointly. Note that although
some of these tasks are subsets of others, every
task needs LLMs’ unique ability to perform well.
It is worth to explore the performances on these
fine-grained IE tasks.
3.2 Standard-IE Setting and OpenIE Setting
To comprehensively evaluate the overall perfor-
mance of ChatGPT on IE tasks, we ask ChatGPT
to generate the responses from the Standard-IE
setting and the OpenIE setting. The Standard-IE
setting is commonly used in previous works, which
uses the task-specific dataset with supervised learn-
ing paradigm to fine-tune a model. For ChatGPT,
as we can not directly fine-tune the parameters, we
evaluate the ChatGPT’s ability to select the most
appropriate answer from a set of candidate labels
instead. Specifically, this setting is based on an
instruction that includes the task description, the
input text, the prompt, and the label set. Where
the task description describes the specific IE task,
the prompt involves the utterances that guide the
ChatGPT outputs the required keys (which will be
introduced in § 3.3), and the label set contains all
candidate labels based on each dataset. The Ope-
nIE setting is a more advanced and challenging
scenario than Standard-IE setting. In this setting,
we do not provide any candidate labels to ChatGPT
and rely solely on its ability to comprehend the task
description, the prompt, and input text to generate
predictions. Our goal is to assess the ChatGPT’s
ability to produce reasonable factual knowledge.
 Keys Explanation
Performance
Open Directly ask ChatGPT to predict the class without the label set.
Standard ChatGPT’s most likely correct class with a given label set.
Top3 The three most likely classes of the given label set from ChatGPT.
Top5 The five most likely classes of the given label set from ChatGPT.
ifOpen_Correct(Manual) Manually annotate whether the "Open" is reasonable.
Explainability
Reason_Open The reason why ChatGPT chooses the class in "Open".
Reason_Standard The reason why ChatGPT chooses the class in "Standard".
ifR_Open Does ChatGPT think that "Reason_Open" is reasonable?
ifR_Standard Does ChatGPT think that "Reason_Standard" is reasonable?
ifR_Open(Manual) Manually annotate whether the "Reason_Open" is reasonable.
ifR_Standard(Manual) Manually annotate whether the "Reason_Standard" is reasonable.
Calibration
Confidence_Open The confidence of ChatGPT in predicting "Open".
Confidence_Standard The confidence of ChatGPT in predicting "Standard".
Faithfulness
FicR_Open(Manual) Manually annotate whether the "Reason_Open" is fictitious.
FicR_Standard(Manual) Manually annotate whether the "Reason_Standard" is fictitious.

Table 1: We gather 15 keys in this research, consisting of 10 keys automatically generated by ChatGPT and 5 keys
that required manual annotation (denoted as Manual). These keys provide insight into ChatGPT’s ability in four
dimensions, namely: 1) performance, 2) explainability, 3) calibration, and 4) faithfulness.

3.3 Collecting Keys From ChatGPT And
Human Annotation
In this subsection, we first describe 15 keys that are
collected from the ChatGPT and domain experts.
In Table 1, we show 10 keys that are extracted
from ChatGPT and 5 keys that involves human in-
volvements. These keys could systemically assess
ChatGPT’s ability from the following four aspects:
Performance. One important aspect of our re-
search is to comprehensively evaluate the overall
performance of ChatGPT on various tasks and com-
pare it with other popular models. By examining
its performance from different aspects, we seek
to provide a detailed understanding of ChatGPT’s
capability on the downstream IE tasks.
Explainability. The explainability of ChatGPT
is crucial for its application in real-world sce-
narios (Rajani et al., 2019; Aghajanyan et al.,
2021; Zini and Awad, 2023). In our study,
we will measure both the self-check and human-
check explainability of ChatGPT, with a fo-
cus on its ability to provide useful and accu-
rate explanations of its reasoning process for
humans. Specially, we ask ChatGPT to pro-
vide reasons for its predictions (Reason_Open
and Reason_Standard), and whether Chat-
GPT approves its explanations (ifR_Open and
ifR_Standard). Additionally, we also man-
ually evaluate the acceptability of these rea-
sons to humans (ifR_Open(Manual) and
ifR_Standard(Manual)).
Calibration. Measuring the calibration helps to
evaluate the predictive uncertainty of a model (Guo
et al., 2017; Kumar et al., 2019). A properly
calibrated classifier should have predictive scores
that accurately reflect the probability of correct-
ness (Thulasidasan et al., 2019; Minderer et al.,
2021). Given the tendency of modern neural net-
works to be overconfident in their predictions, we
aim to identify potential uncertainties or overcon-
fidence phenomenon of ChatGPT. To evaluate the
calibration, ChatGPT is required to provide a con-
fidence score (ranging from 1 to 100) for each
prediction it makes (Confidence_Open and
Confidence_Standard).
Faithfulness. The faithfulness of ChatGPT’s
explanation is important to ensure its trustwor-
thiness (Maynez et al., 2020; He et al., 2023).
In evaluating the faithfulness, we have included
two keys that assess whether the reasons pro-
vided by ChatGPT are faithful to the original in-
put. These keys, FicR_Open(Manual) and
FicR_Standard(Manual), require manual
annotation by domain experts.
Due to the space limitation, we show an intuitive
example in the Appendix A.3 to help readers better
understand the annotation process.
4 Performance
4.1 Setup
To ensure a comprehensive evaluation of Chat-
GPT’s capabilities, we conduct manual annotation
and analysis on a diverse range of IE tasks, in-
cluding 7 fine-grained tasks spanning 14 datasets.
We collected 15 keys for each dataset from both
ChatGPT and domain experts (§ 3). Only the test
sets are annotated, as our aim is to analysis Chat-
GPT’s abilities without any training. For space
reasons, the detail of each dataset is shown in the
Appendix A.1. Due to the time-consuming na-
ture of obtaining responses from domain experts,
we randomly select nearly 3,000 samples in total
for our analysis. The number of manually anno-
tated samples for each dataset is reported in the
Appendix A.1. As for the outputs from ChatGPT,
we use the official API to evaluate the whole test
sets. 4
Besides, we compare ChatGPT with several pop-
ular baselines: 1) BERT (Devlin et al., 2019) and
RoBERTa (Liu et al., 2019), and 2) State-of-the-
Art (SOTA) on the single dataset. Due to the space
limitation, the details of state-of-the-art methods
are shown in the Appendix A.2. As for the metric,
we use Micro-F1 score for all tasks except RE and
EE. For the RE task, we report the named entity
recognition F1-score and the relation classification
F1-score. As for the EE task, we show the trigger
F1-score and argument F1-score. 5
4.2 Performance on the Standard-IE Setting
In this subsection, we report the performances of
different models on the Standard-IE setting, as de-
picted in Table 2. It is clear from the table that
ChatGPT’s performance is not comparable to
that of baseline models and SOTA methods in
most cases. This is not surprising given that di-
rectly asking ChatGPT for the prediction is more
like a zero-shot scenario, whereas the other com-
pared methods are trained on task-specific datasets
4
To prevent any historical chat biases, we cleared every
conversation after generating each response.
5
These metrics are all following previous works.
under a supervised learning paradigm. Another
reason may be ChatGPT directly choose an answer
from the given label set, and some labels are not
easy to understand, thereby negatively impact the
performance.
Moreover, our research indicates that ChatGPT
performs well on relatively simple IE tasks but
struggles with more complex and challenging
tasks. For example, the entity typing (ET) task
only involves classifying entities into pre-defined
types without any further contextual analysis, and
ChatGPT excels at this task, demonstrating that
the model can generate accurate factual knowledge
when the task is simple. However, in complex and
challenging IE tasks such as RE, ChatGPT strug-
gles as it requires to first identify the entities that
exist in the input and then classify the relationship
between them, which is a more challenging task
than ET. Despite ChatGPT’s acceptable results on
the ET, NER and RC tasks, it still faces challenges
with more multifaceted IE tasks like RE and EE,
where deeper contextual analysis and reasoning
abilities are required. In summary, ChatGPT’s per-
formance varies based on the complexity of the
task, and it performs well on straightforward tasks.
Furthermore, the conclusion that ChatGPT per-
forms worse than other models seems inconsistent
with previous studies (Wei et al., 2023; Gao et al.,
2023), which suggest that ChatGPT can achieve
desirable performance in some IE tasks. One pos-
sible explanation for the difference in conclusions
is that we report the performance of the entire test
set for each task in our study, while prior studies
reported on a very small set of test samples drawn
at random, which may have substantial variance.
Another factor may be that we used a concise and
relatively unified prompt to guide ChatGPT, while
other research relied on domain-specific prompts
or included a large number of label descriptions in
their prompts, which needs lots of domain knowl-
edge and thereby limits the ability to generalize
across various tasks.
4.3 Performance on the OpenIE Setting
In this subsection, we report both the accuracy of
Standard-IE setting and OpenIE setting on the sam-
pled dataset. 6 For the Standard-IE setting, we pro-
vide the pre-defined label set and ask the ChatGPT
to choose an answer for a given input, and the ac-
6
We randomly selected around 200 samples for each
dataset.
 Task Dataset BERT RoBERTa SOTA ChatGPT
Entity BBN 80.3 79.8 82.2 (Zuo et al., 2022) 85.6
Typing(ET) OntoNotes 5.0 69.1 68.8 72.1 (Zuo et al., 2022) 73.4
Named Entity CoNLL2003 92.8 92.4 94.6 (Wang et al., 2021) 67.2
Recognition(NER) OntoNotes 5.0 89.2 90.9 91.9 (Ye et al., 2022) 51.1
Relation TACRED 72.7 74.6 75.6 (Li et al., 2022a) 20.3
Classification(RC) SemEval2010 89.1 89.8 91.3 (Zhao et al., 2021) 42.5
Relation ACE05-R 87.5 | 63.7 88.2 | 65.1 91.1 | 73.0 (Ye et al., 2022) 40.5 | 4.5
Extraction(RE) SciERC 65.4 | 43.0 63.6 | 42.0 69.9 | 53.2 (Ye et al., 2022) 25.9 | 5.5
Event ACE05-E 71.8 72.9 75.8 (Liu et al., 2022a) 17.1
Detection(ED) ACE05-E+ 72.4 72.1 72.8 (Lin et al., 2020) 15.5
Event Argument ACE05-E 65.3 68.0 73.5 (Hsu et al., 2022) 28.9
Extraction(EAE) ACE05-E+ 64.0 66.5 73.0 (Hsu et al., 2022) 30.9
Event ACE05-E 71.8 | 51.0 72.9 | 51.9 74.7 | 56.8 (Lin et al., 2020) 17.0 | 7.3
Extraction(EE) ACE05-E+ 72.4 | 52.7 72.1 | 53.4 71.7 | 56.8 (Hsu et al., 2022) 16.6 | 7.8

Table 2: The performances of ChatGPT and several baseline models on 14 IE datasets on the Standard-IE setting.
We report the performance on the whole test set. All results are directly cited from public papers or re-implemented
using official open-source code.

Standard-IE OpenIE sonable by the domain experts in ET, NER, and

BBN(ET) 86.8% 97.2% RC tasks. However, the performance is relatively
CoNLL(NER) 69.0% 93.3% poorer for more challenging tasks, such as RE and
SemEval2010(RC) 43.3% 84.3% EE. Overall, compared with Standard-IE setting,
ACE05-R(RE) 14.9% 23.9% ChatGPT’s performance on the OpenIE setting is
ACE05-E(ED) 12.4% 42.6% exciting. Our findings suggest that under the Ope-
ACE05-E(EAE) 17.3% 65.3% nIE setting, ChatGPT could generate reliable fac-
ACE05-E(EE) 4.9% 28.8% tual knowledge and reasonable output.

4.4 The top-k Recall Analysis

Table 3: The accuracy of Standard-IE setting and Ope-
nIE setting on the sampled test set. Our results show
that ChatGPT could generate reasonable outputs on the
OpenIE setting.
curacy was calculated by matching the predictions
to the ground truth labels. On the other hand, the
OpenIE setting refers to asking ChatGPT to make
predictions without the pre-defined label set (Open
in § 3.3). Three domain experts evaluate these pre-
dictions and vote on whether they were reasonable
in light of the input and background knowledge,
named as ifOpen_Correct in § 3.3. Our main
goal is to determine if ChatGPT could produce
logical and reasonable predictions without given
the pre-defined label set, so we do not require the
prediction to match with the ground truth.
The results presented in Table 3 indicate that
ChatGPT’s performance is somewhat inspiring
under the OpenIE setting. For example, more
than 84% of the predictions are considered rea-
While generating the most likely prediction may
be unsatisfactory on the Standard-IE setting, we
seek to investigate whether ChatGPT could be a
useful advisor. Therefore, we examine the recall
of its top-k predictions, with k = 1, 3, or 5. As
shown in Table 5, the results indicate that compared
with the top-1 recall, the top-3 recall increases sig-
nificantly, e.g., the improvement is 19.6% on Se-
mEval2010. Moreover, the top-5 recall reaches
an impressive 94.9% on BBN and 76.0% on Se-
mEval2010, demonstrating a favorable outcome.
Our findings suggest that ChatGPT is a com-
petent answer candidate generator for a given
task under the Standard-IE setting, which could
help users select the most probable prediction from
the top-5 predictions.
5 Explainability, Calibration and
Faithfulness
While ChatGPT’s performance in evaluations is
noteworthy, it is equally important to evaluate its
 Stardand Setting OpenIE Setting
Self-check Human-check Overlap Self-check Human-check Overlap
BBN (ET) 100.0% 99.2% 99.2% 100.0% 99.5% 99.5%
CoNLL (NER) 100.0% 99.3% 99.3% 100.0% 99.7% 99.7%
SemEval (RC) 100.0% 100.0% 100.0% 100.0% 99.7% 99.7%
ACE05-R (RE) 100.0% 90.0% 90.0% 100.0% 100.0% 100.0%
ACE05-E (ED) 100.0% 96.3% 96.3% 100.0% 90.2% 90.2%
ACE05-E (EAE) 100.0% 74.1% 74.1% 100.0% 90.4% 90.4%
ACE05-E (EE) 100.0% 47.1% 47.1% 94.0% 78.0% 74.0%

Table 4: The explainability of ChatGPT measured on the sampled test set. We report the ratio of samples with
reasonable reasons discriminated by ChatGPT (self-check) and domain experts (human-check) under different
settings. Besides, we also compute the overlap ration for both of them. These results indicate that in most cases,
ChatGPT exhibits strong explainability for its prediction.

top-1 top-3 top-5 predictions in the Standard-IE setting to ensure

BBN 85.6% 92.7% 94.9% (+9.3%)
SemEval2010 42.5% 62.1% 76.0% (+33.5%)
Table 5: The top-k recall analysis on the whole test set.
We report two datasets due to the space limitation, other
datasets show similar observation. The results show
that ChatGPT could server as a good advisor.
ability from diverse dimensions that could offer
important insights for future research directions. In
this section, we analyze several relevant factors, in-
cluding explainability, calibration, and faithfulness,
to comprehensively evaluate ChatGPT’s abilities.
Overall, our findings suggest that ChatGPT can pro-
vide high-quality and reliable explanations for its
predictions, but it tends to display overconfidence
in most cases, leading to low calibration. Addi-
tionally, ChatGPT displays high faithfulness to the
original text, making it an reliable tool for users.
5.1 Explainability
Explainability is a critical requirement for LLMs,
as it allows users to understand how the model ar-
rives at its predictions (Peng et al., 2023). In this
study, we investigate whether ChatGPT could pro-
vide a reasonable explanation for its output. To
be specific, we request ChatGPT to provide rea-
sons for its predictions in the Standard-IE and Ope-
nIE settings. The corresponding keys are denoted
as Reason_Standard and Reason_Open, as
explained in § 3.3. These reasons are then evalu-
ated for their reasonableness by both ChatGPT and
three domain experts, with the resulting evaluations
referred to as self-check and human-check, respec-
tively. We only consider the samples with correct
a robust evaluation of ChatGPT’s explainability
ability. 7 This is because evaluating the reasons
provided by ChatGPT for incorrect predictions is
less valuable.
The ratio of samples with reasonable explana-
tions (termed as reasonable score) is summarized
in Table 4, from which we can derive the following
conclusions. Firstly, both ChatGPT and domain
experts highly approve of the reasons given by
ChatGPT, with the majority of datasets achieving
a reasonable score of over 90% in the Standard-IE
and OpenIE settings. The above results demon-
strate that ChatGPT gives very high-quality expla-
nation for its prediction. Secondly, we observe that
ChatGPT displays a high level of confidence in
the reasons provided for its predictions when
compared with human evaluation. In fact, Chat-
GPT achieves nearly a 100% reasonable score
among almost all datasets. This suggests that
ChatGPT is very confident in its ability to pro-
vide reasonable explanations. Thirdly, we find that
when ChatGPT provides a reasonable explana-
tion for a prediction, there is a high level of
agreement between ChatGPT and human eval-
uations. This suggests that ChatGPT may have a
similar understanding of explanations as humans.
Overall, our findings suggest that ChatGPT is ca-
pable of providing high-quality and reliable expla-
nations for its predictions. This is a crucial step
towards developing trustworthy and reliable LLMs.
7
We randomly select around 200 samples from each
dataset for human annotation.
 Correct Confidence Incorrect Confidence
BERT RoBERTa ChatGPT BERT RoBERTa ChatGPT
BBN(ET) 0.971 0.968 0.888 0.904 0.885 0.828
CoNLL(NER) 0.990 0.991 0.864 0.866 0.886 0.785
SemEval(RC) 0.983 0.989 0.868 0.871 0.852 0.839
ACE05-R(RE) 0.995 0.991 0.760 0.883 0.810 0.764
ACE05-E(ED) 0.882 0.944 0.852 0.770 0.871 0.737
ACE05-E(EAE) 0.762 0.785 0.956 0.525 0.555 0.910
ACE05-E(EE) 0.763 0.782 0.845 0.612 0.628 0.764

Table 6: The prediction confidence of various models on the whole test set. We show both the correct confidence
and incorrect confidence based on various methods. We find that ChatGPT is overconfidence for its prediction in
most cases.

5.2 Calibration BERT RoBERTa ChatGPT

In this subsection, we first investigate the level of
confidence for both the correct and incorrect sam-
ples. Confidence is typically described in terms of
a probability value, indicating the likelihood of be-
longing to a specific category. To obtain prediction
probabilities from ChatGPT, we ask it to output
the probability (Confidence_Standard and
Confidence_Open), as discussed in § 3.3. Our
aim is to investigate whether ChatGPT can provide
a reasonable prediction confidence scores for its
predictions, thus reducing the risk of misinterpreta-
tion. In Table 6, we present the confidence scores
of correct and incorrect predictions from different
models, referred to correct confidence and incor-
rect confidence, respectively. Our observations re-
veal that all the models exhibit high confidence lev-
els in their predictions, this is consistent with previ-
ous research on large models (Guo et al., 2017).
Although ChatGPT performs worse than its
BERT-based counterparts in Standard-IE set-
ting, it displays overconfidence in both correct
and incorrect predictions. Consequently, this
overconfidence could lead to misguidance of users.
Furthermore, we note a significant confidence gap
between correct and incorrect predictions, indicat-
ing the need for careful evaluation when ChatGPT’s
prediction has relatively low confidence.
We then focus on calibration, a critical property
of LLMs as it could estimate the predictive un-
certainty for the secure application of LLMs. A
well-calibrated model not only produces accurate
predictions but also provides reliable and infor-
mative uncertainty estimates, necessary for sound
decision-making. In this research, we evaluate the
calibration using the Expected Calibration Error
(ECE) metric which measures the deviation be-
BBN(ET) 0.012 0.012 0.026
CoNLL(NER) 0.052 0.044 0.204
SemEval(RC) 0.023 0.031 0.460
ACE05-R(RE) 0.020 0.014 0.745
ACE05-E(ED) 0.161 0.226 0.656
ACE05-E(EAE) 0.154 0.168 0.699
ACE05-E(EE) 0.211 0.288 0.699
Table 7: The expected calibration error (ECE) is used
to measure the calibration of a given model, and the
lower, the better. Results are calculated on the whole
test set.
tween predicted confidence and accuracy. 8 The
results are shown in Table 7, and from that we
can observe that ChatGPT shows much poorer cali-
bration compared to BERT-based methods, which
indicates that ChatGPT tends to produce confi-
dences that do not represent true probabilities
easily. Furthermore, although ChatGPT displays
low ECE in tasks such as ET and NER, miscali-
bration phenomenon dominates most cases. These
findings suggest that ChatGPT needs improvement
in terms of calibration, especially for IE tasks.
5.3 Faithfulness
Recent works show that ChatGPT may provide
false information to users, potentially affecting
their decision-making (Huang et al., 2023). There-
fore, assessing the faithfulness of the ChatGPT
model to the original text is a crucial measure-
ment in developing a trustworthy information ex-
traction model. Our study uses faithfulness as a
metric to evaluate the ChatGPT model, specifi-
8
We set the bin size to 50, dividing the prediction proba-
bilities into 50 equally spaced bins for analysis.
 Stardand-IE OpenIE
BBN(ET) 98.3% 99.3%
CoNLL(NER) 100.0% 98.7%
SemEval(RC) 100.0% 99.1%
ACE05-R(RE) 90.0% 93.8%
ACE05-E(ED) 100.0% 100.0%
ACE05-E(EAE) 100.0% 96.5%
ACE05-E(EE) 100.0% 97.0%
Table 8: The evaluation of faithfulness for ChatGPT.
Faithfulness refers to whether ChatGPT’s explanation
align with the original text. Experimental results show
that ChatGPT’s explanation maintains a very high de-
gree of faithfulness to the original text and provide
nearly no false explanation.
cally referring to if the explanation provided by
ChatGPT aligns with the original text when its
predictionis correct, as the original text is the
most important source for extraction information.
There are two keys we collect by domain ex-
perts, namely FicR_Standard(Manual) and
FicR_Open(Manual), as we mentioned in
§ 3.3. Our results are shown in Table 8, which
indicate a high degree of faithfulness between Chat-
GPT’s explanations and the original text with rare
false explanations, i.e., with over 95% of samples
considered faithful in nearly all datasets under dif-
ferent settings. We can conclude that ChatGPT’s
decision-making process primarily relies on the
input of the original text, leading to the majority
of its explanations being regarded as truthful and
reliable.
6 Conclusion
In this paper, we propose to systematically analysis
the ChatGPT’s performance, explainability, cali-
bration, and faithfulness. To be specific, based on
7 fine-grained information extraction tasks among
14 datasets, we collect 15 keys identified by ei-
ther ChatGPT or domain experts for our research.
Our findings reveal that ChatGPT’s performance
in Standard-IE settings is not as good as BERT-
based models in most cases. However, we found
that ChatGPT achieved excellent accuracy scores
in the OpenIE setting, as evaluated by human an-
notators. Furthermore, ChatGPT could provide
high-quality and trustworthy explanations for its
predictions. One of the key issues that we identified
is its tendency towards overconfidence, resulting
in low calibration. Furthermore, our analysis also
showed that ChatGPT exhibits a high level of faith-
fulness to the original text, indicating that its pre-
dictions are grounded in the input text. Given these
findings, we hope that our research could inspire
more research on using ChatGPT for information
extraction.
References
Armen Aghajanyan, Sonal Gupta, and Luke Zettle-
moyer. 2021. Intrinsic dimensionality explains the
effectiveness of language model fine-tuning. In Pro-
ceedings of the 59th Annual Meeting of the Associa-
tion for Computational Linguistics and the 11th In-
ternational Joint Conference on Natural Language
Processing, ACL/IJCNLP 2021, (Volume 1: Long
Papers), Virtual Event, August 1-6, 2021, pages
7319–7328. Association for Computational Linguis-
tics.
Rachith Aiyappa, Jisun An, Haewoon Kwak, and Yong-
Yeol Ahn. 2023. Can we trust the evaluation on chat-
gpt? arXiv preprint arXiv:2303.12767.
Peggy M. Andersen, Philip J. Hayes, Steven P. Wein-
stein, Alison K. Huettner, Linda M. Schmandt, and
Irene B. Nirenburg. 1992. Automatic extraction of
facts from press releases to generate news stories.
In 3rd Applied Natural Language Processing Con-
ference, ANLP 1992, Trento, Italy, March 31 - April
3, 1992, pages 170–177. ACL.
Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor
Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin,
Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru,
Giridharan Anantharaman, Xian Li, Shuohui Chen,
Halil Akin, Mandeep Baines, Louis Martin, Xing
Zhou, Punit Singh Koura, Brian O’Horo, Jeffrey
Wang, Luke Zettlemoyer, Mona T. Diab, Zornitsa
Kozareva, and Veselin Stoyanov. 2022. Efficient
large scale language modeling with mixtures of ex-
perts. In Proceedings of the 2022 Conference on
Empirical Methods in Natural Language Processing,
EMNLP 2022, Abu Dhabi, United Arab Emirates,
December 7-11, 2022, pages 11699–11732. Associ-
ation for Computational Linguistics.
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wen-
liang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Zi-
wei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do,
Yan Xu, and Pascale Fung. 2023. A multitask,
multilingual, multimodal evaluation of chatgpt on
reasoning, hallucination, and interactivity. CoRR,
abs/2302.04023.
Zeljana Basic, Ana Banovac, Ivana Kruzic, and Ivan
Jerkovic. 2023. Better by you, better than me, chat-
gpt3 as writing assistance in students essays. CoRR,
abs/2302.04536.
Junyi Bian, Li Huang, Xiaodi Huang, Hong Zhou, and
Shanfeng Zhu. 2021. Grantrel: Grant information
extraction via joint entity and relation extraction. In
 Findings of the Association for Computational Lin-
guistics: ACL/IJCNLP 2021, Online Event, August
1-6, 2021, volume ACL/IJCNLP 2021 of Findings
of ACL, pages 2674–2685. Association for Compu-
tational Linguistics.
Ning Bian, Xianpei Han, Le Sun, Hongyu Lin, Yaojie
Lu, and Ben He. 2023. Chatgpt is a knowledgeable
but inexperienced solver: An investigation of com-
monsense problem in large language models. arXiv
preprint arXiv:2303.16421.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie
Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind
Neelakantan, Pranav Shyam, Girish Sastry, Amanda
Askell, Sandhini Agarwal, Ariel Herbert-Voss,
Gretchen Krueger, Tom Henighan, Rewon Child,
Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu,
Clemens Winter, Christopher Hesse, Mark Chen,
Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin
Chess, Jack Clark, Christopher Berner, Sam Mc-
Candlish, Alec Radford, Ilya Sutskever, and Dario
Amodei. 2020. Language models are few-shot learn-
ers. In Advances in Neural Information Processing
Systems 33: Annual Conference on Neural Informa-
tion Processing Systems 2020, NeurIPS 2020, De-
cember 6-12, 2020, virtual.
Yi Chen, Jiayang Cheng, Haiyun Jiang, Lemao Liu,
Haisong Zhang, Shuming Shi, and Ruifeng Xu.
2022. Learning from sibling mentions with scal-
able graph inference in fine-grained entity typing. In
Proceedings of the 60th Annual Meeting of the As-
sociation for Computational Linguistics (Volume 1:
Long Papers), ACL 2022, Dublin, Ireland, May 22-
27, 2022, pages 2076–2087. Association for Compu-
tational Linguistics.
Hai Leong Chieu, Hwee Tou Ng, and Yoong Keok Lee.
2003. Closing the gap: Learning-based information
extraction rivaling knowledge-engineering methods.
In Proceedings of the 41st Annual Meeting of the As-
sociation for Computational Linguistics, pages 216–
223, Sapporo, Japan. Association for Computational
Linguistics.
Eunsol Choi, Omer Levy, Yejin Choi, and Luke Zettle-
moyer. 2018. Ultra-fine entity typing. In Proceed-
ings of the 56th Annual Meeting of the Associa-
tion for Computational Linguistics, ACL 2018, Mel-
bourne, Australia, July 15-20, 2018, Volume 1: Long
Papers, pages 87–96. Association for Computational
Linguistics.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin,
Maarten Bosma, Gaurav Mishra, Adam Roberts,
Paul Barham, Hyung Won Chung, Charles Sutton,
Sebastian Gehrmann, Parker Schuh, Kensen Shi,
Sasha Tsvyashchenko, Joshua Maynez, Abhishek
Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin-
odkumar Prabhakaran, Emily Reif, Nan Du, Ben
Hutchinson, Reiner Pope, James Bradbury, Jacob
Austin, Michael Isard, Guy Gur-Ari, Pengcheng
Yin, Toju Duke, Anselm Levskaya, Sanjay Ghe-
mawat, Sunipa Dev, Henryk Michalewski, Xavier
Garcia, Vedant Misra, Kevin Robinson, Liam Fe-
dus, Denny Zhou, Daphne Ippolito, David Luan,
Hyeontaek Lim, Barret Zoph, Alexander Spiridonov,
Ryan Sepassi, David Dohan, Shivani Agrawal, Mark
Omernick, Andrew M. Dai, Thanumalayan Sankara-
narayana Pillai, Marie Pellat, Aitor Lewkowycz,
Erica Moreira, Rewon Child, Oleksandr Polozov,
Katherine Lee, Zongwei Zhou, Xuezhi Wang, Bren-
nan Saeta, Mark Diaz, Orhan Firat, Michele Catasta,
Jason Wei, Kathy Meier-Hellstern, Douglas Eck,
Jeff Dean, Slav Petrov, and Noah Fiedel. 2022.
Palm: Scaling language modeling with pathways.
CoRR, abs/2204.02311.
Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan
Martic, Shane Legg, and Dario Amodei. 2017. Deep
reinforcement learning from human preferences. In
Advances in Neural Information Processing Systems
30: Annual Conference on Neural Information Pro-
cessing Systems 2017, December 4-9, 2017, Long
Beach, CA, USA, pages 4299–4307.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar-
ret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi
Wang, Mostafa Dehghani, Siddhartha Brahma, et al.
2022. Scaling instruction-finetuned language mod-
els. arXiv preprint arXiv:2210.11416.
Antonia Creswell and Murray Shanahan. 2022. Faith-
ful reasoning using large language models. CoRR,
abs/2208.14271.
Jeremy Crowe. 1995. Constraint-based event recog-
nition for information extraction. In 33rd Annual
Meeting of the Association for Computational Lin-
guistics, pages 296–298, Cambridge, Massachusetts,
USA. Association for Computational Linguistics.
Hongliang Dai, Yangqiu Song, and Haixun Wang.
2021. Ultra-fine entity typing with weak supervi-
sion from a masked language model. In Proceed-
ings of the 59th Annual Meeting of the Association
for Computational Linguistics and the 11th Interna-
tional Joint Conference on Natural Language Pro-
cessing, ACL/IJCNLP 2021, (Volume 1: Long Pa-
pers), Virtual Event, August 1-6, 2021, pages 1790–
1799. Association for Computational Linguistics.
Sarkar Snigdha Sarathi Das, Arzoo Katiyar, Rebecca J.
Passonneau, and Rui Zhang. 2022. Container: Few-
shot named entity recognition via contrastive learn-
ing. In Proceedings of the 60th Annual Meeting of
the Association for Computational Linguistics (Vol-
ume 1: Long Papers), ACL 2022, Dublin, Ireland,
May 22-27, 2022, pages 6338–6353. Association for
Computational Linguistics.
Ameet Deshpande, Vishvak Murahari, Tanmay Ra-
jpurohit, Ashwin Kalyan, and Karthik Narasimhan.
2023. Toxicity in chatgpt: Analyzing persona-
assigned language models. arXiv preprint
arXiv:2304.05335.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and
Kristina Toutanova. 2019. BERT: pre-training of
 deep bidirectional transformers for language under-
standing. In NAACL-HLT 2019, pages 4171–4186.
Association for Computational Linguistics.
Xinya Du and Claire Cardie. 2020. Event extraction by
answering (almost) natural questions. In Proceed-
ings of the 2020 Conference on Empirical Methods
in Natural Language Processing, EMNLP 2020, On-
line, November 16-20, 2020, pages 671–683. Asso-
ciation for Computational Linguistics.
Xinya Du and Heng Ji. 2022. Retrieval-augmented gen-
erative question answering for event argument ex-
traction. In Proceedings of the 2022 Conference on
Empirical Methods in Natural Language Processing,
EMNLP 2022, Abu Dhabi, United Arab Emirates,
December 7-11, 2022, pages 4649–4666. Associa-
tion for Computational Linguistics.
Simon Frieder, Luca Pinchetti, Ryan-Rhys Grif-
fiths, Tommaso Salvatori, Thomas Lukasiewicz,
Philipp Christian Petersen, Alexis Chevalier, and
Julius Berner. 2023. Mathematical capabilities of
chatgpt. CoRR, abs/2301.13867.
Tsu-Jui Fu, Peng-Hsuan Li, and Wei-Yun Ma. 2019.
Graphrel: Modeling text as relational graphs for
joint entity and relation extraction. In Proceedings
of the 57th Conference of the Association for Compu-
tational Linguistics, ACL 2019, Florence, Italy, July
28- August 2, 2019, Volume 1: Long Papers, pages
1409–1418. Association for Computational Linguis-
tics.
Yao Fu, Hao Peng, and Tushar Khot. 2022. How does
gpt obtain its ability? tracing emergent abilities of
language models to their sources. Yao Fu’s Notion.
Jun Gao, Huan Zhao, Changlong Yu, and Ruifeng Xu.
2023. Exploring the feasibility of chatgpt for event
extraction. CoRR, abs/2303.03836.
Dan Gillick, Nevena Lazic, Kuzman Ganchev, Jesse
Kirchner, and David Huynh. 2014. Context-
dependent fine-grained entity type tagging. CoRR,
abs/1412.1820.
Andrej Zukov Gregoric, Yoram Bachrach, and Sam
Coope. 2018. Named entity recognition with par-
allel recurrent neural networks. In Proceedings of
the 56th Annual Meeting of the Association for Com-
putational Linguistics, ACL 2018, Melbourne, Aus-
tralia, July 15-20, 2018, Volume 2: Short Papers,
pages 69–74. Association for Computational Lin-
guistics.
Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang,
Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng
Wu. 2023. How close is chatgpt to human ex-
perts? comparison corpus, evaluation, and detection.
CoRR, abs/2301.07597.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q.
Weinberger. 2017. On calibration of modern neu-
ral networks. In Proceedings of the 34th Inter-
national Conference on Machine Learning, ICML
2017, Sydney, NSW, Australia, 6-11 August 2017,
volume 70 of Proceedings of Machine Learning Re-
search, pages 1321–1330. PMLR.
Mubin Ul Haque, Isuru Dharmadasa, Zarrin Tasnim
Sworna, Roshan Namal Rajapakse, and Hussain Ah-
mad. 2022. "i think this is the most disruptive
technology": Exploring sentiments of chatgpt early
adopters using twitter data. CoRR, abs/2212.05856.
Hangfeng He, Hongming Zhang, and Dan Roth. 2023.
Rethinking with retrieval: Faithful large language
model inference. CoRR, abs/2301.00303.
Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva,
Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian
Padó, Marco Pennacchiotti, Lorenza Romano, and
Stan Szpakowicz. 2010. Semeval-2010 task 8:
Multi-way classification of semantic relations be-
tween pairs of nominals. In Proceedings of the
5th International Workshop on Semantic Evaluation,
SemEval@ACL 2010, Uppsala University, Uppsala,
Sweden, July 15-16, 2010, pages 33–38. The Associ-
ation for Computer Linguistics.
I-Hung Hsu, Kuan-Hao Huang, Elizabeth Boschee,
Scott Miller, Prem Natarajan, Kai-Wei Chang, and
Nanyun Peng. 2022. DEGREE: A data-efficient
generation-based event extraction model. In Pro-
ceedings of the 2022 Conference of the North
American Chapter of the Association for Computa-
tional Linguistics: Human Language Technologies,
NAACL 2022, Seattle, WA, United States, July 10-
15, 2022, pages 1890–1908. Association for Compu-
tational Linguistics.
Fan Huang, Haewoon Kwak, and Jisun An. 2023. Is
chatgpt better than human annotators? potential
and limitations of chatgpt in explaining implicit hate
speech. CoRR, abs/2302.07736.
Katharina Jeblick, Balthasar Schachtner, Jakob Dexl,
Andreas Mittermeier, Anna Theresa Stüber, Johanna
Topalis, Tobias Weber, Philipp Wesp, Bastian Sabel,
Jens Ricke, and Michael Ingrisch. 2022. Chatgpt
makes medicine easy to swallow: An exploratory
case study on simplified radiology reports. CoRR,
abs/2212.14882.
Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing
Wang, and Zhaopeng Tu. 2023. Is chatgpt A
good translator? A preliminary study. CoRR,
abs/2301.08745.
Tushar Khot, Ashish Sabharwal, and Peter Clark. 2017.
Answering complex questions using open informa-
tion extraction. In Proceedings of the 55th Annual
Meeting of the Association for Computational Lin-
guistics (Volume 2: Short Papers), pages 311–316,
Vancouver, Canada. Association for Computational
Linguistics.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu-
taka Matsuo, and Yusuke Iwasawa. 2022. Large
language models are zero-shot reasoners. CoRR,
abs/2205.11916.
Gerd Kortemeyer. 2023. Could an artificial-
intelligence agent pass an introductory physics
course? arXiv preprint arXiv:2301.12127.
Fajri Koto, Jey Han Lau, and Timothy Baldwin. 2022.
Can pretrained language models generate persuasive,
faithful, and informative ad text for product descrip-
tions? In Proceedings of The Fifth Workshop on
e-Commerce and NLP (ECNLP 5), pages 234–243.
Sebastian Krügel, Andreas Ostermaier, and Matthias
Uhl. 2023. The moral authority of chatgpt. CoRR,
abs/2301.07098.
Ananya Kumar, Percy Liang, and Tengyu Ma. 2019.
Verified uncertainty calibration. In Advances in Neu-
ral Information Processing Systems 32: Annual Con-
ference on Neural Information Processing Systems
2019, NeurIPS 2019, December 8-14, 2019, Vancou-
ver, BC, Canada, pages 3787–3798.
Bo Li, Wei Ye, Jinglei Zhang, and Shikun Zhang.
2022a. Reviewing labels: Label graph network with
top-k prediction set for relation extraction. CoRR,
abs/2212.14270.
Bo Li, Dingyao Yu, Wei Ye, Jinglei Zhang, and
Shikun Zhang. 2022b. Sequence generation with
label augmentation for relation extraction. CoRR,
abs/2212.14266.
Xiaoya Li, Jingrong Feng, Yuxian Meng, Qinghong
Han, Fei Wu, and Jiwei Li. 2020. A unified MRC
framework for named entity recognition. In Pro-
ceedings of the 58th Annual Meeting of the Asso-
ciation for Computational Linguistics, pages 5849–
5859, Online. Association for Computational Lin-
guistics.
Xiaoya Li, Fan Yin, Zijun Sun, Xiayu Li, Arianna
Yuan, Duo Chai, Mingxin Zhou, and Jiwei Li. 2019.
Entity-relation extraction as multi-turn question an-
swering. In Proceedings of the 57th Conference of
the Association for Computational Linguistics, ACL
2019, Florence, Italy, July 28- August 2, 2019, Vol-
ume 1: Long Papers, pages 1340–1350. Association
for Computational Linguistics.
Ying Lin, Heng Ji, Fei Huang, and Lingfei Wu. 2020.
A joint neural model for information extraction with
global features. In Proceedings of the 58th Annual
Meeting of the Association for Computational Lin-
guistics, ACL 2020, Online, July 5-10, 2020, pages
7999–8009. Association for Computational Linguis-
tics.
Jian Liu, Yufeng Chen, and Jinan Xu. 2022a. Saliency
as evidence: Event detection with trigger saliency
attribution. In Proceedings of the 60th Annual Meet-
ing of the Association for Computational Linguistics
(Volume 1: Long Papers), ACL 2022, Dublin, Ire-
land, May 22-27, 2022, pages 4573–4585. Associa-
tion for Computational Linguistics.
Xiao Liu, Heyan Huang, Ge Shi, and Bo Wang.
2022b. Dynamic prefix-tuning for generative
template-based event extraction. In Proceedings
of the 60th Annual Meeting of the Association for
Computational Linguistics (Volume 1: Long Papers),
ACL 2022, Dublin, Ireland, May 22-27, 2022, pages
5216–5228. Association for Computational Linguis-
tics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man-
dar Joshi, Danqi Chen, Omer Levy, Mike Lewis,
Luke Zettlemoyer, and Veselin Stoyanov. 2019.
Roberta: A robustly optimized BERT pretraining ap-
proach. CoRR, abs/1907.11692.
Dongfang Lou, Zhilin Liao, Shumin Deng, Ningyu
Zhang, and Huajun Chen. 2021. Mlbinet: A cross-
sentence collective event detection network. In Pro-
ceedings of the 59th Annual Meeting of the Associa-
tion for Computational Linguistics and the 11th In-
ternational Joint Conference on Natural Language
Processing, ACL/IJCNLP 2021, (Volume 1: Long
Papers), Virtual Event, August 1-6, 2021, pages
4829–4839. Association for Computational Linguis-
tics.
Yaojie Lu, Qing Liu, Dai Dai, Xinyan Xiao, Hongyu
Lin, Xianpei Han, Le Sun, and Hua Wu. 2022. Uni-
fied structure generation for universal information
extraction. In Proceedings of the 60th Annual Meet-
ing of the Association for Computational Linguistics
(Volume 1: Long Papers), pages 5755–5772, Dublin,
Ireland. Association for Computational Linguistics.
Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh
Hajishirzi. 2018. Multi-task identification of enti-
ties, relations, and coreference for scientific knowl-
edge graph construction. In Proceedings of the 2018
Conference on Empirical Methods in Natural Lan-
guage Processing, Brussels, Belgium, October 31 -
November 4, 2018, pages 3219–3232. Association
for Computational Linguistics.
Yubo Ma, Zehao Wang, Yixin Cao, Mukai Li, Meiqi
Chen, Kun Wang, and Jing Shao. 2022. Prompt
for extraction? PAIE: prompting argument interac-
tion for event argument extraction. In Proceedings
of the 60th Annual Meeting of the Association for
Computational Linguistics (Volume 1: Long Papers),
ACL 2022, Dublin, Ireland, May 22-27, 2022, pages
6759–6774. Association for Computational Linguis-
tics.
Kyle Mahowald, Anna A. Ivanova, Idan Asher Blank,
Nancy Kanwisher, Joshua B. Tenenbaum, and
Evelina Fedorenko. 2023. Dissociating language
and thought in large language models: a cognitive
perspective. CoRR, abs/2301.06627.
Pedro Henrique Martins, Zita Marinho, and André F. T.
Martins. 2019. Joint learning of named entity recog-
nition and entity linking. In Proceedings of the 57th
Conference of the Association for Computational
Linguistics, ACL 2019, Florence, Italy, July 28 - Au-
gust 2, 2019, Volume 2: Student Research Workshop,
 pages 190–196. Association for Computational Lin-
guistics.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and
Ryan T. McDonald. 2020. On faithfulness and fac-
tuality in abstractive summarization. In Proceedings
of the 58th Annual Meeting of the Association for
Computational Linguistics, ACL 2020, Online, July
5-10, 2020, pages 1906–1919. Association for Com-
putational Linguistics.
Matthias Minderer, Josip Djolonga, Rob Romijnders,
Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin
Tran, and Mario Lucic. 2021. Revisiting the cali-
bration of modern neural networks. In Advances
in Neural Information Processing Systems 34: An-
nual Conference on Neural Information Processing
Systems 2021, NeurIPS 2021, December 6-14, 2021,
virtual, pages 15682–15694.
Sandra Mitrovic, Davide Andreoletti, and Omran Ay-
oub. 2023. Chatgpt or human? detect and explain.
explaining decisions of machine learning model
for detecting short chatgpt-generated text. CoRR,
abs/2301.13852.
Oded Nov, Nina Singh, and Devin M. Mann. 2023.
Putting chatgpt’s medical advice to the (turing) test.
CoRR, abs/2301.10035.
Miguel Ortega-Martín, Óscar García-Sierra, Alfonso
Ardoiz, Jorge Álvarez, Juan Carlos Armenteros, and
Adrián Alonso. 2023. Linguistic ambiguity analysis
in chatgpt. arXiv preprint arXiv:2302.06426.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car-
roll L. Wainwright, Pamela Mishkin, Chong Zhang,
Sandhini Agarwal, Katarina Slama, Alex Ray, John
Schulman, Jacob Hilton, Fraser Kelton, Luke Miller,
Maddie Simens, Amanda Askell, Peter Welinder,
Paul F. Christiano, Jan Leike, and Ryan Lowe.
2022. Training language models to follow instruc-
tions with human feedback. CoRR, abs/2203.02155.
Kunyuan Pang, Haoyu Zhang, Jie Zhou, and Ting
Wang. 2022. Divide and denoise: Learning from
noisy labels in fine-grained entity typing with
cluster-wise loss correction. In Proceedings of the
60th Annual Meeting of the Association for Compu-
tational Linguistics (Volume 1: Long Papers), ACL
2022, Dublin, Ireland, May 22-27, 2022, pages
1997–2006. Association for Computational Linguis-
tics.
Giovanni Paolini, Ben Athiwaratkun, Jason Krone,
Jie Ma, Alessandro Achille, Rishita Anubhai,
Cícero Nogueira dos Santos, Bing Xiang, and Ste-
fano Soatto. 2021. Structured prediction as transla-
tion between augmented natural languages. In ICLR
2021, Virtual Event, Austria, May 3-7, 2021. Open-
Review.net.
Baolin Peng, Michel Galley, Pengcheng He, Hao
Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars
Liden, Zhou Yu, Weizhu Chen, and Jianfeng Gao.
2023. Check your facts and try again: Improving
large language models with external knowledge and
automated feedback. CoRR, abs/2302.12813.
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao
Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is
chatgpt a general-purpose natural language process-
ing task solver? arXiv preprint arXiv:2302.06476.
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie
Millican, Jordan Hoffmann, H. Francis Song, John
Aslanides, Sarah Henderson, Roman Ring, Susan-
nah Young, Eliza Rutherford, Tom Hennigan, Ja-
cob Menick, Albin Cassirer, Richard Powell, George
van den Driessche, Lisa Anne Hendricks, Mari-
beth Rauh, Po-Sen Huang, Amelia Glaese, Jo-
hannes Welbl, Sumanth Dathathri, Saffron Huang,
Jonathan Uesato, John Mellor, Irina Higgins, An-
tonia Creswell, Nat McAleese, Amy Wu, Erich
Elsen, Siddhant M. Jayakumar, Elena Buchatskaya,
David Budden, Esme Sutherland, Karen Simonyan,
Michela Paganini, Laurent Sifre, Lena Martens,
Xiang Lorraine Li, Adhiguna Kuncoro, Aida
Nematzadeh, Elena Gribovskaya, Domenic Do-
nato, Angeliki Lazaridou, Arthur Mensch, Jean-
Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grig-
orev, Doug Fritz, Thibault Sottiaux, Mantas Pa-
jarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama,
Cyprien de Masson d’Autume, Yujia Li, Tay-
fun Terzi, Vladimir Mikulik, Igor Babuschkin,
Aidan Clark, Diego de Las Casas, Aurelia Guy,
Chris Jones, James Bradbury, Matthew J. Johnson,
Blake A. Hechtman, Laura Weidinger, Iason Gabriel,
William S. Isaac, Edward Lockhart, Simon Osin-
dero, Laura Rimell, Chris Dyer, Oriol Vinyals, Ka-
reem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis
Hassabis, Koray Kavukcuoglu, and Geoffrey Irv-
ing. 2021. Scaling language models: Methods,
analysis & insights from training gopher. CoRR,
abs/2112.11446.
Nazneen Fatema Rajani, Bryan McCann, Caiming
Xiong, and Richard Socher. 2019. Explain yourself!
leveraging language models for commonsense rea-
soning. In Proceedings of the 57th Conference of
the Association for Computational Linguistics, ACL
2019, Florence, Italy, July 28- August 2, 2019, Vol-
ume 1: Long Papers, pages 4932–4942. Association
for Computational Linguistics.
Erik F. Tjong Kim Sang and Fien De Meulder.
2003. Introduction to the conll-2003 shared task:
Language-independent named entity recognition. In
Proceedings of the Seventh Conference on Natural
Language Learning, CoNLL 2003, Held in cooper-
ation with HLT-NAACL 2003, Edmonton, Canada,
May 31 - June 1, 2003, pages 142–147. ACL.
Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo
Lee, Percy Liang, and Tatsunori Hashimoto. 2023.
Whose opinions do language models reflect? arXiv
preprint arXiv:2303.17548.
Shaden Smith, Mostofa Patwary, Brandon Norick,
Patrick LeGresley, Samyam Rajbhandari, Jared
 Casper, Zhun Liu, Shrimai Prabhumoye, George
Zerveas, Vijay Korthikanti, Elton Zheng, Rewon
Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia
Song, Mohammad Shoeybi, Yuxiong He, Michael
Houston, Saurabh Tiwary, and Bryan Catanzaro.
2022. Using deepspeed and megatron to train
megatron-turing NLG 530b, A large-scale genera-
tive language model. CoRR, abs/2201.11990.
Teo Susnjak. 2022. Chatgpt: The end of online exam
integrity? CoRR, abs/2212.09292.
Teo Susnjak. 2023. Applying bert and chatgpt for senti-
ment analysis of lyme disease in scientific literature.
arXiv preprint arXiv:2302.06474.
Romal Thoppilan, Daniel De Freitas, Jamie Hall,
Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze
Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du,
YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng,
Amin Ghafouri, Marcelo Menegali, Yanping Huang,
Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao
Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts,
Maarten Bosma, Yanqi Zhou, Chung-Ching Chang,
Igor Krivokon, Will Rusch, Marc Pickett, Kath-
leen S. Meier-Hellstern, Meredith Ringel Morris,
Tulsee Doshi, Renelito Delos Santos, Toju Duke,
Johnny Soraker, Ben Zevenbergen, Vinodkumar
Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen
Olson, Alejandra Molina, Erin Hoffman-John, Josh
Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna,
Matthew Lamm, Viktoriya Kuzmina, Joe Fenton,
Aaron Cohen, Rachel Bernstein, Ray Kurzweil,
Blaise Aguera-Arcas, Claire Cui, Marian Croak,
Ed H. Chi, and Quoc Le. 2022. Lamda: Lan-
guage models for dialog applications. CoRR,
abs/2201.08239.
Sunil Thulasidasan, Gopinath Chennupati, Jeff A.
Bilmes, Tanmoy Bhattacharya, and Sarah Michalak.
2019. On mixup training: Improved calibration and
predictive uncertainty for deep neural networks. In
Advances in Neural Information Processing Systems
32: Annual Conference on Neural Information Pro-
cessing Systems 2019, NeurIPS 2019, December 8-
14, 2019, Vancouver, BC, Canada, pages 13888–
13899.
Ruibo Tu, Chao Ma, and Cheng Zhang. 2023. Causal-
discovery performance of chatgpt in the context of
neuropathic pain diagnosis. CoRR, abs/2301.13819.
Amir Pouran Ben Veyseh, Viet Dac Lai, Franck Der-
noncourt, and Thien Huu Nguyen. 2021. Unleash
GPT-2 power for event detection. In Proceedings of
the 59th Annual Meeting of the Association for Com-
putational Linguistics and the 11th International
Joint Conference on Natural Language Processing,
ACL/IJCNLP 2021, (Volume 1: Long Papers), Vir-
tual Event, August 1-6, 2021, pages 6271–6282. As-
sociation for Computational Linguistics.
David Wadden, Ulme Wennberg, Yi Luan, and Han-
naneh Hajishirzi. 2019. Entity, relation, and event
extraction with contextualized span representations.
In Proceedings of the 2019 Conference on Empiri-
cal Methods in Natural Language Processing and
the 9th International Joint Conference on Natural
Language Processing, EMNLP-IJCNLP 2019, Hong
Kong, China, November 3-7, 2019, pages 5783–
5788. Association for Computational Linguistics.
Jindong Wang, Xixu Hu, Wenxin Hou, Hao Chen,
Runkai Zheng, Yidong Wang, Linyi Yang, Haojun
Huang, Wei Ye, Xiubo Geng, Binxing Jiao, Yue
Zhang, and Xing Xie. 2023a. On the robustness of
chatgpt: An adversarial and out-of-distribution per-
spective. CoRR, abs/2302.12095.
Xiao Wang, Weikang Zhou, Can Zu, Han Xia, Tianze
Chen, Yuansen Zhang, Rui Zheng, Junjie Ye,
Qi Zhang, Tao Gui, Jihua Kang, Jingsheng Yang,
Siyuan Li, and Chunsai Du. 2023b. Instructuie:
Multi-task instruction tuning for unified information
extraction. CoRR, abs/2304.08085.
Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang,
Zhongqiang Huang, Fei Huang, and Kewei Tu.
2021. Automated concatenation of embeddings
for structured prediction. In Proceedings of the
59th Annual Meeting of the Association for Com-
putational Linguistics and the 11th International
Joint Conference on Natural Language Processing,
ACL/IJCNLP 2021, (Volume 1: Long Papers), Vir-
tual Event, August 1-6, 2021, pages 2643–2660. As-
sociation for Computational Linguistics.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormo-
labashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva
Naik, Arjun Ashok, Arut Selvan Dhanasekaran, An-
jana Arunkumar, David Stap, Eshaan Pathak, Gian-
nis Karamanolakis, Haizhi Gary Lai, Ishan Puro-
hit, Ishani Mondal, Jacob Anderson, Kirby Kuz-
nia, Krima Doshi, Kuntal Kumar Pal, Maitreya Pa-
tel, Mehrad Moradshahi, Mihir Parmar, Mirali Puro-
hit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit
Verma, Ravsehaj Singh Puri, Rushang Karia, Savan
Doshi, Shailaja Keyur Sampat, Siddhartha Mishra,
Sujan Reddy A, Sumanta Patro, Tanay Dixit, and
Xudong Shen. 2022. Super-naturalinstructions:
Generalization via declarative instructions on 1600+
NLP tasks. In Proceedings of the 2022 Conference
on Empirical Methods in Natural Language Process-
ing, EMNLP 2022, Abu Dhabi, United Arab Emi-
rates, December 7-11, 2022, pages 5085–5109. As-
sociation for Computational Linguistics.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raf-
fel, Barret Zoph, Sebastian Borgeaud, Dani Yo-
gatama, Maarten Bosma, Denny Zhou, Donald Met-
zler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals,
Percy Liang, Jeff Dean, and William Fedus. 2022a.
Emergent abilities of large language models. CoRR,
abs/2206.07682.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten
Bosma, Ed H. Chi, Quoc Le, and Denny Zhou.
2022b. Chain of thought prompting elicits reasoning
in large language models. CoRR, abs/2201.11903.
Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang,
Xin Zhang, Shen Huang, Pengjun Xie, Jinan Xu,
Yufeng Chen, Meishan Zhang, Yong Jiang, and Wen-
juan Han. 2023. Zero-shot information extraction
via chatting with chatgpt. CoRR, abs/2302.10205.
Ralph Weischedel and Ada Brunstein. 2005. Bbn pro-
noun coreference and entity type corpus. Linguistic
Data Consortium, Philadelphia, 112.
Fei Wu and Daniel S. Weld. 2010. Open information
extraction using Wikipedia. In Proceedings of the
48th Annual Meeting of the Association for Compu-
tational Linguistics, pages 118–127, Uppsala, Swe-
den. Association for Computational Linguistics.
Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki
Takeda, and Yuji Matsumoto. 2020. LUKE: deep
contextualized entity representations with entity-
aware self-attention. In EMNLP 2020.
Deming Ye, Yankai Lin, Peng Li, and Maosong Sun.
2022. Packed levitated marker for entity and relation
extraction. In Proceedings of the 60th Annual Meet-
ing of the Association for Computational Linguistics
(Volume 1: Long Papers), ACL 2022, Dublin, Ire-
land, May 22-27, 2022, pages 4904–4917. Associa-
tion for Computational Linguistics.
Wei Ye, Bo Li, Rui Xie, Zhonghao Sheng, Long Chen,
and Shikun Zhang. 2019. Exploiting entity BIO tag
embeddings and multi-task learning for relation ex-
traction with imbalanced data. In Proceedings of
the 57th Conference of the Association for Compu-
tational Linguistics, ACL 2019, Florence, Italy, July
28- August 2, 2019, Volume 1: Long Papers, pages
1351–1360. Association for Computational Linguis-
tics.
Daojian Zeng, Kang Liu, Yubo Chen, and Jun Zhao.
2015. Distant supervision for relation extraction via
piecewise convolutional neural networks. In Pro-
ceedings of the 2015 Conference on Empirical Meth-
ods in Natural Language Processing, pages 1753–
1762.
Kailin Zhao, Xiaolong Jin, Long Bai, Jiafeng Guo,
and Xueqi Cheng. 2022. Knowledge-enhanced self-
supervised prototypical network for few-shot event
detection. In Findings of the Association for Com-
putational Linguistics: EMNLP 2022, Abu Dhabi,
United Arab Emirates, December 7-11, 2022, pages
6266–6275. Association for Computational Linguis-
tics.
Kang Zhao, Hua Xu, Yue Cheng, Xiaoteng Li, and Kai
Gao. 2021. Representation iterative fusion based
on heterogeneous graph neural network for joint en-
tity and relation extraction. Knowl. Based Syst.,
219:106888.
Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and
Dacheng Tao. 2023. Can chatgpt understand too?
a comparative study on chatgpt and fine-tuned bert.
arXiv preprint arXiv:2302.10198.
Wenxuan Zhou and Muhao Chen. 2021. An im-
proved baseline for sentence-level relation extrac-
tion. CoRR, abs/2102.01373.
Terry Yue Zhuo, Yujin Huang, Chunyang Chen, and
Zhenchang Xing. 2023. Exploring AI ethics of chat-
gpt: A diagnostic analysis. CoRR, abs/2301.12867.
Julia El Zini and Mariette Awad. 2023. On the explain-
ability of natural language processing deep models.
ACM Comput. Surv., 55(5):103:1–103:31.
Xinyu Zuo, Haijin Liang, Ning Jing, Shuang Zeng,
Zhou Fang, and Yu Luo. 2022. Type-enriched hierar-
chical contrastive strategy for fine-grained entity typ-
ing. In Proceedings of the 29th International Confer-
ence on Computational Linguistics, COLING 2022,
Gyeongju, Republic of Korea, October 12-17, 2022,
pages 2405–2417. International Committee on Com-
putational Linguistics.

Bowen Zhang, Daijun Ding, and Liwen Jing. 2022a.

How would stance detection techniques evolve after
the launch of chatgpt? CoRR, abs/2212.14548.

Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor An-

geli, and Christopher D. Manning. 2017. Position-
aware attention and supervised data improve slot fill-
ing. In EMNLP 2017.

Zhisong Zhang, Emma Strubell, and Eduard H. Hovy.

2022b. Transfer learning from semantic role la-
beling to event argument extraction with template-
based slot querying. In Proceedings of the
2022 Conference on Empirical Methods in Natural
Language Processing, EMNLP 2022, Abu Dhabi,
United Arab Emirates, December 7-11, 2022, pages
2627–2647. Association for Computational Linguis-
tics.
A Appendix a given sample to review candidate labels in the
Top-k prediction set and learns the connections be-
A.1 Dataset
tween them. It also includes a dynamic k-selection
We report the dataset used in each task in this sub- mechanism to learn more powerful and discrim-
section. For each task, we use two commonly used inative relation representation. This method sets
datasets for the evaluation. Table 10 shows the SOTA results on the TACRED dataset. Zhao et al.
detailed statistical information. (2021) proposed RIFRE, which models relations
Besides, we also report the number of manu- and words as nodes on a graph, and iteratively fuses
ally annotated samples for each dataset, denoted as the two types of semantic nodes using message
#Ann., as shown in Table 9. passing. This approach obtains node representa-
tions that are better suited for relation extraction
Task DataSet #Ann. tasks. The model then performs relation extraction
ED BBN 385 on the updated node representations. This method
NER CoNNL 300 sets SOTA results on the SemEval2010 dataset.
RC SemEval 400 Relation Extraction (RE): Ye et al. (2022) pro-
RE ACE05-R 66 posed PL-Marker, a novel span representation
ED ACE05-E 218 approach that considers the interrelation between
EAE ACE05-E 313 span pairs by packing markers in the encoder.
EE ACE05-E 552 To better model entity boundary information, PL-
Marker proposes a neighborhood-oriented pack-
Table 9: The number of manually annotated samples ing strategy that considers neighbor spans inte-
for each dataset.
grally. For more complicated span pair classifica-
tion tasks, this paper also designs a subject-oriented
A.2 The State-of-the-Art Methods on Single packing strategy, which packs each subject and its
Dataset objects to model the interrelation between same-
subject span pairs. This method sets SOTA re-
In this section, we introduce the state-of-the-art sults on ACE05-R and SciERC datasets. It also
method on each dataset: achieves the best performance on the OntoNotes
Entity Typing (ET): Zuo et al. (2022) proposed 5.0 datasets of NER task.
a type-enriched hierarchical contrastive strategy for
Event Detection (ED): Liu et al. (2022a)
entity typing task, named PICOT. PICOT mod-
proposed SaliencyED, a novel training mech-
els differences between hierarchical types to dis-
anism for ED, which can distinguish between
tinguish similar types at different levels of gran-
trigger-dependent and context-dependent types,
ularity. It also embeds type information into en-
and achieves promising results on ACE05-E
tity contexts and employ a constrained contrastive
dataset. Lin et al. (2020) proposed ONEIE neu-
strategy on the hierarchical structure. This method
ral framework aims to globally optimize informa-
achieves SOTA results on the BBN and OntoNotes
tion extraction as a graph from an input sentence,
5.0 datasets.
capturing cross-subtask and cross-instance inter-
Named Entity Recognition (NER): Wang et al.
dependencies, and and achieves promising results
(2021) proposed a model named ACE, which auto-
on ACE05-E+ dataset.
mates finding better embeddings for structured pre-
diction tasks. It uses a neural architecture search- Event Argument Extraction (EAE): Hsu et al.
inspired formulation where a controller updates (2022) proposed DEGREE, which formulates
belief based on a reward. The reward is the ac- event extraction as a conditional generation prob-
curacy of a task model trained on a task dataset lem, summarizing events mentioned in a passage
with the concatenated embeddings as input. This into a natural sentence following a predefined pat-
method achieves SOTA results on the CoNLL2003 tern, as learned from a prompt. Extracted event pre-
dataset. dictions are then obtained from the generated sen-
Relation Classification (RC): Li et al. (2022a) tence using a deterministic algorithm. DEGREE
proposed Label Graph Network with Top-k Pre- sets the best results on both ACE05-E and ACE05-
diction Set (KLG), to effectively utilize the Top- E+ datasets.
k prediction set. KLG builds a label graph for Event Argument Extraction (EAE): the
 Task Dataset #class #test
Entity BBN (Weischedel and Brunstein, 2005) 17 23542
Typing(ET) OntoNotes (Gillick et al., 2014) 41 13393
Named Entity CoNLL 2003 (Sang and Meulder, 2003) 4 3453
Recognition(NER) OntoNotes 9 18 8233
Relation TACRED (Zhang et al., 2017) 42 15517
Classification(RC) SemEval2010 (Hendrickx et al., 2010) 10 2717
Relation ACE05-R 10 7/6 2050
Extraction(RE) SciERC (Luan et al., 2018) 6/7 551
Event ACE05-E (Wadden et al., 2019) 33 832
Detection(ED) ACE05-E+ (Lin et al., 2020) 33 676
Event Argument ACE05-E (Wadden et al., 2019) 22/33 403
Extraction(EAE) ACE05-E+ (Lin et al., 2020) 22/33 424
Event ACE05-E (Wadden et al., 2019) 22/33 832
Extraction(EE) ACE05-E+ (Lin et al., 2020) 22/33 676

Table 10: The table presents several key statistical characteristics of the datasets used in our research, including 14
datasets that belonging to 7 different IE tasks.

SOTA methods are ONEIE for ACE05-E, and DE-

GREE for ACE05-E+.

A.3 Exemplar of the Input

In this section, we show an input examples for the
event detection task to help readers understand our
implement, as shown in Table 11.
 Input of Event Detection (ED)
Task Description: Given an input list of words, identify all triggers in the list, and categorize
each of them into the predefined set of event types. A trigger is the main word that most clearly
expresses the occurrence of an event in the predefined set of event types.
Pre-defined Label Set: The predefined set of event types includes: [Life.Be-Born, Life.Marry,
Life.Divorce, Life.Injure, Life.Die, Movement.Transport, Transaction.Transfer-Ownership,
Transaction.Transfer-Money, Business.Start-Org, Business.Merge-Org, Business.Declare-
Bankruptcy, Business.End-Org, Conflict.Attack, Conflict.Demonstrate, Contact.Meet, Contact.
Phone-Write, Personnel.Start-Position, Personnel.End-Position, Personnel.Nominate, Personnel.
Elect, Justice.Arrest-Jail, Justice.Release-Parole, Justice.Trial-Hearing, Justice.Charge-Indict,
Justice.Sue, Justice.Convict, Justice.Sentence, Justice.Fine, Justice.Execute, Justice.Extradite,
Justice.Acquit, Justice.Appeal, Justice.Pardon].
Input and Task Requirement: Perform ED task for the following input list, and print the output:
[’Putin’, ’concluded’, ’his’, ’two’, ’days’, ’of’, ’talks’, ’in’, ’Saint’, ’Petersburg’, ’with’, ’Jacques’,
’Chirac’, ’of’, ’France’, ’and’, ’German’, ’Chancellor’, ’Gerhard’, ’Schroeder’, ’on’, ’Saturday’,
’still’, ’urging’, ’for’, ’a’, ’central’, ’role’, ’for’, ’the’, ’United’, ’Nations’, ’in’, ’a’, ’post’, ’-’,
’war’, ’revival’, ’of’, ’Iraq’, ’.’] The output of ED task should be a list of dictionaries following
json format. Each dictionary corresponds to the occurrence of an event in the input list and should
consists of "trigger", "word_index", "event_type", "top3_event_type", "top5_event_type",
"confidence", "if_context_dependent", "reason" and "if_reasonable" nine keys. The value of "word_
index" key is an integer indicating the index (start from zero) of the "trigger" in the input list. The
value of "confidence" key is an integer ranging from 0 to 100, indicating how confident you are that
the "trigger" expresses the "event_type" event. The value of "if_context_dependent" key is either 0
(indicating the event semantic is primarily expressed by the trigger rather than contexts) or 1
(indicating the event semantic is primarily expressed by contexts rather than the trigger). The value
of "reason" key is a string describing the reason why the "trigger" expresses the "event_type", and
do not use any " mark in this string. The value of "if_reasonable" key is either 0 (indicating the reason
given in the "reason" field is not reasonable) or 1 (indicating the reason given in the "reason" field is
reasonable). Note that your answer should only contain the json string and nothing else.

Table 11: The input example of event detection task. This example is extracted from ACE05-E, and all the above
three parts are jointly imported into ChatGPT.