A Theoretically Grounded Question Answering Data Set for Evaluating Machine Common Sense
No Thumbnail Available
Authors
Santos, Henrique
He, Shen
Mulvehill, Alice M.
Kejriwal, Mayank
McGuinness, Deborah L.
Issue Date
2024-02-01
Type
Language
Keywords
Alternative Title
Abstract
Achieving machine common sense has been a longstanding problem within Artificial Intelligence. Thus far, benchmark data sets that are grounded in a theory of common sense and can be used to conduct rigorous, semantic evaluations of common sense reasoning (CSR) systems have been lacking. One expectation of the AI community is that neuro-symbolic reasoners can help bridge this gap towards more dependable systems with common sense. We propose a novel benchmark, called Theoretically Grounded common sense Reasoning (TG-CSR), modeled as a set of question answering instances, with each instance grounded in a semantic category of common sense, such as space, time, and emotions. The benchmark is few-shot i.e., only a few training and validation examples are provided in the public release to avoid the possibility of overfitting. Results from recent evaluations suggest that TG-CSR is challenging even for state-of-the-art statistical models. Due to its semantic rigor, this benchmark can be used to evaluate the common sense reasoning capabilities of neuro-symbolic systems.
Description
Full Citation
Henrique Santos, Ke Shen, Alice M. Mulvehill, Mayank Kejriwal, Deborah L. McGuinness; A Theoretically Grounded Question Answering Data Set for Evaluating Machine Common Sense. Data Intelligence 2024; 6 (1): 1–28. doi: https://doi.org/10.1162/dint_a_00234
Publisher
MIT Press