Exploring and Analyzing Machine Commonsense Benchmarks
Authors
Santos, Henrique
Gordon, Minor
Liang, Jason
Forbush, Gretchen
McGuinness, Deborah L.
ORCID
No Thumbnail Available
Other Contributors
Issue Date
2021-02-01
Keywords
Machine Common Sense (MCS) Multi-modal Open World Grounded Learning and Inference (MOWGLI)
Degree
Terms of Use
Full Citation
Abstract
Commonsense question-answering (QA) tasks, in the form of benchmarks, are constantly being introduced for challenging and comparing commonsense QA systems. The benchmarks provide question sets that systems’ developers can use to train and test new models before submitting their implementations to official leaderboards. Although these tasks are created to evaluate systems in identified dimensions (e.g. topic, reasoning type), this metadata is limited and largely presented in an unstructured format or completely not present. Because machine common sense is a fast-paced field, the problem of fully assessing current benchmarks and systems with regards to these evaluation dimensions is aggravated. We argue that the lack of a common vocabulary for aligning these approaches' metadata limits researchers in their efforts to understand systems' deficiencies and in making effective choices for future tasks. In this paper, we first discuss this MCS ecosystem in terms of its elements and their metadata. Then, we present how we are supporting the assessment of approaches by initially focusing on commonsense benchmarks. We describe our initial MCS Benchmark Ontology, an extensible common vocabulary that formalizes benchmark metadata, and showcase how it is supporting the development of a Benchmark tool that enables benchmark exploration and analysis.
Description
Department
Publisher
Relationships
https://tw.rpi.edu/project/machine-common-sense-mcs-multi-modal-open-world-grounded-learning-and-inference-mowgli