TG-CSR: A human-labeled dataset grounded in nine formal commonsense categories
No Thumbnail Available
Authors
Santos, Henrique
Mulvehill, Alice M.
Shen, Ke
Kejriwal, Mayank
McGuinness, Deborah L.
Issue Date
2023-12-01
Type
Article
Language
Keywords
Alternative Title
Abstract
Machine Common Sense Reasoning is the subfield of Artificial Intelligence that aims to enable machines to behave or make decisions similarly to humans in everyday and ordinary situations. To measure progress, benchmarks in the form of question-answering datasets have been developed and published in the community to evaluate machine commonsense models, including large language models. We describe the individual label data produced by six human annotators originally used in computing ground truth for the Theoretically-Grounded Commonsense Reasoning (TG-CSR) benchmark's composing datasets. According to a set of instructions, annotators were provided with spreadsheets containing the original TG-CSR prompts and asked to insert labels in specific spreadsheet cells during annotation sessions. TG-CSR data is organized in JSON files, individual raw label data in a spreadsheet file, and individual normalized label data in JSONL files. The release of individual labels can enable the analysis of the labeling process itself, including studies of noise and consistency across annotators.
Description
Full Citation
Santos, H., Mulvehill, A. M., Shen, K., Kejriwal, M., & McGuinness, D. L. (2023). TG-CSR: A human-labeled dataset grounded in nine formal commonsense categories. In Data in Brief (Vol. 51, p. 109666). Elsevier BV. https://doi.org/10.1016/j.dib.2023.109666
Publisher
Elsevier