Incorporating external knowledge into food representation and recommendation

Authors
Li, Diya
ORCID
https://orcid.org/0000-0002-1569-7221
Thumbnail Image
Other Contributors
Zaki, Mohammed J., 1971-
Hendler, James A.
McGuinness, Deborah L.
Chen, Ching-hua
Zaki, Mohammed J., 1971-
Issue Date
2022-08
Keywords
Computer science
Degree
PhD
Terms of Use
This electronic version is a licensed copy owned by Rensselaer Polytechnic Institute (RPI), Troy, NY. Copyright of original work retained by author.
Full Citation
URI
https://hdl.handle.net/20.500.13015/6237
Abstract
Food is our most basic need, the very stuff of life. Food is fundamental for human beings and eating well is essential to good health. Food computing has attracted great interest from various fields. Representing food as vectors can capture hidden information gleaned from massive datasets and help further studies on food. As an important task of food computing, food recommendation can be a means to help people find food they might love, and also to help them eat more healthily. Acting as a special case of food recommendation, food substitution is an open task to explore potential food alternatives. Structured knowledge furnishes an in-depth understanding of the world. At the same time, knowledge graphs (KG) that represent structural relations between entities have become an increasingly popular research tool towards enabling artificial intelligence. KGs can boost the studies in food computing by its structural and semantic information as well as providing interpretability. In this thesis, we focus on two main research questions: (1) How can we capture the semantics of recipes for use in food representation and recommendation (QR1)? (2) How can we employ external knowledge like KGs to facilitate tasks of food computing (QR2)? To address the two research questions, we propose a novel model for recipe representation learning from pure text.Learning recipe embeddings is a challenging task, since there is a lack of high quality annotated food datasets. We tackle this problem by adopting a triplet loss for model optimization so that related recipes are closer in the latent semantic space. An external knowledge source like the food KG is employed to construct feasible recipes triples by performing recipe sample mining. We provide a joint approach for learning effective pretrained recipe embeddings using both the ingredients and cooking instructions. A set transformer is adopted to encode the ingredient set to preserve its permutation-invariant property. In terms of food recommendation, it is challenging to offer users food that both meets users' preference and a health goal.We introduce health-guided recipe recommendation as a way to incrementally shift users towards healthier recipe options while respecting the preferences reflected in their historical choices. The food KG aids this task by providing relationships among food along with their nutrition information. Thus, we consider the task of recipe recommendation over KGs. In particular, we jointly learn recipe representations via graph neural networks over two KG subgraphs, which target user preferences and recipe healthiness. Another challenge in food recommendation is that it is hard to balance the trade-off between preference and healthiness. We utilize a knowledge transfer scheme to enable the transfer of useful semantic information across the preferences and healthiness aspects instead of simple fusion. Combining the above two tasks, we further propose an approach to learn KG representations with adversarial food substitution.To enlarge the scope of food representation learning that is not limited to recipe data and to fully utilize the external knowledge, we perform knowledge representation learning over a food KG. We employ a pretrained language model to encode entities and relations, thus emphasizing contextual information in food KGs. The model is trained on two tasks -- predicting a masked entity from a given triple and predicting the plausibility of a triple obtained from the KG. As an open food recommendation task, analysis of food substitutions helps in identifying optimal dietary choices and supporting different user needs. It is hard to evaluate the substitutions due to the lack of an adequate validation set. To tackle this challenge, we propose a collection of adversarial sample generation strategies for different food substitutions over our learnt KG embeddings. To meet different purposes, we generate high quality context-aware recipe and ingredient substitutions by replacing, adding, and detecting actions over token and entities. We also provide generalized ingredient substitutions to meet the needs of general substitution purpose.
Description
August 2022
School of Science
Department
Dept. of Computer Science
Publisher
Rensselaer Polytechnic Institute, Troy, NY
Relationships
Rensselaer Theses and Dissertations Online Collection
Access
Restricted to current Rensselaer faculty, staff and students in accordance with the Rensselaer Standard license. Access inquiries may be directed to the Rensselaer Libraries.
Collections
RPI Theses Online (Complete)