Semantically enabled medical image understanding

Abstract

Medical imaging examination is the most common form of routine medical analysis, which involves several stages of reasoning and careful analysis to reach a final decision by the radiologist. Most deep learning frameworks utilize only the imaging data in its entirety to reach a clinical decision, ignoring important patient information that might be vital for the diagnosis. This includes symptoms, age, and patient history information. Furthermore, Most deep learning systems ignore domain based information and commonsense reasoning when they make their prediction. In this work, we designed and implemented an AI system that can use both the anatomical information and patient history information while making decisions. In particular, we focused on using semantic web technologies, ontologies and knowledge graphs, to bridge the gap in existing deep learning systems. We developed a deep learning system that can detect chest x-ray pathologies within the correct anatomical location. We developed a Medical Imaging and Diagnostic Ontology, MIDO,to provide a standardized format for modelling medical imaging tasks, including disease classification and disease localization. We then generated knowledge graphs from the chest x-ray images using the ontology for modelling the entities are relationships. Furthermore, we demonstrated the possibility of semantic reasoning to enhance transparency, trust, interpretability and explainability of the deep learning system through the application of the knowledge graph to increase the physician's trust in the individual patient's deep learning prediction by determining the likelihood of the system being correct, enriching the knowledge graph with additional domain based knowledge and reasoning causal relationships between different diseases. Not only that, but we conducted an in-depth evaluation of the various parts of our workflow and compared them to existing state-of-the art whenever possible.