Tetherless World Videos/Presentations

Presentations

Browse

Recent Submissions

Now showing 1 - 5 of 281
  • Item
    Psychometric Assessment Issues and Potential Opportunities
    (The Healthcare and Life Sciences Symposium (HCLS), co-located with The 2024 Knowledge Graph Conference (KGC), 2024-05-07) Sprung, Manuel S.; Santos, Henrique; Rook, Kelsey; Pinheiro, Paulo; McGuinness, Deborah L.; Chorpita, Bruce F.
    Psychometric assessment is essential for mental health care. However, which assessment instruments are best suited for particular use cases remains largely opaque to researchers, clinicians and policy makers alike. Although there have been metrics proposed to indicate the strength of evidence for assessment resources (SOEFA), the reporting of research evidence needed for these metrics is currently inconsistent and highly fragmented, which complicates evaluation of SOEFA. In efforts to improve the systematic collection and reporting of SOEFA, Hunsley and Mash (2008) and Youngstrom et al. (2017) have proposed a standard set of criteria to evaluate the SOEFA. Twelve categories, including norms, internal consistency, interrater reliability, test-retest reliability (stability), repeatability, content validity, construct validity, discriminative validity, prescriptive validity, validity generalization, treatment sensitivity, and clinical utility, are suggested to evaluate mental health assessment resources as adequate, good or excellent. In an effort to apply these criteria to a widely used measure of youth anxiety and depression, the Revised Child Anxiety and Depression Scale (https://rcads.ucla.edu/; Chorpita et al., 2000), we encountered a variety of challenges due to the fit between standards and the knowledge base represented in published research papers. First, it is difficult to map and connect the proposed criteria to the inconsistent, disintegrated and fragmented research evidence, such as varied use of criteria to determine validity or accuracy of measurement. Second, many assessment instruments exist in different versions, such as translations in different languages, derivatives (e.g., short forms) or respondent formats (e.g., youth or caregiver forms). The provenance of different versions (e.g., which items are newly created or are reused from already existing instruments) is highly opaque, and there is minimal guidance about how SOEFA metrics (indicators about the degree of uncertainty in the expected performance of a specific version) can be applied across resources with shared provenance. For example, one could assume that different versions inherit the SOEFA (1) of the parent class of instruments or (2) from sibling classes or (3) or not at all. Third, psychometric assessment instruments are always used in a specific context, i.e., with a specific cohort (with a specified age, gender, language, nationality, etc.) and with a specific purpose (of assessment), i.e., screening, supporting diagnosis, or treatment progress monitoring. To inform end users of the potential suitability of an assessment resource therefore requires marshaling large amounts of meta-data about the contexts and cohorts in which the SOEFA metrics were established for each measure. Thus, despite a laudable aim to apply standardized metrics to inform users about the evidence supporting specific assessment, the practical implementation of these standards (or their evolution) requires a significant change to the knowledge infrastructure of psychometric assessment. In our joint work that includes psychological clinical assessment experts along with semantic technology experts, we are exploring a semantic infrastructure that supports the encoding of SOEFA metrics in the context of mental health screening questionnaires. Our current effort involves the modeling of statistical evidence using standardized terminology from established ontologies (such as HAScO, STATO, and ECO). Our goal is to provide a provenance-aware, semantics-powered data portal that can be used by a broad set of users to help understand some of the important nuances of assessment instruments, to guide which instruments are best suited to which purpose, and to expose the reasons for (or against) such choices, in a way that is aligned with the guidance of the best scholars in mental health assessment.
  • Item
    Towards an Ontology of Psychometric Measures
    (The Healthcare and Life Sciences Symposium (HCLS), co-located with The 2024 Knowledge Graph Conference (KGC), 2024-05-07) Rook, Kelsey; Santos, Henrique; Chorpita, Bruce F.; Sprung, Manuel S.; Pinheiro, Paulo; McGuinness, Deborah L.
    The field of psychometrics is a subdomain of psychology concerned with assessment of various social and psychological dimensions of humans. In the areas of mental healthcare and the clinical science that supports it, assessment with psychometric instruments remains a dominant strategy for discovery, understanding, and decision-making. This method predominantly relies on text-based distribution of assessment instruments (usually in the form of questionnaires), and significant limitations in terms of interoperability exist. To bridge this gap, we are developing the Psychometric Ontology of Experiences and Measures (POEM) as a domain ontology tailored to capture the relationships and semantics inherent in psychometric instruments. The central aim of this new ontology is to enable the structured representation of psychometric instruments to facilitate standardized assessment across different dimensions and contexts. Other aims are to support the accessibility and construction of instruments, and making machine-readable formats more readily available. POEM considers the connections of observable questionnaire elements and the complex set of latent (unobservable) variables that are the common targets of psychometric assessment. Further POEM supports the tracking of the provenance of assessments at the questionnaire, scale, and item levels in order to capture how assessments are created, used, reused, and translated. The present work is the result of a collaboration of semantic engineers and clinical scientists to develop an ontology for psychometric instruments, adopting a bottom-up approach informed by competency questions and use cases generated by the domain experts in the team to ensure sufficient coverage. The developed ontology is designed to address diverse use cases spanning clinical services, research, and development of new psychometric instruments. POEM builds on the foundation of existing taxonomies and ontologies for psychological disorders like SNOMED-CT, the ICD, and the DSM, and integrates them with semantics describing the internal structure of psychometric assessments. While these existing taxonomies and ontologies include a comprehensive representation of psychological disorders and symptoms, POEM includes the semantics necessary to describe assessments, capturing items, scales, and provenance. POEM’s architecture builds upon established ontologies like the Virtual Solar-Terrestrial Observatory (VSTO), which is a semantic data framework that supports formal representations of physical quantities and their underlying representations with an instrument-focused subset (VSTO-I), and the Human-Aware Science Ontology (HAScO), which supports the description of scientific data for acquisition, preparation, and annotation across a variety of domains. POEM facilitates the integration of observable assessment features, those that describe the architecture of an assessment instrument and its essential parts, with the unobservable semantics of the underlying concepts they represent. The majority of document-level features in POEM align with VSTO-I entities, including questionnaires, questionnaire items, and response options. POEM relates assessments to their underlying concepts on an item-level, relating items to the specific symptoms or constructs they address; more deeply, each response option for an item is shown to estimate a specific experience of the symptom in question. Crucially, POEM also supports the encoding of questionnaire items into scales and subscales, each of which assays a construct (e.g., a clinical syndrome such as depression) that may be present at some level in a human informant, further implying the existence or non-existence of a condition (e.g., a depressive disorder). Our current work focuses on the Revised Children’s Anxiety and Depression Scale (RCADS), which estimates elevations of common psychological disorders and symptoms in children and adolescents, with scales for depression and anxiety, as well as subscales for several specific anxiety disorders [2]. POEM also supports the documentation of the creation and reuse of assessments, scales, and items, including derivation, translation, and related provenance. We evaluate POEM using our competency questions and use case scenarios with regards to the RCADS, an assessment that is suitable for evaluation due to its multiple nested scales, derived versions of different length and target respondents, many translations, and wide usage. Additionally, POEM’s utility is to be demonstrated by its integration into the Semantic Instrument Repository (SIR), a software infrastructure for managing and distributing knowledge graphs related to data acquisition instruments. SIR will harness POEM’s semantic richness to enable semantic search and retrieval functionalities, and facilitate instrument sharing in various formats, including the rendering of questionnaires into human-readable formats. SIR also supports the tracking of the provenance of assessments and their elements, in order to facilitate the creation and reuse of psychometric instruments. We maintain a GitHub repository of the POEM ontology as well as related artifacts and documentation, available at the following: https://github.com/tetherless-world/POEM
  • Item
    Eliciting Survey Knowledge with Semantic Data Dictionaries
    (2024-02-28) Santos, Henrique; Pinheiro, Paulo; McGuinness, Deborah L.
    Many countries perform surveys to gather data from their population for supporting decision-making and development of public policies. Questionnaires are possibly the most used type of data acquisition instrument in surveys, although additional kinds may be employed (especially in health-related surveys). In the United States, the NHANES is a national health and nutrition examination survey conducted by the National Center for Health Statistics, designed to collect data on adults' and children's health and nutritional status. Data is organized in several tables, each containing variables to a specific theme, such as demographics, and dietary information. In addition, data dictionaries are available to (sometimes partially) document the tables' contents. While data is mostly provided by survey participants, instruments might be collecting data related to other entities (e.g. from participants' households and families, as well as laboratory results from participants' provided blood and urine samples). All this complex knowledge can often only be elicited by humans when analyzing and understanding the data dictionaries in combination with the data. The representation of this knowledge in a machine-interpretable format could facilitate further use of the data. We detail how Semantic Data Dictionaries (SDDs) have been used to elicit knowledge about surveys, using the publicly available NHANES data and data dictionaries. In SDDs, we formalize the semantics of variables, including entities, attributes, and more, using terminology from relevant ontologies, and demonstrate how they are used in an automated process to generate a rich knowledge graph that enables downstream tasks in support of survey data analysis.
  • Item
    Facilitating Reuse of Mental Health Questionnaires via Knowledge Graphs
    (The Healthcare and Life Sciences Symposium, 2023-05-08) Santos, Henrique; Rook, Kelsey; Pinheiro, Paulo; Gruen, Daniel M.; Chorpita, Bruce F.; McGuinness, Deborah L.
    Questionnaires are one of the most common instrument types for screening patients for mental disorders. They are composed of items whose answers are typically scored to determine the elevation on a specified dimension, and hence the statistical probabilities associated with the corresponding disorder or diagnosis. The Patient Health Questionnaire (PHQ-9) and the Generalized Anxiety Disorder (GAD-7) questionnaire, for instance, measure levels of depression and anxiety respectively, and can be used to support diagnosis of depression and generalized anxiety disorder. Some questionnaires are multidimensional, such as the Revised Children's Anxiety and Depression Scale (RCADS), and can thereby estimate elevations on multiple dimensions that underlie a variety of disorders. Mental health screening questionnaires are designed so that each item assesses specific symptoms whose pattern of co-occurence (often organized in a subscale) allows estimation of how likely such symptoms would occur in the absence of the disorder whose symptoms the items represent. Questionnaire users typically estimate how likely a set of co-occuring symptoms would be (i.e., a score) in the general population as a strategy to estimate the likelihood that the respondent has a disorder warranting mental health services. The RCADS is a 47-item, youth self-report questionnaire with subscales (separation anxiety disorder, social phobia, generalized anxiety disorder, panic disorder, obsessive compulsive disorder, and major depressive disorder). It also yields a Total Anxiety Scale (sum of the 5 anxiety subscales) and a Total Internalizing Scale (sum of all 6 subscales). Items are rated on a 4-point Likert-scale from 0 ("never") to 3 ("always"). A Parent Version (RCADS-P) similarly assesses parent report of a youth’s symptoms across the same six subscales. Brief versions of the RCADS questionnaires are available as well (RCADS-25), yielding only three scores: Total Anxiety, Total Depression, and Total Anxiety and Depression. RCADS questionnaires have been translated to 19 languages. Recently there has been increased interest in supporting widespread adoption of common measures in mental health to support estimation and measurement of clinical dimensions across settings, contexts, and nations. However, the current measurement architecture in mental health is essentially based on a text-only document representation of each questionnaire, with limited knowledge of how they were created, how they relate to other questionnaires, how items relate to symptoms, which in turn relate to disorders, how short and long versions are related, etc. The result is significant constraints on the types of use cases that can be supported, with especially limited support for such pursuits as shortening the number of items, translations to new languages, reuse of items in new questionnaires, and even the combination of items from different questionnaires. We present our progress towards tackling these challenges. Our solution is composed of: (1) a modeling of mental health symptoms, scales, disorders, and their relationships as an ontology; (2) the representation of questionnaire instruments as a knowledge graph, using standardized terminology; and (3) a software infrastructure for operationalizing the management and distributions of semantic questionnaires (Semantic Instrument Repository - SIR). Using the RCADS questionnaires as a use case, we encode their (sub)scales in an ontology, reusing existing terminology from relevant sources. We expand our base Human-Aware Science Ontology (HAScO) to include questionnaire structure, and propose a new ontology for encoding and aligning mental health terminology, such as symptoms, scales, and disorders. SIR supports authoring, curation, and dissemination of questionnaires, their elements, and relationships between these elements, thus allowing questionnaires to contain mental health semantics.
  • Item
    RPIrates: Fun with OpenAI, GPTStudio and R!
    (The Rensselaer IDEA, 2023-02-14) Erickson, John S.
    Seemingly everyone has been talking about the impact of OpenAI's ChatGPT on, well, everything​, including writing code. In this very special RPIrates we talk specifically about generating great, and sometimes not-so-great, R code based on OpenAI's "Codex" models, which are easily accessed via the OpenAI API with the help of RStudio extensions (addins) provided by the GPTStudio package. After a quick icebreaker in which we demo GPTStudio in action, we briefly review transformer neural networks; we talk about how code completion tools like Microsoft's IntelliCode Compose apply TNN frameworks to write scarily excellent code; we tour OpenAI's Codex documentation; and then we get back to more hands-on with GPTstudio. We also talk about why code completion frameworks like Codex are so great at Python and Javascript but spotty with R, and of course, the ethics; oh, the ethics!​ Many links for further learning and research are provided.