Tetherless World Publications

Publications

Browse

Recent Submissions

Now showing 1 - 5 of 758
  • Item
    LLM-Based Code Generation for Querying Temporal Tabular Financial Data
    (IEEE, 2024-10-22) Lashuel, Mohamed; Kurdistan, Gulrukh; Green, Aaron; Erickson, John S.; Seneviratne, Oshani; Bennett, Kristin P.
    We examine the question of ``how well large language models (LLMs) can answer questions using temporal tabular financial data by generating code?''. Leveraging advanced language models, specifically GPT-4 and Llama 3, we aim to scrutinize and compare their abilities to generate coherent and effective code for Python, R, and SQL based on natural language prompts. We design an experiment to assess the performance of LLMs on natural language prompts on a large temporal financial dataset. We created a set of queries with hand-crafted R code answers. To investigate the strengths and weaknesses of LLMs, each query was created with different factors that characterize the financial meaning of the queries and their complexity. We demonstrate how to create specific zero-shot prompts to generate code to answer natural language queries about temporal financial tabular data. We develop specific system prompts for each language to ensure they correctly answer time-oriented questions. We execute this experiment on two LLMs (GPT-4 and Llama 3), assess if the outputs produced are executable and correct, and assess the efficiency of the produced code for Python, SQL, and R. We find that while LLMs have promising performance, their performance varies greatly across the languages, models, and experimental factors. GPT-4 performs best on Python (95.2\% correctness) but has significantly weaker performance on SQL (87.6\% correctness) and R (79.0\% correctness). Llama 3 is less successful at generating code overall, but it achieves its best results in R (71.4\% correctness). A multi-factor statistical analysis of the results with respect to the defined experimental factors provides further insights into the specific areas of challenge in code generation for each LLM. Our preliminary results on this modest benchmark demonstrate a framework for developing larger, comprehensive, unique benchmarks for both temporal financial tabular data and R code generation. While Python and SQL already have benchmarks, we are filling in the gaps for R. Powerful AI agents for text-to-code generation, as demonstrated in this work, provide a critical capability required for the next-generation AI-based natural language financial intelligence systems and chatbots, directly addressing the complex challenges presented by querying temporal tabular financial data.
  • Item
    Towards a Progression-Aware Autonomous Dialogue Agent
    (Association for Computational Linguistics, 2022-07-10) Sanders, Abraham; Strzalkowski, Tomek; Si, Mei; Chang, Albert; Dey, Deepanshu; Braasch, Jonas; Wang, Dakuo
    Recent advances in large-scale language modeling and generation have enabled the creation of dialogue agents that exhibit human-like responses in a wide range of conversational scenarios spanning a diverse set of tasks, from general chit-chat to focused goal-oriented discourse. While these agents excel at generating high-quality responses that are relevant to prior context, they suffer from a lack of awareness of the overall direction in which the conversation is headed, and the likelihood of task success inherent therein. Thus, we propose a framework in which dialogue agents can evaluate the progression of a conversation toward or away from desired outcomes, and use this signal to inform planning for subsequent responses. Our framework is composed of three key elements: (1) the notion of a “global” dialogue state (GDS) space, (2) a task-specific progression function (PF) computed in terms of a conversation’s trajectory through this space, and (3) a planning mechanism based on dialogue rollouts by which an agent may use progression signals to select its next response.
  • Item
    Adapting Emotion Detection to Analyze Influence Campaigns on Social Media
    (Association for Computational Linguistics, 2023-07-01) Bhaumik, Ankita; Bernhardt, Andy; Katsios, Gregorios A; Sa, Ning; Strzalkowski, Tomek
    Social media is an extremely potent tool for influencing public opinion, particularly during important events such as elections, pandemics, and national conflicts. Emotions are a crucial aspect of this influence, but detecting them accurately in the political domain is a significant challenge due to the lack of suitable emotion labels and training datasets. In this paper, we present a generalized approach to emotion detection that can be adapted to the political domain with minimal performance sacrifice. Our approach is designed to be easily integrated into existing models without the need for additional training or fine-tuning. We demonstrate the zero-shot and few-shot performance of our model on the 2017 French presidential elections and propose efficient emotion groupings that would aid in effectively analyzing influence campaigns and agendas on social media.
  • Item
    Uncovering Agendas: A Novel French & English Dataset for Agenda Detection on Social Media
    (International Committee on Computational Linguistics (ICCL), 2024-05-20) Katsios, Gregorios A; Sa, Ning; Bhaumik, Ankita; Strzalkowski, Tomek
    The behavior and decision making of groups or communities can be dramatically influenced by individuals pushing particular agendas, e.g., to promote or disparage a person or an activity, to call for action, etc.. In the examination of online influence campaigns, particularly those related to important political and social events, scholars often concentrate on identifying the sources responsible for setting and controlling the agenda (e.g., public media). In this article we present a methodology for detecting specific instances of agenda control through social media where annotated data is limited or non-existent. By using a modest corpus of Twitter messages centered on the 2022 French Presidential Elections, we carry out a comprehensive evaluation of various approaches and techniques that can be applied to this problem. Our findings demonstrate that by treating the task as a textual entailment problem, it is possible to overcome the requirement for a large annotated training dataset.
  • Item
    Social Convos: Capturing Agendas and Emotions on Social Media
    (International Committee on Computational Linguistics (ICCL), 2024-05-20) Bhaumik, Ankita; Sa, Ning; Katsios, Gregorios A; Strzalkowski, Tomek
    Social media platforms are popular tools for disseminating targeted information during major public events like elections or pandemics. Systematic analysis of the message traffic can provide valuable insights into prevailing opinions and social dynamics among different segments of the population. We are specifically interested in influence spread, and in particular whether more deliberate influence operations can be detected. However, filtering out the essential messages with telltale influence indicators from the extensive and often chaotic social media traffic is a major challenge. In this paper we present a novel approach to extract influence indicators from messages circulating among groups of users discussing particular topics. We build upon the concept of a convo to identify influential authors who are actively promoting some particular agenda around that topic within the group. We focus on two influence indicators: the (control of) agenda and the use of emotional language.