Principled and practical static analysis for python
Loading...
Authors
Rak-amnouykit, Ingkarat
Issue Date
2024-12
Type
Electronic thesis
Thesis
Thesis
Language
en_US
Keywords
Computer science
Alternative Title
Abstract
Python is a popular dynamic language for data science and machine learning, and is increasingly adopted for large and complex programming tasks; however, static analysis tools to support software quality and productivity are largely lacking.In programming languages type systems are the most fundamental form of static analysis. Therefore, we first embark on a study to understand the current state of Python type systems. We perform a large-scale empirical study of Python 3 types, usage of types by developers, and two popular static type checking tools, MyPy and PyType. The study shows that Python 3 types are not widely used, and that existing type systems tend to disagree with each other and with user annotations. It demonstrates the need to further study and develop types and type inference analyses as well as deeper and automatic static analyses to support Python development.
The result of the study on Python 3 types motivates the remainder of this thesis. We create a principled approach for static analysis for Python with the initial goal of improving type inference analysis and later applying the concept to other client static analyses.
Our approach to static analysis for Python requires analysis formalization into two phases: (1) syntax and (2) interpretation semantics. Given a specific static analysis, analysis designers first define a subset of Python syntax called PetPy that specifies explicitly what Python constructs are interpreted precisely and what constructs are interpreted approximately. The designers then define interpretation over these syntactic constructs as needed by the specific analysis. The novelty of our approach is the separation of precisely-interpreted constructs from approximately-interpreted ones, thus making analysis approximation, unavoidable in Python, explicit.
Following the principled and practical static analysis approach, we build three novel static analyses for Python: (1) a reference immutability analysis, (2) an interprocedural weakest precondition analysis, and (3) an Andersen-style points-to analysis. Each analysis defines a version of PetPy and interpretation of syntax suited to its needs. The analyses are mostly static, with a novel integration of concrete evaluation.
The analyses scale to production-strength real-world Python codes and largely outperform state-of-the-art techniques and tools. The weakest precondition analysis uncovers bugs in both code and documentation of popular machine learning libraries. We submitted issues and pull requests, which developers promptly merged, to sklearn, TensorFlow, numpy, and IBM’s Lale framework.
In conclusion, our techniques advance static analysis for Python. In future work, we will continue to refine the formalization and build new static analyses for Python.
Description
December 2024
School of Science
School of Science
Full Citation
Publisher
Rensselaer Polytechnic Institute, Troy, NY