Efficiency in modern memory systems: data organization, compression, and reliability management

Abstract

Modern computing systems are increasingly constrained by memory capacity, memory bandwidth, and the cost of high-performance memory. This pressure is especially visible in two workload settings studied in this dissertation: cache-backed online services and artificial intelligence (AI) inference. It also appears in a deployment constraint that becomes critical in AI systems: practical high-bandwidth memory (HBM) reliability. In all three settings, useful data quickly outgrows the fastest memory tiers, while memory interfaces continue to move data in fixed units such as cache lines, pages, bursts, and packets. System efficiency therefore depends not only on how much information an application needs, but also on how many bytes must be stored, moved, reconstructed, and protected in order to serve each request. This dissertation studies three problems that arise under these constraints. First, in hybrid dynamic random-access memory (DRAM)--solid-state drive (SSD) key-value (KV) caching with tiny objects, compression is attractive because it can increase effective DRAM capacity, but tiny objects compress poorly in isolation and larger compression groups can increase reconstruction and update cost. Second, in AI inference, model weights and the attention key-value (KV) cache used during transformer decoding create severe memory pressure, yet conventional tensor layouts limit how much useful capacity gain lossless compression can deliver and also limit when runtime precision choices reduce the bytes actually fetched. Third, HBM-based systems require strong reliability. Enforcing that reliability inside the memory device increases the burden on the device and contributes to deployment cost, which creates an incentive to move part of the protection burden to the controller. The dissertation develops and evaluates memory-system mechanisms for these three settings. For KV caching, it shows how cache organization, grouping, and decompression granularity can be co-designed so that lossless compression remains useful under tiny-object access and update patterns. For AI inference, it shows how bit-plane-oriented data layout and controller-path support can make lossless compression improve effective capacity, and can make runtime precision choices reduce traffic for model weights and the attention KV cache under unchanged memory interfaces. For HBM reliability, it shows how controller-managed protection can support strong coding under fixed transactions and high throughput, opening a path toward operation at higher raw bit error rates without losing the benefit to excessive traffic or controller cost. Taken together, these results show that substantial gains remain available inside memory-system mechanisms under fixed software-visible interfaces. In hybrid KV caching, those gains come from making compression usable for tiny objects. In AI inference, they come from changing data layout so that denser storage improves effective capacity and runtime precision choices reduce transferred bytes. In HBM reliability, they come from making strong controller-managed protection practical under fixed transactions and extreme bandwidth.