Low-Rank Quantization and Activation Compression for Resource-Constrained Large Language Model Inference

Abstract

The deployment of large language models on resource-constrained devices presents a formidable challenge due to the massive memory footprint and computational requirements inherent in architectures containing billions of parameters. As these models scale, the memory bandwidth required to load parameters during autoregressive inference becomes the primary bottleneck, significantly degrading generation throughput and increasing latency. This paper introduces a comprehensive framework that synergistically combines low-rank quantization with dynamic activation compression to alleviate these hardware constraints while preserving near-original predictive accuracy. By identifying that quantization errors are primarily concentrated in a low-rank subspace, we propose a method that maintains the bulk of the model weights in ultra-low precision formats while utilizing a highly compressed set of low-rank adapters to compensate for precision loss. Furthermore, we address the challenge of outlier features in intermediate activations through an adaptive compression scheme that dynamically scales token-wise numerical ranges, enabling fully quantized matrix multiplications without severe degradation in downstream tasks. Through extensive empirical evaluation across multiple model families and deployment environments, the proposed joint optimization methodology demonstrates substantial improvements in both latency and memory utilization. The results indicate that the integration of low-rank error compensation and structural activation clipping yields a robust deployment paradigm for pervasive artificial intelligence applications at the edge.