The technical write-up behind my Deep Learning Indaba poster. Paper: "Deep Microcompression: Structured Pruning and Bit-packed Quantization for Microcontrollers," presented at GlobalSouthML @ ICML 2026. Full paper on arXiv.
The gap TinyML ignores
The promise of TinyML is to infer (run AI models) directly on edge devices, which saves power and secures data locally. However, the reality of deploying this hardware reveals a basic arithmetic problem. Look at what standard models demand compared to what legacy chips provide:
*activations are the output values generated by a layer after processing the input; they have to be stored on the RAM, a temporary storage for the computer and microcontroller to perform computation.
LeNet-5 is a 1998 Convolutional Neural Network (CNN) model for image classification. Running it is trivial on modern devices. Yet it requires roughly 3x the popular Arduino Uno microcontroller's entire memory (SRAM) and over 4x its flash storage, making it infeasible before a single line of program code is written. Running this model on the board is the baseline gap for the smallest serious benchmark. However, modern networks widen this gap exponentially.
We need to focus on that second row, because across the world, 8-bit microcontrollers like the ATmega328P are already powering legacy farming sensors and medical monitors in the field. Suggesting a Raspberry Pi Zero as an upgrade ignores deployment physics, partly because it requires running a full Linux OS, which drains batteries far too quickly for remote use. Making inference work on the $2 hardware practitioners currently own completely removes their reliance on cloud servers.
This forces a direct question. Is it possible to run a standard CNN on an ATmega328P-class hardware?
Why existing compression falls short
Deep Compression (Han et al., 2015) set the standard of model compression by combining pruning (the process of removing excess parts of a model), trained quantization, and Huffman coding. These techniques yield massive compression ratios; however, structural constraints prevent them from working on bare-metal hardware.
First, unstructured pruning strips out individual weights, creating irregular memory access patterns. Without a dedicated sparse execution path on the chip, standard dense kernels cannot process this efficiently.
Second, Huffman coding relies on variable-length data. Therefore, decoding it requires serial, data-dependent processing, making latency entirely unpredictable. However, real-time bare-metal execution requires strict deterministic timing.
TensorFlow Lite Micro is the standard off-the-shelf option, but it introduces over 250 KB of runtime overhead. That single footprint consumes more than 8x the ATmega328P's entire flash budget.
In summary, existing compression algorithms assume hardware infrastructure that microcontrollers simply lack.
The DMC pipeline
Our approach relies on three stages tailored directly to hardware limitations.
1. Structured channel pruning. Using an L2-norm magnitude criterion (Li et al., 2017), we dropped the least important convolutional filters and fully connected neurons, followed by retraining for accuracy recovery. This maintains a dense, contiguous memory layout. Thus, deterministic latency is preserved without requiring specialized sparse linear algebra libraries.
2. Quantization-aware training (QAT) with static quantization. QAT simulates discretization noise during training so the network adapts to lower precision. It pre-computes all scaling factors during calibration, which locks in the dynamic range. This creates a fully integer inference pipeline, completely bypassing the runtime overhead of dynamic quantization. We used ReLU to keep activations integer-friendly and approximated Softmax using precomputed flash lookup tables.
3. Fixed-length bit-packing. Computing bit-offsets at runtime via division and modulo is highly expensive on 8-bit and 16-bit cores. We pre-calculated the layout of every tensor at compile time. The extraction logic compiles directly into C code as basic shifts and masks.
Operation 4-bit 2-bit
i ÷ n_b i >> 1 i >> 2
b_offset ← (i mod n_b) × b i & 0b1 << 2 i & 0b11 << 1
Packing four 2-bit weights into a uint8 means decompression executes in constant time (O(1)) regardless of bitwidth. Huffman coding cannot provide this guarantee. Signed values run through a sign-extension step, and we matched the packed byte's endianness (storage order) to the target architecture to prevent runtime byte-swapping.
The final output is a completely dependency-free C header. We bypassed runtime interpreters and frameworks to prioritize absolute hardware portability.
Results: LeNet-5 on MNIST
Running LeNet-5 on the MNIST handwritten digit dataset is the ultimate stress test for 8-bit hardware. The neural network's early layers generate activation volumes nearly 3x larger than the total memory SRAM of a standard Arduino Uno.
Using layer-sensitivity analysis and neural architecture search, we generated two specific configurations:
- DMC-Ultra — tests the pipeline's limits for minimum weight storage.
- DMC-Tiny — maximizes accuracy within the strict 2 KB SRAM boundary.
Baseline and Ultra exceed ATmega328P resources, so we measured these rows on an RP2040 with software-emulated FPU for fair comparison.
Reframing the memory bottleneck
DMC-Ultra achieved a 55.8× weight compression. It reduced the 145 KB baseline down to 2.66 KB with 98.77% accuracy, enabling it to fit inside the 32 KB flash easily.
Surprisingly, it cannot run on the ATmega328P. The peak activation workspace hit 5.74 KB against our 2 KB SRAM limit. This led us to realise that weights were never the actual constraint.
This blind spot exists because activation memory is practically infinite on GPUs and mobile processors. The compression literature had naturally focused on parameter footprint as the definitive metric. However, when scaling down to microcontrollers, the physics invert. Weights become static storage, and live activations dictate whether the model can execute a single forward pass.
DMC-Tiny solves this through aggressive early-layer pruning. We cut the peak activation workspace to 1.27 KB, leaving roughly 700 bytes for the system stack and hardware drivers.
Here are the measurements on the physical Arduino Uno:
To the best of our knowledge, this represents the first documented deployment of a standard CNN on an ATmega328P.
We deliberately sacrificed weight compression, dropping from 55.8× to roughly 9× to achieve a viable activation footprint. Optimizing either dimension independently causes immediate failure in the other.
Direct Comparisons
Against TFLite Micro (LeNet-5, Int8, RP2040):
DMC produces a binary roughly 3× smaller at equal accuracy, running about 4× slower. TFLite relies on highly optimized CMSIS-NN kernels. DMC outputs generic scalar C code. We traded ARM-specific execution speed for the ability to run on legacy architectures entirely lacking CMSIS-NN.
Against prior ATmega328P work: LogNNet (Izotov et al., 2021) fits into memory but only achieves 84% accuracy at 7.11 s. Gural & Murmann (2019) hit 99.11% at 684 ms using 16-bit accumulation, downsampling inputs to 14×14, and keeping everything in single-cycle SRAM. We chose to stream weights from flash to support larger future models. Adopting their constraints would match their latency. We also prioritized architectural flexibility over single-task speed.
Two lessons from the ablations
Ablations were profiled using DMC-Ultra on an ATmega2560 because its 8 KB SRAM could actually host the 5.74 KB activation workspace. The 2 KB Uno could not.
- Compiler flags dictate deployment. The default -O0 build generates a 17.1 KB binary, threatening an 8-bit flash limit instantly. Switching to -Os cuts size by 38.7% down to 10.7 KB. Meanwhile, -O3 delivers the best latency and energy numbers (4.77 s / 1.41 J). Flag selection is a strict engineering decision rather than a background detail.
- Software unpacking creates a new bottleneck. Shifting from 8-bit to 4-bit packing halves storage needs but increases BOPs (bit operations) for unpacking during on-the-fly decoding. The energy and latency penalties scale with it. The space savings are also real, and the decode tax is heavy. Future low-power MCUs will require barrel-shifter-class hardware support to handle this efficiently.
Takeaways
- Measure the actual binding constraint. Hitting 55.8× compression looked excellent on paper but failed on the device because our objective function ignored SRAM.
- Match the pipeline to the hardware. We selected structured pruning, static integer quantization, and fixed-length packing specifically because a bare-metal MCU can execute them without overhead. A different chip demands a different pipeline.
- Portability justifies the latency tax. Running slower than CMSIS-NN is a worthwhile trade-off when the alternative is failing to run on legacy hardware altogether.
Future work will test transformer architectures against this same pipeline.