Unifying Depth and Width Pruning for LLMs via Binary Knapsack Optimization

1.1Modern LLMs are HUGE!#

Developments in AI have given Large Language Models (LLMs) exceptional linguistic skills and reasoning capabilities. Unfortunately, these abilities come at a significant cost: modern LLMs have grown to enormous sizes (sometimes containing trillions of parameters!), making them quite expensive to run. In fact, many models are simply too large to fit on everyday devices such as laptops & phones, and require specialized hardware to run.

1.2LLM Pruning, A Solution#

Researchers have figured out a neat idea to reduce the size of these huge models: simply remove components from the model that are relatively unimportant in a way that preserves as much performance as possible. This is called Structured Pruning.

Structured pruning methods can be broadly classified into 3 categories based on the axis along which they prune models.

Irrespective of the pruning axis, many methods follow a similar operational recipe:

1. Compute an importance score for each prunable component
2. Remove the components with the lowest importance scores (or equivalently, retain components with the highest importance scores)

1.3Limitations#

While the aforementioned modus operandi is simple enough, the authors argue that there are some problems with it.

1. Local Heuristics: Several methods tend to rely on importance scores that only represent how good a component is in isolation without measuring how good it is in the presence of other components. As a result, a component that works well in combination with other components might still get a low importance score if it is individually weak.
2. Myopic Component Selection: The decision to remove or retain a component depends solely upon whether its importance scores is among the lowest or not. However, such greedy decisions do not account for cases where retaining a weaker component (which would otherwise be removed) can lead to a stronger pruned model down the line. In other words, greedy decisions are short-sighted.
3. Imprecise Adherence to Target Budget: Many existing pruners suffer from a practical and fundamental problem - they don't meet the target compression ratio precisely! Asking the existing tools to compress a model by 35% can yield a model that is 19-30% smaller. If you have 16 GB of memory to fit a model into, a tool that quietly leaves the model 30% larger than requested is not much use.

4. Calibration-Induced Deviations: Most existing pruners use some data (aka. calibration data) to compute importance scores. However, they often tend to rely so much on this data that the pruned model often exhibits inconsistencies in performance across a variety of tasks, performing well at some but disproportionately bad at others.

This paper proposes SNIPER, a novel Structured Knapsack-optimization-based Pruner that aims to resolves these limitations.

SNIPER is a dual-axis approach that first prunes along the model's depth and then uses width pruning to meet the target compression ratio precisely.

2.1Stage 1: Depth Pruning#

SNIPER's depth pruning strategy can be thought of as packing a suitcase while ensuring its weight limit is not exceeded.

Every component of the model has a weight (how many parameters it costs) and a value (how important it is). The compression budget gives you a fixed capacity for the pruned model (the suitcase). You want the most valuable set of components that fits.

This objective is exactly the knapsack problem and has a known, exact solution via dynamic programming (DP).

Unlike greedy methods which only pick the locally best choice at each step, DP explores all available choices to ensure that future results are optimal.

The prunable components are the attention block and the feed-forward block of each layer, plus the option of keeping the layer whole. Keeping a whole layer and keeping its two halves are mutually exclusive choices, so they're grouped into a single decision.

2.2Importance Estimation#

While computing a component's weight is easy (simply count how many parameters it holds), the harder question to answer is: how do you compute each component's importance?

A popular answer is marginal degradation (aka. leave-one-out scoring). From the original model, temporarily delete one component, and measure the drift in the model's predictions. Put it back, and repeat for each remaining component. The bigger the drift, the more important the component must have been.

SNIPER uses the model's logits to measure this drift. Let \mathbb{Z} be the logits the original model produces on a batch of calibration data. For component x_i, it compares two versions of the model: one where the component is retained, one where it is dropped. The authors take the difference in how far each has drifted from \mathbb{Z} to quantify how important x_i is:

v(x_i) = \lVert \mathbb{Z} - \mathbb{Z}^{(i)}_{\text{drop}} \rVert_2^2 - \lVert \mathbb{Z} - \mathbb{Z}^{(i)}_{\text{retain}} \rVert_2^2

This formulation might seem similar to marginal degradation and if x_i is kept/removed from the original model, then \mathbb{Z} = \mathbb{Z}^{(i)}_{\text{retain}} and the equation becomes:

v(x_i) = \lVert \mathbb{Z} - \mathbb{Z}^{(i)}_{\text{drop}} \rVert_2^2

which is exactly, a marginal degradation-style formulation. However, SNIPER differs at exactly this assumption: it doesn't just use the original model to compute its heuristics.

2.2.1Scoring on the model as it is actually being pruned#

SNIPER estimates importance iteratively: the score for component x_i is measured not on the original model but on M^{(i-1)}, the model as pruned up to the first i components. Each score is therefore conditioned on the decisions already made, and a component whose partner has just been removed immediately becomes expensive rather than staying cheap.

The result is an importance estimation scheme that evolves with various pruning decisions instead of comparing against a stationary model.

Unfortunately, this method does not come without its limitations. Conditioning importance estimates on iterative pruning decisions makes these scores indirectly dependent on the model's compression ratio. This means that unlike marginal degradation-style heuristics which can be computed once for each model and re-used at any compression ratio, iterative importance estimates must be computed separately at each ratio, making them more computationally expensive. The authors provide an analysis wherein they show that it is practically viable to bypass this limitation to an extent.

2.3Stage 2: Width Pruning#

Stage 1 operates on discrete components of fixed sizes. As a result, it leaves some of the compression budget unused. In order to fill this residual capacity, SNIPER prunes each MLP's width by removing individual neurons from it. This offers much finer control over the number of pruned parameters.

However, for this stage to be successful, two questions need to be answered:

1. How do we rank each MLP's neurons?
2. How many neurons do we remove from each MLP?

2.3.1Neuron-Importance Estimation#

While SNIPER's iterative importance estimation step is quite useful in stage 1, it relies on the number of prunable components being relatively small (often <100). The number of neurons, however, is multiple orders of magnitude higher, making it infeasible to gauge the effect of removing each neuron one at a time.

Therefore, SNIPER resorts to a more straightforward heuristic to rank neurons. The authors define each neuron's sensitivity as the magnitude of its contribution to the MLP's output. The idea is simple: the higher a neuron's sensitivity is, the more important that neuron must be.

This is an extremely cheap-to-compute heuristic since its computation does not require any data (only raw weights are multiplied) and can easily be vectorized.

2.3.2Budget Allocation#

Having ranked all neurons, the only thing left to do is to compute how many neurons must be removed from each MLP. To do so, the authors make use of each layer's importance score, as computed via iterative importance estimation.

The intuition: layers that are more important receive less of the total compression budget, i.e., are pruned less.

3.1Models Used#

The authors conduct experiments on 4 diverse model architectures:

1. Qwen3-8B: Reasoning-oriented model
2. Llama-3.1-8B-Instruct: Instruction-tuned model
3. Phi4-14B: Fused MLP projections used instead of separate projection matrices
4. GPT-OSS-20B: Mixture-of-Experts (MoE) model

Qwen3-8B and Llama-3.1-8B are tested under pruning ratios of 25% and 35% while Phi4-14B and GPT-OSS-20B, being bigger models, are tested only under the more aggression 35% compression.

3.2Baselines#

The authors compare SNIPER's performance with that of the following baselines:

1. Width Pruners: SliceGPT & LLM-Pruner
2. Depth Pruners: ReplaceMe, SLEB & ShortGPT
3. Dual-Axis Pruner: 2SSP

3.3Evaluation Criteria#

In the absence of a single evaluation setup for pruning methodologies, the authors evaluate all methods against a comprehensive set of metrics, spanning 5 diverse domains.

Many pruning methods benefit from a small post-compression fine-tuning phase to allow the model to recover from the significant distributional shift induced by the removal of its components. This is known as recovery fine-tuning (RFT). The authors test SNIPER under both no-RFT and RFT settings.

4.1Performance Retention#

SNIPER wins at 9 out of 10 tested configurations. The only configuration where it doesn't win is when pruning LLaMA-3.1-8B-Instruct by 25% with RFT, where it is only marginally outperformed by ReplaceMe and ShortGPT.

It adapts better to larger and more complex architectures. It consistently outperforms its competitors by margins of up to 11.42\% and 16.48\% on Phi-4-14B and GPT-OSS-20B, respectively.

4.2Performance Consistency#

In addition to demonstrating excellent performance retention, SNIPER demonstrates unparalleled consistency in its performance.

Unlike its baselines, SNIPER never collapses. Each baseline has at least one configuration where it disintegrates completely, falling to being one of the worst performing methods for that configuration. SNIPER, on the other hand, tops the rankings at a nearly perfect mean ranking of 1.25, sliding down to a rank of 3 at its worst.

It is also steadier across tasks. A pruned model may perform well on average while being disproportionately bad on a few specific tasks. Measuring the spread across tasks, SNIPER is the most consistent method in every configuration, including the ones where it doesn't win on average.

4.3Ablation Study#

To analyze whether each of component of SNIPER is even required or not, the authors conduct an ablation study where they test its performance after removing components one at a time.

Depth pruning (stage 1) alone demonstrates strong performance. However, it lags behind SNIPER since its removal of large components can end up removing important individual neurons that are present in them. This leads to noticeably higher instability in the form of its higher standard deviation across tasks.

In contrast, width pruning (stage 2) alone is relatively more precise and stable with lower standard deviation. When both stages are operating together, stage 1's conditionally optimal decisions are likely able to balance the greedy decisions made by stage 2. However, in the absence of the first stage, the pruning pipeline ends up relying too much on these greedy decisions, leading to subpar performance.

4.4Efficiency Gains#

Apart from reducing model size, pruning has another important benefit: it oftens produces models that perform faster than their base counterparts. Therefore, the authors analyze the speed-ups that each pruning method induces.

SNIPER yields competitive inference speed-ups, sitting right between width and depth pruning methods. Interestingly, 2SSP is able to beat even depth pruners but that's likely because even during its width pruning phase, it bypasses the irregular tensor shape problem by simply pruning the same number of neurons from each MLP. Unfortunately, this is a cost that SNIPER has to bear due to its dynamic budget allocation.

While the main results have been summarized in the previous section, there are some other interesting details that the authors analyse.

5.1Runtime Analysis#

The authors also analyze the time it takes for SNIPER to prune a model. Firstly, as discussed earlier, SNIPER divides all parameter counts by a discretizing factor, \alpha. This reduces its time complexity to O(\lfloor \frac{NC}{\alpha} \rfloor).

However, this comes with a tradeoff to manage: if \alpha is too big, parameter counts may end up being scaled down so much that components of vastly different sizes can end up being assigned the same weight. On the other hand, if it's too big, it won't reduce stage 1's time complexity enough, making it slow and impractical.

A simple sweep of different values of \alpha reveals 32 to be a good enough conservative estimate that balances this tradeoff fairly well.

Next, they compare SNIPER's runtime with a width pruner (SliceGPT) and a depth pruner (SLEB). They find that SNIPER is only marginally slower than SLEB with most of its time going towards computing the importance estimates from stage 1.

5.2Transferability of Importance Scores#

Seeing how importance estimation is the most computationally expensive step in SNIPER's pipeline, the question arises: can we re-use importance estimates across compression ratios?

Turns out, the answer is yes! While transferring scores across compression ratios leads to weaker performance than using the compression-specific scores, the drop in performance is not catastrophic by any means. This makes it viable to compute importance scores once on a compression ratio and re-use them as required.

5.3Interpretable Importance Patterns Across Layers#

Quite interestingly, the scores computed via iterative estimation show a clear intuitive structure where early and late layers are given higher importance and intermediate layers are deemed relatively less important.

Moreover, there is a substantial overlap in the layers that are evicted across multiple compression ratios. This indicates that these importance estimates truly capture architectural importance rather than being influenced purely by the available budget.

5.4Consistency Across Different Calibration Setups#

Limitation #4 points out that several pruners tend to over-rely on the calibration data used by them, causing them to perform disproportionately badly on out-of-distribution tasks. To analyze whether SNIPER also suffers from this issue, this paper observes how its performance changes with changes in:

• The number of calibration samples
• The calibration data distribution

SNIPER maintains excellent consistency across all calibration paradigms, with its average performance across each configuration differing by less than 1\%. Its per-task standard deviation is also consistently the lowest and does not peak significantly when changing the number of calibration samples or their distribution.

Other ways of making models smaller leave the architecture intact: quantization stores each number with fewer bits, and distillation trains a small model to imitate a big one. Both save memory, but the model still has the same number of layers to walk through, so they don't speed it up proportionally.

Unstructured pruning creates sparse weight matrices by zeroing out individual weights in the model. It preserves accuracy well but needs special hardware to actually run faster. Structured pruning removes whole coherent pieces, leading to speedups on ordinary GPUs.

Within structured pruning, methods have historically picked a side: depth or width. A few recent hybrids use both, but they still make their choices greedily, one deletion at a time. SNIPER's departure is not that it uses both axes; it is that it stops choosing greedily.