To perform the experiments, the PTB-XL dataset (Wagner et al., 2020) was selected. It contained 21837 12-lead ECG 10-second recordings collected from 18885 patients. The recordings were available at sampling frequencies of 100 Hz and 500 Hz. They were supplemented with metadata, including identifiers, patient and measurement information, detailed diagnostic data, signal information, and recommended folds that split the data into training, validation, and test sets. Here, the diagnostic labels were divided into five categories: NORM (normal), MI (myocardial infarction), CD (conduction disturbances), HYP (hypertrophy), and STTC (ST-T changes). These labels could co-occur for a single recording.

For the experiments conducted in the presented research, the ECG time-series data were prepared for multi-label classification in the following way. Overall, standard approaches were followed (Berkaya et al., 2018). Recordings at the sampling frequency of 100 Hz were used. A fifth-order Butterworth high-pass filter (Butterworth, 1930) was used to remove low-frequency noise below 0.5 Hz. Power-line noise at 50 Hz was reduced using a bidirectional moving average filter. Each of the twelve leads was normalized separately. The training set was used to calculate the values needed for normalization, and then the remaining data were normalized using the computed statistics. The splits into training, validation, and test sets were performed using the recommended folds included in the dataset. Each recording was assigned a binary label vector y of length 5, reflecting its membership in the diagnostic categories, as shown in (1). (1) y = ( 0 / 1 NORM 0 / 1 MI 0 / 1 CD 0 / 1 HYP 0 / 1 STTC )

In accordance with the recommendation provided by the dataset authors (Wagner et al., 2020), the metric for assessing and comparing the models was the macro-averaged (the score was calculated for each label and then averaged) area under the receiver operating characteristic (AUROC) (Fawcett, 2006). This popular metric was selected because it measures the ability of a classifying model to discriminate between classes in a threshold-independent manner, eliminating the need to select a specific decision threshold (either globally or per model), on which many other metrics depend. AUROC has an intuitive probabilistic interpretation that its value corresponds to the probability that a classifier assigns a higher score to a randomly chosen positive instance than to a randomly chosen negative instance (Fernández et al., 2018).

The following describes the NNs used in the experiments. The choice of the NN architectures was guided by their popularity in ECG classification. First, the RNN and the CNN architectures were created specifically for the purposes of this work, with their full, uncompressed versions intentionally designed to be compact (thus affecting the maximum compression rate). Next, the exact ResNet architecture was selected due to the superior benchmark results reported in Strodthoff et al. (2020). The chosen architecture types are generally common, supporting the usefulness of the gathered results. All NNs were implemented in PyTorch (version 2.2.2).

The RNN (Elman, 1990) was designed to capture the temporal dependencies inherent in ECG waveforms. The structure of the implemented RNN included an LSTM layer (Hochreiter and Schmidhuber, 1997) (with a hidden state dimension of 64), a layer normalization, a 1D max pooling layer (with stride and a kernel size of 125), a dropout layer (10%), a linear layer with 512 input features, and a sigmoid activation output. The 1D pooling layer divided the sequence of 1000 data points into eight segments (1.25 seconds each) and selected the maximum values. This reduced the data dimensionality and improved the results (by making the NN more robust to signal shifts). The model consisted of 22661 parameters. The Adam optimization algorithm was used, with an initial learning rate of 10 − 3 and a reduce-on-plateau scheduler that decreased the learning rate by half after every ten epochs without improvement, with a minimum learning rate of 10 − 5 . The batch size was 64.

The CNN (Lecun et al., 1998) was created for hierarchical feature extraction from the input signals. The proposed CNN consisted of four convolutional blocks. Each block included a 1D convolution layer (kernel size of 3), a batch normalization layer, a ReLU activation function, and a 1D max pooling layer (kernel size of 3). The number of output channels, for each 1D convolution, was 32, 64, 96, and 32, respectively. The NN ended with a dropout layer (5%), a linear layer with 352 input features, and a sigmoid activation function. The model consisted of 37157 parameters. SGD with Nesterov momentum ( μ = 0.995) was used for training. The L 2 penalty coefficient, λ, was experimentally set to 0.007. The learning rate was varied using a cosine annealing scheduler with a period of 30 epochs. The maximum learning rate was set at 10 − 3 , while the minimum was set at 10 − 6 . The learning rate was adjusted after each training epoch. The batch size was 64.

The ResNet (He et al., 2016) models were engineered to facilitate stable training in very deep models. The implementation of the 1D ResNet was based on the approach described in (Wang et al., 2017), with the following modifications. All shortcut connections were changed to 1D convolutional layers, with a filter length of 1 (He et al., 2016) (followed by a batch normalization) to enable pruning of all layers (Section 3.4). The output of the average pooling layer was passed to a linear layer with 128 input features, and the NN ended with a sigmoid activation function. The model consisted of 500869 parameters. The Adam optimizer was utilized. Cosine annealing was used to schedule the learning rate, with an annealing period of five epochs, a maximum learning rate of 10 − 3 , and a minimum learning rate of 10 − 6 . The batch size was 128.

To train the NNs for the multi-label classification, the cross-entropy loss (Goodfellow et al., 2016), shown in equation (2), was used. (2) L C E = − ∑ i = 1 n y i log ( p i ) + ( 1 − y i ) log ( 1 − p i ) . Here, n ∈ N denotes the number of labels, y ∈ { 0 , 1 } n represents the binary label vector from the dataset (Section 3.1), and p ∈ ( 0 ; 1 ) n is the vector of label occurrence scores returned by the model for the considered example.

Let us now describe the applied compression techniques, starting with quantization. In this work, uniform post-training quantization (PTQ) (Gholami et al., 2022), expressed in (3), was used. By default, static quantization was applied because it offers better inference time performance. (3) q ( x ) = r ( x β − α ( 2 b − 1 ) + z ) . Here, x represents the value to be quantized, q ( x ) denotes the quantized result, α and β define the range used for clipping values, b specifies the number of bits of the quantized values, z represents the zero point, and r rounds its argument to the nearest integer value.

Inference, using quantized models, was run exclusively on a CPU. The quantization backend used a combination of FBGEMM1¹

https://github.com/pytorch/FBGEMM

The implementation utilized observers that collected information on floating-point tensors passing through the NN during calibration (Gholami et al., 2022; Wu et al., 2020), and derived quantization parameters (α, β, z in (3)) based on these statistics. For monitoring weights, PyTorch’s PerChannelMinMaxObserver was used. It identified the minimum and maximum values for each channel, along the specified axis, to derive quantization parameters. The weights followed a symmetric quantization scheme (Gholami et al., 2022) with per-channel granularity (Wu et al., 2020). They were represented as 8-bit signed integers, with range [ − 127 ; 127 ] (sacrificing one value for symmetry).

The activation observer was PyTorch’s HistogramObserver, which minimized the error between the floating-point and the quantized data distributions. Layer activations used the asymmetric quantization scheme (Gholami et al., 2022) with per-tensor granularity (Wu et al., 2020). Their quantized form was represented using 8-bit unsigned integers, although 7 bits were used (range 0 to 127) to avoid saturation (Li and Alvarez, 2021) caused by the assembler instruction VPMADDUBSW in the FBGEMM library implementation (AVX-512 VNNI instructions were not available on the CPU used). The exception was the output sigmoid activation function, which used the full 8-bit range.

The LSTM layer was handled differently. PyTorch’s MinMaxObserver was used to determine the quantization parameters for the weights. It found the quantization parameters collectively for all weights at once (using static quantization). Thus, it was per-tensor granularity, where the quantization parameters were shared for all weights. The remaining weight quantization configuration was the same as before. Here, activations were dynamically quantized (Gholami et al., 2022) during inference, so the quantization parameters were based on the values observed at that moment. The remaining activation quantization configuration was unchanged.

Additionally, operator fusion was used to combine the convolutional layer, the batch normalization layer, and the ReLU activation function (if it occurred in the sequence) into a single layer. Finally, 128 randomly selected examples from the validation dataset were used to calibrate all models.

A structured variant of pruning (Cheng et al., 2024) was chosen, because it does not require specific hardware to observe the acceleration benefits. It was implemented for the NNs proposed in Section 3.2.

The LSTM (Hochreiter and Schmidhuber, 1997) layer consists of the input-hidden weights W x i , W x f , W x c , W x o ∈ R r × d and the hidden-hidden weights W h i , W h f , W h c , W h o ∈ R r × r . Where d ∈ N is the number of input features and r ∈ N is the number of features in the hidden state. At time step t ∈ N, the weights are used to compute the input gate in (4), the forget gate in (5), the cell gate in (6), and the output gate in (7). Here, x ( t ) ∈ R d is the input vector at time t, and h ( t − 1 ) ∈ R r is the hidden state at time t − 1. For brevity, the bias terms (b) are omitted from the considerations because they follow the pruning pattern derived from the weights in the solution. (4) i ( t ) = σ ( W x i x ( t ) + W h i h ( t − 1 ) + b x i + b h i ) , (5) f ( t ) = σ ( W x f x ( t ) + W h f h ( t − 1 ) + b x f + b h f ) , (6) c ( t ) = tanh ( W x c x ( t ) + W h c h ( t − 1 ) + b x c + b h c ) , (7) o ( t ) = σ ( W x o x ( t ) + W h o h ( t − 1 ) + b x o + b h o ) . Therefore, the weights related to the s-th neuron in the input, forget, cell, and output gates can be expressed as W s ∈ R 4 × ( d + r ) in (8), where s ∈ { 1 , … , r }. (8) W s = W x i s 1 … W x i s d W h i s 1 … W h i s r W x f s 1 … W x f s d W h f s 1 … W h f s r W x c s 1 … W x c s d W h c s 1 … W h c s r W x o s 1 … W x o s d W h o s 1 … W h o s r . For each W s , either the L 1 or the L 2 norm was calculated. The neurons with the lowest values were pruned (thus, the weights and the biases associated with these neurons were removed).

An approach similar to (Li et al., 2017) was applied to prune the convolutional layers. Consider a 1D convolutional layer, where weights are stored in a 3D tensor of size O × I × K. Here, O is the number of output channels, I is the number of input channels, and K is the length of the filter (the bias terms are again omitted for brevity). Layer output channels are formed independently, so let us consider a single output channel, a matrix W s of size I × K, where s ∈ { 1 , … , O }. Now, suppose that u ∈ { 0 , … , I − 1 } input channels become unavailable. Then, these missing channels carry no information. Hence, effectively, the matrix W s becomes W s ′ of size ( I − u ) × K, without the rows corresponding to the unavailable channels. During pruning, W s ′ were considered, thus avoiding missing input channels from earlier layers (Li et al., 2017). Similarly to the LSTM layer, for W s ′ values, the norm L 1 or L 2 was calculated. Then, the layer’s output channels with the smallest values were pruned. In the ResNet, its shortcut connections followed the pruning pattern of the last convolutional layer in their residual blocks.

The pruning procedure started from the input layer and proceeded sequentially until the output layer. For the RNN model, pruning began with the LSTM layer. Based on the norm values, calculated for W s (8), neurons were removed, and the pruning schema was passed to the rest of the NN. For the CNN and the ResNet, in a convolutional layer, after receiving information about the unavailable input channels, the selected norm of W s ′ was computed, and the output channels, with the lowest norm values, were removed and the pruning schema was passed down deeper into the NN. The remaining layers, such as the batch normalization layers, the linear layers, etc., reacted to the missing channels by pruning the appropriate weights and biases, in effect, shrinking as well.

Ideally, the model reduction achieved with structured pruning should lower the computational load, e.g. measured in simple hardware-agnostic proxy metrics such as FLOPs (floating-point operations), making the inference faster. However, in practice, structured pruning may lead to a suboptimal configuration of NN structures (Dong et al., 2021; Liberis et al., 2021). This suboptimality manifests itself in not utilizing the full potential of the underlying hardware and the library implementations, e.g. highly optimized vectorized kernels, effectively leading to notable slowdowns.

During the experiments, pruning was performed in one or more rounds to achieve the same degree of structural reduction measured in the fraction of structures left in the final pruned model. The fraction of available structures pruned in a single layer in one round was expressed as p = 1 − f 1 / r , where r ∈ N denoted the number of pruning rounds and f ∈ ( 0 ; 1 ) was the target fraction of structures left after all rounds (note the difference between weights and structures). In the solution, f was set to 75%, 50%, and 12.5%; r was 1, 5, or 10.

After each pruning round, the models were fine-tuned, using generally the same approach as for training the full (uncompressed) NNs, or using knowledge distillation (Section 3.5). However, for the CNN model, the maximum learning rate was reduced to 5 · 10 − 4 and the annealing period was changed to 20 epochs. The RNN and the ResNet configurations were unchanged.

Overall, response-based knowledge distillation (Gou et al., 2021) was used, offering a simple and general approach. However, the typical scheme introduced in (Hinton et al., 2015) suits multi-class, not multi-label classification. Thus, the authors of (Yang et al., 2023) proposed applying response-based knowledge distillation with one teacher and one student to multi-label classification, by decomposing the multi-label task into independent binary classification tasks. In these tasks, two two-element probability distributions (of the student and the teacher) are created by taking the probability of label occurrence and its complement and, finally, expressing the loss function as the divergence ( D) between these distributions. This loss function, L M L D (Yang et al., 2023), is presented in (9), where the superscript t denotes the teacher’s output and s, the student’s output. See (2) for the remaining notation. (9) L M L D = ∑ i = 1 n D ( [ p i t , 1 − p i t ] | | [ p i s , 1 − p i s ] ) .

Ultimately, the loss function followed expression (10). In L C E (2), p was set to be the student’s output, p s . As divergence, D, in (9), Kullback-Leibler divergence (Kullback and Leibler, 1951) was used. For all models, the best coefficient α was empirically determined to be 0.4. (10) L K D = α L C E + ( 1 − α ) L M L D .

During the experiments, the teacher model was a full floating-point model trained without knowledge distillation with the same base architecture as the student. The student was either a new instance of an uncompressed model to see if the approach can improve the base models’ results, or a pruned (smaller) model to observe the impact during the fine-tuning stage.

For memory allocation, instead of malloc, tcmalloc was used for the RNN, and jemalloc for the CNN and the ResNet models, as they offered slightly faster performance.

The average inference time was calculated for the models running on the CPU (Section 4). Although it may not be correlated with the execution on other hardware, clock time was selected in order to show the actual latency that captures various factors, such as memory access or kernel launch overheads. The number of threads available to PyTorch was limited to 1 to mitigate the influence of parallel execution, which was beyond the scope of this research.

Before the time measurements, the models were warmed up on 100 examples. The batch size was set to 1. Inference time was measured for 10000 examples for the CNN, and 2000 examples for the RNN and the ResNet, and the results were averaged. In PyTorch, the floating-point version of the RNN model can use an efficient implementation for the LSTM layer from the oneDNN library (formerly known as MKL-DNN and DNNL). Therefore, the floating-point RNN inference time was measured twice, with and without access to the library.

The models’ size was measured by storing PyTorch’s state_dict on the disk and then obtaining its size. Note that, in this representation, additional data are stored in the created file, such as information about tensors representing weights or quantization parameters. Therefore, the reported size was worse than the ideal theoretical reduction (i.e. a 4-fold reduction in the case of quantizing the model from 32-bit floating-point to 8-bit integer representation).

What follows is a summary of the results obtained by the RNN during the experiments. First, we analyse the AUROC results in Table 1. Models M F , M FKD , and M QKD achieved their best scores at 61.45% of the original parameters left, demonstrating that the applied pruning method (both with and without knowledge distillation) has the potential of improving the quality of the base model. This behaviour can be attributed to the regularizing effect of pruning (Hohman et al., 2024). However, more aggressive parameter reduction that shrank the models to 31.94% of parameters resulted in degraded quality, and further reduction to 4.61% led to a poor fit of the models. Therefore, practitioners must carefully consider the trade-off between the model size and the predictive quality while applying pruning.

The models fine-tuned with knowledge distillation (M FKD , M QKD ) outperformed their respective counterparts fine-tuned without knowledge distillation (M F , M Q ) in terms of predictive quality across all pruning levels. Interestingly, for the models M FKD and M QKD at 31.94% and 4.61% different pruning approaches resulted in highest scores (i.e. 5/ L 2 versus 10/ L 2, and 5/ L 1 versus 1/ L 2). This observation indicate that the best quantized model does not necessarily derive from the highest-scoring floating-point model.

Next, Fig. 1 presents the average inference time of the both floating-point (M F , M FKD ) and the quantized (M Q , M QKD ) models during pruning. The time measurements of the floating-point models were performed with and without access to the oneDNN library (Section 3.6). As expected, the floating-point models without access to the oneDNN implementation were slower than their quantized counterparts. Moreover, the floating-point models using the oneDNN library were significantly faster than the other two (those without access to the library and quantized ones), thus showing that quantization may not always be universally advantageous in terms of performance. However, it has to be stressed that this conclusion is likely to be platform-specific. Additionally, since during pruning, the absolute differences of latency improvements for floating-point models were similar to each other and more significant than the ones observed in the quantized models, the biggest relative improvements were observed for the floating-point models with access to the oneDNN library.

During pruning, the size of the models decreased linearly; the same was true also for the CNN and the ResNet. The floating-point RNN models shrank from 93.677 KB to 7.149 KB, while the quantized models decreased from 31.194 KB to 7.770 KB. Therefore, the storage advantage of using quantization gradually diminished with size, to a certain point where the quantized NN actually consumed more disk space, due to additional overhead data stored on the disk (Section 3.6). Figure 2 visualizes these size measurements throughout pruning.

Next, we discuss the results obtained by the subsequent proposed model, CNN. The data in Table 2 summarize the AUROC scores obtained by the CNN at different target stages. In contrast to the patterns observed for the RNN (Section 4.1) and ResNet (Section 4.3), the CNN models suffered a noticeable drop in AUROC scores once parameters were removed. The smallest degradation appeared in the models trained with knowledge distillation, suggesting that it helped the pruned networks to retain part of the teacher’s representational power. However, the benefit was not uniform. At 27.27% parameters left, the model fine-tuned with knowledge distillation, M FKD , was slightly wore than its non-distilled counterpart, M F . Furthermore, at a high sparsity level of 2.56% of parameters left, M Q was a bit better than M QKD . Another aspect to note in the results is that different combinations of pruning rounds and norms led to the best-performing floating-point and quantized models at the same pruning level. For instance, at 57.95% of parameters left, the best results for the floating-point model, M F , were achieved with 10/ L 2 pruning method, but the best results for the quantized model, M Q , were achieved after quantizing the floating-point model obtained with 1/ L 1 pruning method. Therefore, one cannot always assume that among multiple floating-point models, the one with the best results will still be the best after quantization.

Although relatively small and compact, the CNN architecture used in the reported experiments turned out to be well-performing on the given (ECG) dataset. In fact, pruning the CNN so that only 27.27% of its original parameters remained, it still outperformed the best RNN results. Even more encouraging is that the base M FKD CNN reached the AUROC score that surpassed the best single model scores reported by (Strodthoff et al., 2020) for the very same task. These findings suggest that a carefully designed, lightweight CNN can match (or even beat) larger and more complex models on ECG classification while using a fraction of the parameters, highlighting the importance of tailored architecture design.

Next, Fig. 3 illustrates the inference latency of the CNN variants in the experiments. Initially, the quantized models outperformed their floating-point counterparts, achieving lower inference times on the tested hardware. However, during pruning, the trend shows that the models performed worse after quantization, mainly due to the degraded performance of the quantized max pooling operation with less than 32 channels, causing quantized models to exhibit higher latency than the floating-point versions despite their reduced arithmetic precision.

The models’ on-disk size measurements are presented in Fig. 4. The original floating-point networks occupied 159.734 KB; the smallest models required only 13.174 KB of storage. The quantized networks started at 51.962 KB and were pruned down to 11.898 KB. Thus, even though the pruned quantized models incurred a latency penalty, they consistently maintained a smaller on-disk size compared to their floating-point equivalents.

The following section summarizes the AUROC results obtained by the proposed ResNet-based architecture (Table 3). Several insights emerge when we compare unpruned baselines, pruned variants, and models trained with knowledge distillation. It is also worth mentioning that this (baseline) model contained significantly more parameters than the other two analysed NNs.

First, pruning positively impacted some of the scores. The removal of about half of the parameters yielded slight AUROC improvements for models M F , M Q , M FKD . Similar to the RNN, this increase can be attributed to the regularizing effect of pruning, which mitigates overfitting and encourages the network to focus on the most informative feature maps. In contrast, M QKD experienced a slight drop in AUROC following a comparable pruning. Interestingly, it was observed that M F pruned in a single round to 56.60% and 25.47% of parameters left performed better than the same model pruned (and fine-tuned) in 5 or 10 rounds, which led to over-fitting; this was not observed for M FKD . However, M F and M FKD shared a mutual behaviour for much more disruptive pruning, to 1.77% parameters left, where a gradual approach with more rounds offered superior results compared to faster pruning in fewer rounds. Moreover, across the experiments, the model variants obtained using knowledge distillation (M FKD and M QKD ) consistently surpassed their non-distilled counterparts. This behaviour demonstrated that transferring label information during training not only enhances generalization but also protects the models from the adverse effects of parameter removal. Finally, it is worth noting that the unpruned baseline M F achieved the AUROC score of 0.9282, matching the performance reported in (Strodthoff et al., 2020) for the same task using an analogous model that scored 0.930. This parity can be treated as a confirmation of the validity of the proposed implementation.

Next, Fig. 5 shows how inference latency for ResNet models changed during the experiments. For the vast majority of pruning levels, the quantized models were noticeably faster than their 32-bit floating-point counterparts; the reduced arithmetic complexity translated directly into lower inference times. However, near the maximum level of pruning, when the remaining parameter count (and thus model size) of the ResNet is roughly on par with that of the analysed baseline CNN (both architectures were convolution-oriented), the observed trend flipped. In this ultra-pruned regime, the quantized ResNet actually incurred slightly higher latency than the floating-point version, a behaviour that mirrored the pattern observed for the CNN in Fig. 3.

The initial floating-point models occupied 2.042902 MB at their largest and were pruned down to just 0.065302 MB in their most compact form. As anticipated, applying quantization reduced these footprints even further, and the quantized models consistently outpaced their floating-point counterparts in storage efficiency. Their size ranged from 0.553758 MB (100% of parameters) to 0.041374 MB (1.77% of parameters left). The trends are visualized in Fig. 6.

The uncompressed, full-precision versions of each proposed NN achieved classification performance (measured by macro-averaged AUROC) that closely matched or slightly exceeded the benchmark results reported in Strodthoff et al. (2020). Moreover, these types of NNs represent different computation characteristics that range from the sequential processing of RNNs to the shortcut-connected layers of ResNets. Therefore, they were deemed good candidates for the subsequent experimentation with the compression techniques, which were the focal point of this work.

In the conducted experiments, in almost all tested cases but one, quantization reduced the models’ storage footprint owing to the use of lower precision numbers to represent the weights, i.e. 8-bit integers instead of 32-bit floating-point parameters. In theory, the use of this kind of quantization should result in a four-fold reduction in the on-disk size storage requirements, yet in practice, we observed smaller (worse) savings. Moreover, for the heavily pruned RNN, the quantization was detrimental size-wise, i.e. it resulted in a bigger footprint than the full-precision model. Both the discrepancy between the actual and four-fold reduction and the RNN exception were traced back to the implementation details of PyTorch and the method used to measure the disk usage (Section 3.6).

Despite making the models smaller and using a representation that theoretically offers faster computation with integer arithmetic, the effect of quantization on the inference speed on the tested CPU was variable, did not always result in faster inference, and heavily depended on the model architecture, the NN size, and the libraries used. Quantized RNN models were faster than their full-precision counterparts. However, using the oneDNN library for the floating-point models proved to be much faster and outperformed even the integer variants. Next, quantized CNN models experienced a slowdown at the beginning of the pruning. Although the inference speed kept improving, they remained slower than the full-precision models. In effect, during pruning, the floating-point CNNs were superior to the quantized models in terms of inference speed. The ResNet architecture, which was the largest network in the experiments, benefited the most from the quantization. However, once it reached a size comparable to that of the used CNNs, a similar slowdown occurred that resulted in the quantized models being slower than their full-precision counterparts. These results underscore that the impact of quantization on speed is highly context-dependent. However, it is important to note that the results could also differ on other hardware, e.g. without a dedicated floating-point unit, or in other runtime environments, e.g. due to the specifics of quantized kernel implementations.

As expected, replacing floating-point 32-bit weights with quantized integers degraded the predictive quality of the tested models. However, the exact magnitude of impact on the AUROC score depended on the NN architecture and the NN size. The largest considered model, ResNet, suffered the biggest negative impact on the relative scores between non-quantized and quantized models. At the same time, the most negligible negative effect was observed for the RNN architecture, which had the smallest number of parameters. In the case of the CNN architecture, the penalty resulting from quantization was between the other two architectures. Nonetheless, in all cases, the loss in predictive quality was gradual rather than catastrophic. Potentially, more sophisticated calibration or quantization-aware training could help recover part of the gap.

Structured pruning was naturally beneficial for reducing the models’ disk usage because, in the proposed implementation, the pruned structures were actually removed from the network and not merely masked. As expected, the decrease followed a linear trend as a function of the number of parameters left in the model.

Beyond storage savings, structured pruning yielded improvements in inference latency across all architectures, although the magnitude and pattern of these improvements varied. Overall, floating-point models experienced more substantial speedups than their quantized counterparts. However, the exact impact was more nuanced between the tested architectures. Across floating-point and 8-bit quantized RNNs (both with and without access to the oneDNN library), pruning reduced inference time and models followed a similar pattern. It should be noted that the quantized models were faster only than the floating-point models that did not use the oneDNN implementation. For the CNN architecture, the quantized models were faster only at the very beginning of the experiments. Moreover, the latency of the floating-point models decreased more steadily than that of the quantized ones. As the models became smaller, the more erratic graph of the quantized models’ latency diverged from the floating-point models’ graph even more, underlining the runtime overheads of the quantized kernels implementation. For the largest tested architecture, ResNet, during pruning, the quantized models remained faster than the floating-point models, up to about 10% of parameters left in the models. Beyond that point, a behaviour was observed that was similar to the one described for the CNN architecture.

Additionally, an interesting side finding emerged when we examined pruning patterns whose remaining structures aligned with multiples of eight. NNs with a dominant number of structures divisible by eight were faster than their slightly bigger or smaller versions, e.g. a layer with 32 output channels had a faster execution than a layer with 33 or 31 channels. This “hardware sweet spot” pointed to the phenomenon of a suboptimal configuration (Section 3.4) and the need to carefully consider the target hardware and runtime implementation while choosing the pruning pattern to obtain the lowest latency.

Surprisingly, pruning could improve the results obtained by the models, which was observed in the case of the RNN and the ResNet architectures. This behaviour could be attributed to the regularizing effect of pruning. Hence, it was not observed in the CNN, as it was regularized and quasi-optimally small, so further regularization only had adverse effects.

Finally, no significant differences were noted during the experiments between the L 1 and the L 2 methods that chose the structures to be pruned. These observations were similar to the results reported in Li et al. (2017). Instead, the number of pruning rounds, followed by fine-tuning, turned out to be a more important parameter. However, conclusions about the exact impact of pruning rounds remains an important direction for future work.

The knowledge distillation scheme utilized in the experiments had no direct impact on the model size or its inference latency. Instead, its primary benefit was improving the predictive quality of the base and pruned models (and therefore the quantized models), with only a few exceptions. In practical terms, this means that if a predefined quality acceptance threshold is present (e.g. a maximum allowable drop in accuracy), the use of knowledge distillation makes it possible to achieve more aggressive compression ratios (e.g. by applying pruning).

Our study deliberately focused on small networks of moderate depth. While response-based distillation, where the student mimics the output of the teacher, has proved to be effective in this context, prior work suggests that it may be less suitable for very deep NNs (Gou et al., 2021). In such cases, one might need to resort to other distillation schemes, e.g. matching activations from intermediate layers. Because the experiments did not include very deep models, a thorough investigation of alternative distillation paradigms remains beyond the scope of this work.

Although this study covers various aspects of neural compression methods, its shortcomings must be realized. One significant limitation comes from the use of a single dataset, PTB-XL. It might introduce bias in the findings, which is associated with the specifics of the data, such as bias related to the demographics, the acquisition methods, or the representation of conditions. Another constraint arises from the runtime environment settings for the experiments. The inference latency results were obtained using a standard CPU, which may be too demanding for environments with strictly limited computing resources. Additionally, the experiments were performed using the full PyTorch runtime, which comes with requirements that may be unrealistic for tiny devices. For instance, it requires using an operating system or involves dynamic memory allocation. Other machine learning runtimes can be better suited for small computing devices, e.g. LiteRT (TensorFlow Lite) for Microcontrollers.3³

https://ai.google.dev/edge/litert/microcontrollers