The encoding part of the proposed shallow autoencoder network performs the CS measurement process on an input image, while the decoding part models the CS reconstruction process and reconstructs the input image from the low-dimensional measurement space (Fig. 1).

In the traditional CS measurement process, an image is vectorized to form a one-dimensional vector  x∈RN and is projected into a low-dimensional measurement vector  y∈RM using an inner product with a collection of measurement vectors  {ϕm}m=1M : (4) ym=⟨ϕm,x⟩= ∑i=1Nϕm,ixi. The measurement matrix Φ is created by arranging the measurement vectors  ϕmT as rows. Signal dimensionality (i.e. image dimensions) determines the number of columns in the measurement matrix. Consequently, when image dimensions are large, a block-based CS approach is suitable since it operates on local image patches (Du et al., 2012). The block-based CS results in a lower computational complexity and requires less memory to store the measurement matrix.

In this paper, a linear convolutional layer performs decimated convolution, as in Eq. (5), in order to obtain the measurements. Convolution can be used as an extension of the inner product in which the inner product is computed repeatedly over the image space. (5) Y=X∗∗D{ϕm},Ym[i,j]= ∑k ∑lX[Di+k,Dj+k]ϕm[k,l]. In Eq. (5), decimation factor D equals the size of the block B and the double asterisk ( ∗∗D ) denotes a 2D convolutional operator decimated with the same factor. A two-dimensional measurement filter  ϕm is created column-wise from the measurement vector  ϕm as shown in Fig. 2. In Eq. (5), Y denotes all the measurements obtained using decimated convolution over the whole input image X with the collection of measurement filters {ϕm} . A visualization of the measurement process modelled using 2D convolution is shown in Fig. 3.

The CS reconstruction process is modelled using a transposed convolution (Dumoulin and Visin, 2016), and the decoding part of the autoencoder is trained to learn the optimal pseudo-inverse linear mapping operator Φ+ from the measurement data.

There are two basic approaches for the measurement matrix design. An arbitrary measurement matrix Φ can be used in the measurement process to obtain measurements y from the input images. In the traditional CS, the measurement matrix with independent and identically distributed (i.i.d.) Gaussian measurement vectors is often used. In that case, the encoding layer of the autoencoder is initialized using the weights defined by the vectors from the measurement matrix Φ and is kept fixed during the training process. A signal dimensionality reduction using a predefined (e.q. random Gaussian, Hadamard, DCT) measurement matrix Φ is sub-optimal due to the fact that it does not exploit the underlying structure of the observed signal.

Alternatively, the optimal measurement matrix can be inferred from the training data. Such a matrix better adapts to the dataset and preserves more information in the measurements, resulting in better reconstruction results. In our proposal, we optimize the encoding part of the autoencoder to learn the optimal linear measurement matrix Φ from the training dataset. In the experimental section, we show the effect of the measurement matrix choice on the reconstruction results.

Training neural networks on normalized, mean-centred data became standard in all areas of machine learning (Ioffe and Szegedy, 2015). It is well known that such practice significantly reduces the training time, but the application to the learning based CS is not straightforward. The measurement process needs to be redesigned in order to obtain normalized and mean-centred measurements, since the mean value of the observed signal has to be measured during the CS acquisition process. In this section, we present an efficient measurement process which enables the direct application of data normalization techniques, which is in contrast with the previous work in this area.

In order to measure the mean value  y1  of the observed block (Fig. 2), we fix the first row of the measurement matrix Φ, so that it corresponds to a row vector containing all ones: (6) y1=1B2 ∑i=1B2ϕ1,ixi=1B2 ∑i=1B2xi. The rest of the matrix ( M−1 rows) is left to be optimized in the training procedure. We mean-centre the normalized measurements  ym using the obtained mean measurement y1 : (7) yˆm=1∑i=1B2ϕm,iym−y1,m∈[2,M]. The decoding part of the network is trained using the mean-centred measurement vector  yˆ as its input, and it results in the mean-centred image reconstruction. In order to obtain the final image reconstruction, the mean value for each image block is restored by adding  y1 to each block.

Training the neural network on non-mean-centred data has undesirable consequences. If the data coming into a neuron is always positive (e.q.  x>0 0$]]> element-wise in  f=wTx+b ), then the gradient on the weights w becomes either all-positive, or all-negative (depending on the gradient of the whole expression f) during the back-propagation step. In return, this could introduce the undesirable zig-zagging dynamics in the gradient updates of the weights (Karpathy, 2017). As shown in Fig. 4, zig-zagging is also manifested in the loss function. The training loss function for the unnormalized measurements (red dashed line) and normalized measurements (blue solid line) are shown in the log scale simultaneously. Notice that the loss function for the proposed network that is trained on mean-centred data converges significantly faster than the network trained on non-centred measurements.

In Lohit et al. (2018) and Du et al. (2019), the authors optimize the linear encoder in order to infer the optimal measurement matrix for each measurement rate r. In Baldi and Hornik (1989), it has been shown that the linear autoencoder with the mean squared error (MSE) loss converges to a unique minimum corresponding to the projection onto the subspace generated by the first principal component vectors of the covariance matrix obtained using the principal component analysis (PCA). Thus, it is sub-optimal to retrain the model for each measurement rate r.

Instead, we propose an efficient initialization method for the deep learning CS models based on the observation from Baldi and Hornik (1989). Principal component analysis (PCA) is an analytic method that has a widespread use in dimensionality reduction. The PCA is performed on the covariance matrix of the data vector x : (8) C(x)=E[xxT]−E[x]E[xT], where E denotes the expectation operator. In the case when images are the signals of interest, PCA is performed by calculating an unbiased estimate of the covariance matrix C(x) for the vectorized images, where x is a flattened image vector, and x¯ is its mean value: (9) C(x)=1N−1 ∑i=1N(xn−x¯)(xn−x¯)T. After applying the eigendecomposition (Eq. (10)) to the estimate of the covariance matrix C(x) for the observed images, an eigenvalue matrix Σ contains positive eigenvalues λ sorted in a descending order. The eigenvalues explain the variance in the direction of corresponding eigenvector in the orthonormal matrix U . Under the assumption that the variance reflects the informational content, a subset of M eigenvectors with the largest eigenvalues (i.e. principal components) optimally describes the observed signal in terms of the mean squared error: (10) C(x)=UΣUT≈U1:MΣ1:M(U1:M)T. If the training dataset is formed to faithfully represent the image statistics, the reduced eigenvector matrix U1:MT optimally preserves the informational content of the observed image blocks.

Thus, we propose to use the reduced eigenvector matrix U1:MT to initialize the weights of the encoding part of the CS model: (11) Φ=U1:MT.

Furthermore, we propose to initialize the reconstruction part of the network using the PCA as well. The eigenvector matrix U is a unitary matrix. If the measurement matrix Φ is equal to the reduced eigenvector matrix U1:MT as in our proposal, we can write: (12) y=Φx=U1:MTx. The original image x can be reconstructed using the pseudo-inverse of the measurement matrix: (13) x=Φ+y=(ΦΦT)−1Φy=[(U1:M)TU1:M]−1(U1:M)Ty=U1:My. Since UT is a unitary matrix, the pseudo-inverse matrix Φ+ for the CS reconstruction becomes just a transposition of the measurement matrix. This results in an efficient method for initialization of neural network weights for both the encoding and decoding part of the learning based CS models.

The proposed initialization method for the network weights has several advantages. While a neural network has to be retrained in order to obtain the measurement matrix Φ for a different sub-rate r, the PCA approach outputs the whole eigenvector matrix  U . Thus, for any measurement rate the initial measurement matrix Φ can be formed by selecting a subset of M largest eigenvectors and one can use them in order to initialize the model. The learning based approach is significantly slower since it is extremely hard to learn the optimal measurement operator and the network might not fully converge. Contrary, in the case of linear autoencoder, we obtain the exact solution for optimal measurement and reconstruction operator in a fraction of time needed to train the neural network. Using the PCA initialization for the autoencoder might be beneficial even when the loss function in the training procedure is not pixel-wise Euclidean and when additional regularization is introduced in the training procedure.

As previously mentioned, the first part of the proposed network consists of a linear autoencoder. Non-linearities can be easily introduced into the measurement and reconstruction part of the network to further improve the initial reconstruction obtained by the autoencoder. In our proposal, non-linearities are only introduced into the decoding part of the network. Although there are some methods that learn a non-linear measurement operator from the data (Mousavi et al., 2017), linearity is an important property of measurement systems and we want our CS model to be realizable in real physical measurement setups like Takhar et al. (2006) and Ralašić et al. (2018).

The output of the proposed convolutional autoencoder represents a preliminary reconstruction of the input image from its low-dimensional measurements. We feed the preliminary reconstruction to a residual network (He et al., 2015) that induces non-linearity and reduces potential reconstruction and blocking artifacts, and eliminates the need for an off-the-shelf denoiser such as BM3D (Dabov et al., 2009) used in the competitive methods. Figure 5 shows several examples of the estimated residual. Residual learning compensates for some of the high-frequency loss and improves the initial image reconstruction.

Figure 6 shows the architecture of the residual learning block used in our proposal. The residual network consists of two residual learning blocks and each residual learning block has three layers. The first layer consists of 16 convolutional filters of size 3×3 with stride 1, followed by a ReLU non-linearity. The second layer has 3×3×32 filters with stride 1, also followed by a ReLU non-linearity. The final layer consists of a single filter of size 3×3 , which outputs the inferred residual image. Image dimensions are preserved in each layer by the appropriate zero-padding. Identity shortcuts are added to each residual block and are used to propagate the intermediate image reconstructions.

Reconstructing the high-frequency content in the original image (i.e. edges, texture) is problematic for the linear autoencoder, and the residual network helps to alleviate this problem. Problems occur partly due to the fact that the lower frequency content is dominant in natural images and the learned measurement filters have a low-pass character, and partly due to the choice of the loss function used for training the network. It is known that the MSE loss function yields blurry images (Kristiadi, 2019). Thus, some papers suggest using a different loss function for the network training. As an example, Lohit et al. (2018) uses the adversarial loss function in addition to Euclidean loss to obtain better and sharper reconstructions. Furthermore, Du et al. (2018) uses perceptual loss in order to achieve better reconstruction results. The authors train their model using the Euclidean loss in the latent space of the VGG19 neural network (Simonyan and Zisserman, 2014).

In this paper, we fuse the per-pixel reconstruction loss in the autoencoder with the perceptual loss in latent space in the residual network. This is in contrast with Du et al. (2018), where the authors optimize the whole network using the Euclidean loss in the latent space. As a consequence, their method results in semantically informative reconstructions, but with low per-pixel accuracy. By using a combination of Euclidean and perceptual loss, we obtain semantically informative reconstructions that have high accuracy of per-pixel reconstruction resulting in higher PSNR compared to Du et al. (2018).

Pixel-wise Euclidean loss function for the autoencoder is defined as: (14) L1({Φ,Φ+})=‖x−f{x,{Φ,Φ+}}‖22, where Φ denotes the weights of the measurement operator, Φ+ are the weights of the reconstruction operator, x is the original image and f{x,{Φ,Φ+}} is the image reconstruction obtained by the autoencoder.

The residual part of the proposed network is trained separately from the autoencoder part using perceptual loss function  L2  (Eq. (15)) in the latent space of the VGG19 network similarly to Du et al. (2018). In contrast with Du et al. (2018), we use a linear combination of Euclidean losses defined on the features of second and third max-pooling layer of the VGG19 network instead of the Euclidean loss on individual feature map. The motivation for this is to simultaneously reconstruct both the low-level information contained in the bottom layers, as well as the high-level semantic features contained in the top layers of the VGG19 network. (15) L2({W})=12 ∑j=23‖ϕj(x)−ϕj(f{x,W})‖22. In Eq. (15),  ϕj denotes the feature map of the j-th layer of the VGG19 with input x. Furthermore,  W denotes filter weights in the residual network and f{x,W} is the final image reconstruction.

In this section, we discuss the details of our network training procedure. We use tensorflow (Abadi et al., 2015) deep learning framework for training and testing purposes. The training dataset is formed using uncalibrated JPEG images from the publicly available Barcelona Calibrated Images Database (Párraga et al., 2010). Our training dataset is created by extracting 1676 image patches of size 256×256 , taken from different parts of the original high-resolution ( 2268×1512 ) images. This corresponds to 107264 unique image blocks for training.

Adam optimizer (Kingma and Ba, 2015) ( β1=0.9 , β2=0.999 , ϵ=1e−8 ) is used for the network training. The learning rate for the loss function L1 is set to 0.001 and the learning rate for the loss function L2 is set to 0.0001. The number of epochs in the training stage is set to 256. The training was performed on an Intel i7-4770K@3.50 GHz computer with NVIDIA GeForce GTX780 (GK110) graphic card.

We perform series of experiments to corroborate previous discussions and observations. In order to achieve a fair comparison framework, a set of 11 images (Monarch, Fingerprint, Flintstones, House, Parrot, Barbara, Boats, Cameraman, Foreman, Lena, Peppers – see TestDataset), which were used in the evaluation of the competitive methods are used for testing purposes with four different measurement sub-rates  r=MN , where r∈{0.25,0.1,0.04,0.01} . In our experiments, block size of 32×32 is used.

In Section 3, we have discussed the connection between the measurement matrix learned by the linear encoder and the one obtained by performing the PCA analysis. In addition, we proposed an efficient initialization method for the network weights. In this section, we perform an experiment to show that the performance of the trained linear autoencoder is limited by the performance of the PCA method network in terms of image reconstruction quality.

Table 1 shows the mean reconstruction results in terms of PSNR for the standard test images. Notice that the reconstruction results are comparable. The slightly lower reconstruction performance of the linear encoder is due to the network not fully converging to the global minimum. Reconstruction results obtained by using the PCA method represent an upper boundary for the performance of the linear autoencoder network for CS image reconstruction.

In Fig. 7, reconstruction results obtained using random Gaussian and adaptive measurement matrix for Parrot test image are shown. The reconstructions are presented for measurement rates r={0.01,0.25} . Notice that the adaptive measurement matrix preserves more information compared to the random Gaussian matrix.

In this section, we compare the proposed CS model to other state-of-the-art learning-based CS methods. To provide a fair comparison, we compare our method only to similar methods which use an adaptive linear encoding part.

We compare our method to the ImpReconNet (Lohit et al., 2018), Adp-Rec (Xie et al., 2017), FCMN (Du et al., 2019) and two variants of PCS (Du et al., 2018), namely PCSconv22 and PCSconv34 . In Table 2, mean PSNR reconstruction results (on the same test dataset) for the proposed method and for the competitive methods are shown. ImpReconNet (Euc) denotes a variant of a ReconNet model that uses Euclidean loss function for the network training, while the ImpReconNet (Euc+Adv) denotes a variant which uses a combination of Euclidean and adversarial loss. The competitive PNSR values are shown as reported in the original papers or reproduced using the available algorithms and models. In Fig. 8, “Fingerprint” test image reconstructions are shown compared to the ground-truth.

On one hand, FCMN and ImpReconNet yield similar results in terms of PSNR compared to our method (see Table 2), while on the other hand the aforementioned methods do not preserve structural and high level semantic information. The two PCS methods preserve structural information, but yield images that contain significant amount of noise when observed on pixel-wise level. Our method benefits from the combination of pixel-wise Euclidean loss in image space and the Euclidean loss in the latent space of the VGG19 network resulting in high pixel-wise accuracy as well as good preservation of structural information. Similar observation holds for the “Monarch” reconstructions in Fig. 9 where a comparison between the competitive perceptual CS methods and the proposed method at extremely low measurement rate r=0.01 is presented. Notice the high level of noise in the PCS reconstructions compared to the reconstruction obtained using the proposed method.