Single Image Super Resolution of MR Images Regularized with
Gradient Profile Sharpness and Induced Sparsity
Prabhjot Kaur1 and Anil Kumar Sao2
Abstract— Diverse parameters constrain the resolution of
magnetic resonance(MR) images and limit the accuracy of
medical diagnosis and analysis. In this paper, we explore the
single image based super resolution of MR images, regularized
with desired gradient profiles. The profile of gradients in low
resolution, high resolution and interpolated low resolution is
studied, followed by estimation of a relation among them. This
lead to approximation of gradient profiles similar to gradient
profile present in HR image, and is used to estimate HR
image with sharpened tissue boundaries along with reducing
other artifacts as stair case effect. The performance of this
algorithm is compared with other single images super resolution
algorithms for MR images, verifying on different data sets -
Alzheimer, MCI, phantom MR images.
I. INTRODUCTION
Accurate medical diagnosis and analysis require high
resolution, high signal to noise ratio (SNR), high contrast in
magnetic resonance (MR) images. However, the increment
in these parameters is inherently constrained by the trade-off
among resolution, SNR and acquisition time. For example,
if it takes 1 minute for single acquisition with 128 phase en-
coding steps, it will require more than 1×128×2 minutes for
2 averages to obtain the resolution of 256 × 256. Also, such
acquired images will be poor in SNR, thus look noisy and do
not serve the clinical purposes. Alternatively, it is possible
to increase the resolution along with good SNR and in less
acquisition time by increasing the magnetic field strength,
which is quiet expensive and thus less available (≤100 7T
MR scanner worldwide). Hence post processing algorithms
serve as better alternative to improve the resolution along
with SNR, and is the focus of this paper.
High precision of medical diagnosis and analysis demands
high quality magnetic resonance(MR) images. The quality
of MR images can be defined by three broad parameters- a)
resolution, b) contrast, and c) signal to noise ratio(SNR).
Due to inverse dependency of resolution on SNR, there
always exist a trade off among these parameters including
acquisition time. It is common practice to acquire optimal
resolution with optimal SNR to maintain the trade-off be-
tween acquisition time and quality of image. It is possible
to increase the resolution of MR images by changing the
pulse sequence parameters, magnetic field strength, but it
*This work was financially supported by Ministry of Electronics and
Information Technology, India
1 A.K.Sao is the Faculty in School of Computing and Electrical
Engineering, Indian Institute of Technology Mandi, India anil at
iitmandi.ac.in
2 P. Kaur is with School of Computing and Electrical Engineering,
Indian Institute of Technology Mandi, India prabhjot kaur at
students.iitmandi.ac.in
require either longer acquisition time or high cost. Similarly,
it is possible to increase the SNR but lead to low resolution
MR images. Also the MR images are inherently noisy in k-
space domain by Rayleigh and thus the magnitude images
are noisy in nature with gaussian distribution [?]. Hence,
post processing algorithms serve as better alternative to
increase the resolution as well as denoise the MR images
for improving the quality of MR images,required for accurate
medical diagnosis.
It has been demonstrated in [1], that the post processing
algorithms can improve the trade off among acquisition time,
SNR, and resolution. Following which, various algorithms
have been developed to improve the resolution of MR images
in through-plane(z) direction, or in in-plane(x-y) direction, or
in all three directions. Many of super resolution algorithms
explore the resolution improvement in through plane direc-
tion, with recent developments for in-plane or 3D resolution
improvement. Most of these algorithms super resolve the
MR images in supervised manner and require example low
resolution(LR) and high resolution(HR) MR images. Since
such requirement of example LR and/or HR images in MR
imaging of inter- or intra-modality is expensive and is not
always practically suitable in real scenario, single image
based algorithms have been developed recently.
The few algorithms for super resolution of MR images
avoiding the requirement of example LR-HR MR images,
focus on improving resolution by atmost considering the
local spatial consistency. These algorithms do not specifically
consider the importance of sharp image details in MR images
like clear tissue boundaries, generally required in clinical
practices and thus usually estimate images with blurred
details and/or staircase effects.The clear tissue boundaries or
sharpness of image details can be very well discriminated in
gradient domain of images. Following which, some gradient
priors for natural images are proved to be helpful in super
resolving the images [2]. Speaking in terms of MR images,
the sharp and clear tissue boundaries are very helpful for
medical analysis. However, no such regularization has been
proposed for super resolution of MR images.
In case of denoising the MR images, several algorithms for
rician noise, Rayleigh and Gaussian noise have been devel-
oped. Most of these algorithms require the prior knowledge,
for example noise type to parameterize its distribution and
thus estimate it before denoising. The procedure to denoise
and then super resolve has been widely accepted but it tend
to vanish many minor details in its first step i.e. denoising
which are difficult to restore in super resolution step. The
minor details play significant role in medical diagnosis and
analysis. Thus the step by step procedure is not reliable in
medical scenario.
In this work, we instead propose a single framework
for denoising and super resolution algorithm with focus on
reduced loss of image details using heirical PCA based
denoising strategy and clearer tissue boundaries at higher res-
olution using the regularizer with gradient profile estimation.
Herirical PCA based denoising involves the decomposition of
image patches into local spaces and then denoising the cate-
gorized noisy patches in each of the local space, reducing the
information loss. Gradient profile describes the characteristic
of edge - contrast and width of the edge. Sharp image details
are present in original LR image, and tend to get blurred
when upsampled. In order to obtain the upsampled image
with sharp details, we observe the relation among gradient
profiles for LR, HR, and interpolated LR images(UR). The
relation between LR and UR is estimated in a way that
the desired gradient profile is estimated which is similar to
gradient profile of HR image. Further the estimated gradient
profile is used to constrain the solution space such that the
estimated HR image has sharper gradient profiles.
The contributions of the proposed work are as follows:
i) The proposed work address the single framework for
denoising and super resolution algorithm for MR images that
do not require expensive example LR-HR images. ii) The
denoising is done using heirical PCA based multiple dictio-
naries framework and does not require any prior knowledge
of noise. iii) Image details with sharp gradient profiles are
preserved using gradient profile sharpness regularizer and
multiple dictionaries enriched from same and adjacent slices.
iv) The proposed approach provides HR image estimate
without any artifacts like staircase effect and blurriness. v)
The performance analysis is done on both healthy and non-
healthy patients.
The preliminary work in this direction has been presented
by us in International Conference on Medical Image Com-
puting Computer Assisted Intervention(MICCAI2018) [3].
The work in this paper has been further extended for its
generalization for 3D super resolution, employed with de-
noising algorithm, detailed methodology and experimental
results comprise reconstruction as well as segmentation, and
its application for different diseases like Alzheimer, Multiple
sclerosis, and etc.
The rest of the paper is organized as follows: Section
II describes the related approaches and SR approach in
sparse representation framework. The details of the proposed
methodology is explained in Section III. Section IV explains
the experimental results for validation of proposed work. The
summary of the paper in Section V concludes the paper.
II. R ELATED W ORKS AND C ONTRIBUTIONS
Estimation of HR image from given LR image can be
thought in two ways - reconstruction based methods and
learning based methods. The LR image (x ∈ Rn ) is down-
sampled and blurred version of HR image (y ∈ Rm , m < n),
so there exist many possible HR images as solution for given
LR image. Thus estimation of LR image is inherently ill-
posed in nature, and need regularization while reconstruction
of HR image.
The super resolution problem for MR images can be
categorized on the basis of direction like 1D(through-
plane), 2D(in-plane) and 3D(image volume) super resolution.
The longer acquisition time demand for SR algorithms in
through-plane direction. However in general, increase in
spatial resolution lead to decrease in SNR. Hence it puts
a threshold on in-plane resolution as well. Few algorithms
address the problem of 2D and 3D super resolution [].
A. Sparse Representation Framework for Super Resolution
Simplest idea of super resolution in sparse representation
framework is to represent the estimate of HR patches as
linear combination of atoms learned in dictionary. The over-
complete dictionaries are learned for LR and HR images as
φL and φL respectively, such that
φH = arg min ||x − φH αH ||22 , (1)
φH
φL = arg min ||y − φL αL ||22 , (2)
φL
for y and x be LR and HR the patch vectors respectively.
Due to similarity in average structure of the image in low
and high resolution, the linear combination of dictionary
atoms for representation of LR and HR images is assumed
to be same(αL = αH ). Thus the reconstruction of test
image patch(x̂) is done as x̂ = φL αH . Further it has been
experimented as coupled dictionary to learn both dictionaries
simultaneously assuming the same coefficient vector as [?]
min ||y − φL α||22 + ||x − φH α||22 . (3)
φH ,φL
The reconstruction of MR images using such approach ef-
fectively outperforms conventional approaches [?]. However
it requires LR-HR paired images for learning the dictionaries
as φL and φH respectively.
B. Super Resolution Algorithms for MR Images
The concept of self learning instead of example based
learning has been reported in [?]. Inspired from this, few
super resolution algorithms are developed with dictionary
building from patches extracted from image itself and addi-
tional regularizers to obtain estimate HR image with desired
characteristics in an ill-posed environment[].
Learning based methods make use of examples to learn
the degradation transformation and use it to estimate the HR
image. Examples include LR-HR images other than given LR
image, and thus impose additional cost to algorithms. While
in-plane and volume super resolution, examples are acquired
from scanner. Examples include intra-modality images []
and/or intra-modality images []. In medical imaging, espe-
cially in MRI it is timely and cost wise expensive to acquire
example images and thus reconstruction based methods are
evolving parallely.
The reconstruction based methods rely on the degradation
model comprising downsampling and blurring. In medical
 10 4 10 4

4
350 350
3.5
2
300 300
3

250 250 1.5

2.5

200 2 200

1
150 1.5 150

100 1
100
0.5

0.5
50 50

0 0
20 40 60 80 100 120 140 20 40 60 80 100 120 140

(a)
Fig. 1. (a) Parameters to characterize the edge; (b) Illustration of 2D histogram for ηU (x-axis) and ηH (y-axis) for upscale factor 2 in left image and
for upscale factor 3 in right image.
imaging, the first attempt to increase the resolution in in-
plane direction of MR images without using any example LR
and/or HR images is addressed in [?]. The high resolution
MR images are estimated by weighted linear combination
of 3D cubes present non-locally in MR image volume.
To overcome the difference in range of intensity values in
estimated HR estimate, mean of local pixel values is scaled
using the original LR image intensity values. The recon-
structed images using this algorithm are better in quality as
compared to conventional interpolation techniques. However,
as it involves the weighted average, it tend to blur the image
details.
Further, the super resolution is exploited in sparse rep-
resentation framework regularized with intrinsic low rank
structure of brain MR images and total variation for addi-
tional local spatial consistency. This reconstructed images
using this algorithm are sharp and thus effective for better
visualization. Despite of sharpness, the results are contam-
inated with several artifacts like zigzag, and blurriness at
several regions.
The proposed work focus on single image based super
resolution approach which estimate the HR image with sharp
image details regularized with estimated gradient profiles of
the image. The consistency is taken care while reconstruction
of image patches using dictionary learned from same and
adjacent slices.
III. P ROBLEM F ORMULATION
Each given LR image Y is interpolated to desired res-
olution, extract patches and represent them in vector form
say Pj Y = yj , Pj is the operator to extract jth patch. The
patches extracted are vectorized and then concatenated to
form a patch matrix y and x for LR image and HR image
respectively. There exist various mathematical formulations
to model the noisy LR images and we follow the standard
approach as
y = DHx + η. (4)
Here D and H are the downsampling and blurring operators
respectively, which operate on vectorized patches of high
resolution image (x) and noise (η) is added to give patches
of noisy low resolution image(y).
The objective of the SR problem is to recover HR patch
(b)
matrix x, xi ∈ Rp×1 corresponding to given LR patch matrix
y, yi ∈ Rq×1 with q < p. In the sparse representation
framework it can modeled as
X
x̂ = arg min ||Pi x − xi ||22 + ||y − Hx||F , (5)
x
i
where xi s are the HR patches, and D, H are the degrada-
tion operators comprises downsampling and blurring by a
Gaussian kernel of standard deviation equals to one voxel
size.
IV. P ROPOSED W ORK
The proposed work consider the noisy low resolution
image as input and tend to denoise the image along with
resolution improvement in an iterative fashion. While iterat-
ing, the noisy patches in the image are extracted using the
reconstruction error by PCA based multiple dictionaries and
denoised using simple averaging. The details in the estimated
high resolution image are preserved with multiple PCA based
dictionaries regularized with gradient profile sharpness. The
salient points of the proposed work are discussed below:
A. Noise Removal using Hierarchical Principal Component
Analysis
Given the noisy image, the patches are extracted and
Principal Components (PCs) are computed after zeroing the
mean of the patches. PCs are sorted in descending order
of eigen values. The fact that the noiseless signal posses a
structure and thus follows the power law when projected
on principal components. In the first step, the image is
divided for low and figh frequency components. For the
high frequency components, PCA is computed and signal
is reconstructed with only first few principal components.
Then the reconstruction error is computed. The fact that the
first few components are dominated by the signal structure,
the noisy patches will result in more reconstruction error.
To illustrate the reconstruction error for noiseless signal and
for only noise, we selected 100 random MR noisy images,
computed the PCs and reconstructed noiseless signal and
only noise. The reconstruction error is computed and the
corresponding histogram is shown in Figure. It indicates
that the error for noiseless signal lies on the lower side
as compared to only noise, and are thus thresholded to
 1 1

0.5 0.5

0 0
0 200 400 600 800 1000 0 200 400 600 800 1000

(a) (b)
1 1

Fig. 2. Illustration of reconstruction error for noiseless signal and only 0.5 0.5
noise. X Axis : Reconstruction error value, Y-Axis: Count of the error
values, Blue: Noiseless Signal, Yellow: Noise
0 0
0 200 400 600 800 1000 0 200 400 600 800 1000

(c) (d)

Fig. 4. Histogram of non-zero GPS values with obtained for

randomly selected (a) UR image, (b) LR with downsample factor
2, (c) Super resolved HR image, and (d) the corresponding ground
truth HR image, from MR image volumes in the dataset. Here x-
axis shows the bin number and y-axis indicates the corresponding
count of GPS values scaled in 0 to 1 range.

reconstructed images for single as well as multiple dictionary

framework in experimental section.

C. Super Resolution with Gradient Profile Sharpness

Fig. 3. Illustration of iterative PCA based Denoising
categorize as noisy patches. To denoise these patches in very
simplistic manner, each pixel value in the patch is replaced
by the mean value of that patch.
Further, the reconstructed image is still contaminated by
the noise, and is thus proceeded for the second stage of PCA
by replacing the noisy image by reconstructed image. We
compute the PCs on the reconstructed image and repeat the
same procedure to further denoise the image. This procedure
is illustrated in Fig.3. The noise component decreases and
thus the PCs become more reliable after every iteration. Thus
the number of first principal components used to reconstruct
image is increased in each iteration to incorporate the high
frequencies while denoising.
B. Denoising Strategy in Multiple Dictionary Framework
The idea of hierarchical denoising in single set of PCs tend
to loose many image details due to significant information
present in last few slices. To overcome this, we employ the
multiple dictionary framework by dividing the patches into
several clusters and computing PCs for each of the cluster.
Multiple PCs tend to span the global space by spanning
multiple local spaces. Thus the information present in last
few PCs is not that significant and is illustrated by the
The profile of the gradient can be defined in various
ways depending upon the edge model like triangle model,
Gaussian multimodal and etc [2]. To define an edge, we
can use the parameters height(h) and width(w) of the edge
as shown in Fig. Sharp edge indicates large height and
small width as shown in Fig. We can define gradient profile
sharpness(GPS, η) as the ratio of height and width similar
in [4]
h
η= . (6)
w
To see the effect of degradation and interpolation on GPS
values, GPS values are computed for LR, UR and HR im-
ages and corresponding histograms computed for randomly
selected 50 MR images can be seen in Fig IV-C. The
histograms are normalized from 0 to 1 as the number of
pixels differ in case of LR image. It can be observed, that
most of the GPS values for UR image are lying in small
values region, whereas there exist higher GPS values for LR
and HR images. Such behavior of UR image can be due
to the basic interpolation leading blurred details and hence
leading to small GPS values. However, the percentage of
edge pixels in LR and HR images are generally similar and
can be assumed as
hist(ηL ) = hist(ηH ), (7)
also indicated experimentally in Fig IV-C.
To evaluate how the GPS values in UR and HR images
are related, we randomly select HR and corresponding UR
image, and plot 2D histogram in as shown in FigIV-C.
The linear behavior can be observed, however the linearity
vanish as it tend towards higher scale factors. Also, the
linearity computed quantitatively with Pearson Correlation

(a) Natural Im- (b) Natural Im- (c) 207 Natural
ages, SRF=2 ages, SRF=3 Images, SRF=3
(d) MR Images, (e) MR Images, (f) 207 MR
SRF=2 SRF=3 Images (Single
subject), SRF=3
TABLE I
D EMONSTRATION OF LINEAR RELATION BETWEEN GRADIENT PROFILES
OF UR AND HR OF NATURAL IMAGES , AND RELATIVELY NON - LINEAR
RELATION FOR MR I MAGES .
Coefficients(PCC) is low as compared to natural images. To
show the difference in such behavior of gradient profiles of
natural and MR images, we considered 1000 images from
mother nature and from MR image volumes of different sub-
jects. The gradient profile values for UR and HR images are
computed and 2D hist is computed. To clearly demonstrate
the difference we normalized each column of 2D hist plot
by dividing it with maximum entity of each column and is
shown in Fig IV-C. In order to illustrate the behavior of
a complete subject’s MR images volume also, we randomly
select a volume and illustrate the behavior. It can be observed
that for natural images, the relation is highly linear in contrast
to the case of MR images. It is due to the fact that there exist
many small details in MR images, which vanish rapidly as
compared to natural images and thus high GPS values posses
non-linearity. Hence we preferred to divide the range of ηU
values into four regions to increase the linearity further. For
each region, GPS values of HR can now be approximated as
ηH = βηU . (8)
Also, as the upsampling factor increases the slope of line
increases (not as good as in natural images). This is due to
the increased blurriness at higher downsampling scales. We
also observed that with increase in upscaling factor, more
GPS values tend towards zero or lower values in UR image.
Now, we can use eq(7) and eq(8) to estimate GPS similar
to GPS values of HR. This is based on the fact that the
linearity is distorted as the upscale factor is increased.
Including the improvement in linear correlation, it also helps
in considering the low GPS values (smooth regions) and high
GPS values (regions with edge/texture) differently. We thus
estimate the GPS values for HR image as η̂i = βi ηU i , i =
1 to 4, using LR and UR images only. This is used to
constrain the solution space for estimate of HR image by
using η̂ as
X
x̂ = arg min ||Pi x − xi ||22 + ||y − Hx||22 + λ||η̂ − ηx ||22 .
x
i
(9)
Here, λ is a scalar to weigh the regularizer and is chosen
empirically.
D. Fusing Denoising and Super Resolution of MR Images
in Single Framework
1) GPS behavior with Noise: The gradient profile is com-
puted using both contrast and width of edge. The width edge
created by noise is too less and most of the edges are thus
discarded by thresholding in the ratio defined in 8. Hence, the
GPS map computed for noisy image and noiseless image are
similar except for noisy pixels which are few in number as
compared to image from which GPS is computed, is shown
in Fig 5. It can be observed that the in 2D hist plots, as noise
level is increased the the slope is becoming lower. Following
this, the relation computation remains almost similar and
helps in regularizing the image with sharp details even in
case of noisy low resolution images. However for large noise
levels, the number of noisy pixels increase due to significant
virtual edge width. Thus the weight to the regularizer is
decreased as the estimated noise level is increased to reduce
the effect of few noisy pixels in GPS maps. The noise level
is estimated using the relation among horizontal and vertical
gradients as in [5].
2) Employing the denoising strategy along with super
resolution: The image reconstructed by super resolution
algorithm is denoised in every iteration and the estimated
image is replaced by weighted average of super resolved
image and its denoised version. It tend to denoise the
image along with preservation of many image details. One
can also think of employing the denoising strategy in the
dictionaries used in SR. This tend to loss of image details
which can be otherwise preserved by averaging the images.
Further, the dictionary used to reconstruct high frequencies
in SR algorithm can be used to reconstruct only the average
component of MR image to denoise the image at expense of
loss of minor image details. The resultant images for both
the strategies are shown in experimental section. We here
prefer to preserve the minor image details, at expense noise
left in image.
E. Dictionary Building and NLM Estimate
Patches of the images can be well represented as linear
combination of atoms in dictionary. Here, dictionary is an
important part as its structure defines the reconstruction
quality of estimate of HR image. To emphasize the impor-
tance of high frequency details of MR image, we construct
dictionary only for image details obtained by subtracting
the average component of the image. Further, the structure
of details can significantly vary and intuitively thought to
have least contribution in representation of each other. Thus
different details are grouped in different clusters using k-
means strategy. The basis of each cluster is computed using
PCA as Eigen vectors. The Eigen vectors arranged in the
 a d g

b e h

c f i

Fig. 5. (a),(b),(c) - Represents the 2D hist of GPS values for LR and HR for noiseless, 2%Noise and 5%Noise respectively; (d),(e),(f): represents the LR
image downscaled by factor 3 and added noise as 0, 2 and5% respectively; (g),(h),(i): represents the GPS map for UR image for added noise 0,2 and 5%
respectively.

column wise fashion is used as dictionary in this work and It is known that in MR image volume the adjacent slices
the optimization can now be seen as: include less but important variations. While upsampling,
X these variations are important to maintain the consistency
x̂ = arg min ||Pi x−xi ||22 +||y−HAα||22 +||α||1 +λ||η̂−ηx ||in22 . each direction. Thus while reconstruction of average
x
i
(10) component of MR images, we further expand the variability
Here A is considered as dictionary and α be the correspond- in dictionary atoms by introducing patches from the slices
ing coefficient vector. Thus there exist a set of dictionaries as adjacent to the slice to be super resolved. This improve the
A = {A1 , A2 ...AM }, M be the number of clusters. While regularity in intensity domain as well as spatial smoothness.
reconstruction, the j th dictionaryPfrom various clusters, is In this work, we also use image pyramid, generalizing for
Mj yj 2 2D and 3D super resolution, formed by up-downsampling
selected as j = arg minj ||y − p=1 Mj ||2 , and used to
represent the test image details (y). The number of elements the MR slice by rational factors, using it to enrich the
in j th cluster is represented by Mj . There exist small dictionaries for better reconstruction of in-plane and 3D cube
projection coefficients due compact structure of the images reconstruction.
and can be discarded, inducing the sparsity. The patches and The average component of image represents the contrast
cubes for 2D and 3D SR respectively are represented by of image and is thus significant for medical diagnosis. We
linear combination of selected dictionary atoms. reconstruct the average component separately using non-local
mean computation. Algorithm 1 Estimate HR image
Result: x̂ = arg minx ||y − HAα||22 + ||α||1 + ||η̂ − ηx ||22
Estimate η̂ as η̂ = βηU with β obtained from eq(??)
while not converge do
Step1: Construct PCA based dictionary set A =
{A1 , A2 , .....AM }
Step2: Approximate the low frequency components in
F. Estimation of HR Image image using NLM
Step3: Optimize eq(10) using gradient descent ap-
proach
We follow gradient descent approach for the optimization Step4: Denoise the HR image estimate using procedure
and following steps are done in sequence to reconstruct the illustrated in 3.
estimate of HR image x̂ = x̂h + x̂l : Step5: Replace the HR image estimate by weighted
averaging denoised version and HR image estimate
itself
1) Restoration of image details (x̂h ): The dictionaryAj end
to restore the image details, is chosen on the basis of
X
j = arg min ||y − rj ||22 , (11) 4) Weighted average of denoised version and regularized
j
k estimate of HR image: The regularized image is denoised
by denoising strategy discussed above. The HR estimate is
rj represents the centroid for jth cluster. Further, the coeffi-
weighted average of denoised version and regularized version
cients (α) are obtained by
of super resolved image. The weights depend upon the noise
min ||α̂||1 , such that ||y − HAα||22 ≤ , (12) level added, more the noise higher weightage is given to the
α̂ = min ||y − HAα||22 + ||α̂||1 . (13) denoised version.
α

The required sparsity in the above cost function is achieved V. E XPERIMENTAL R ESULTS
by soft thresholding the coefficient vector []. The obtained
α̂ is used to construct the image details as A. Experimental Settings
x̂h = Aj sof t(α, τ ). (14) The performance of the proposed algorithm is evaluated
for three categories (i) resolution improvement, (ii) denois-
2) MR image estimate: The image x̂ can be approximated ing, (iii) resolution improvement fused with denoising in
as a single framework. The low-resolution image volumes are
x̂ = x̂h + x̂l . (15) simulated by blurring the HR image volumes using Gaussian
kernel of the standard deviation of one voxel size and
The low frequency components x̂l are computed by linearly followed by downsampling. In the case of denoising exper-
combining the patches extracted non-locally from the test iments, noise is added from various distributions (Gaussian,
image as Rayleigh and Uniform), with standard deviation of 1%, 2%
2
X xq exp ||x̂−x̂q ||2 and 3% of maximum intensity value in the image.
h
x̂l = , (16)
z Eight MR image volumes used for these experiments are
q∈ζ
randomly selected from the dataset scanned from 7T MR
ζ representing the set of patches similar to x̂l and h controls scanner and is available online [6]. The spatial resolution
the rate of decay in proportion to dissimilarity of patches. of each volume is 1 × 1 × 1mm3 and is used to simulate
LR images, and then super-resolved and/or denoised using
3) Regularized estimation of HR image: The coefficient different algorithms. The efficacy of proposed algorithm is
vector is already computed with inducing sparsity in eq (13), also analyzed for non-healthy subjects- (i) with structural
thus we can consider eq (9) as cost function. Assuming G deformity [6], (ii) Alzheimer [?], (iii) Mild cognitive impair-
as the operator computing GPS values, the cost function ment [?]. The central 120 axial slices are used of the dataset
in eq (9) can be re-written by replacing ηx = Gx. We in [6], as first and last few slices contain less information.
can also write, xi = Aαi . The closed form solution(x̂)ˆ of
The weight for gradient profile regularizer is fixed as 0.01.
the mentioned cost function, using x as obtained x̂, can be The reconstruction quality of resultant images is evaluated
derived as: using peak signal to noise ratio (PSNR) and structural sim-
P
PT ˆi )−HT (y−Hx̂)−λGT (η̂−Gx̂)=0
i (Pi x̂−x (17) ilarity index (SSIM). Further the binary maps are computed
i
for resultant images by segmenting white matter and gray
T
PT T T T T
(18)
P P
i Pi Pi x̂− i i Aαi −H y+H Hx−λG η̂+λG Gx̂=0
matter. The dice ratio is computed and compared to indicate
( i PT T T T T T
(19)
P P
i Pi +H H+G G)x= i Pi Aα+H y+λG η̂ the clearer tissue boundary and accurate post-processing
ˆ
x̂=(
P T T T −1
( i PT
P T T
(20) provided by proposed algorithm.
i Pi Pi +H H+G G) i Aα+H y+λG η̂)
B. In-plane Resolution Improvement
The in-plane resolution is increased in MR images
acquired from 7T MR scanner with spatial resolution
173×173×207. The images are degraded as explained in
experimental settings. The resolution is improved using
different single image based algorithms. To compare the
reconstruction quality of images and to illustrate the consis-
tency in 3 directions, we encase the axial, sagittal, coronal
with one zoomed version of the image detail in Figure 6.
It can be observed that the images upsampled by nearest
neighbor interpolation are contaminated with artifacts of
zig-zag effects. The resultant images of spline interpolation
and NLM3D [7] are refined version of zig-zag artifacts,
yet with blurred details. The LRTV [8] algorithm tend to
reduce the blur and hence it can be seen that the contrast
of images is improved. However some image details are
still blurred details and stair-case effect. These undesired
effects are significantly reduced and the same can be seen in
proposed images. The quantitative analysis for the same is
computed as PSNR and SSIM values for scale for 2 and 3.
The mean values for these parameters are computed for each
of randomly selected 8 MR image volumes and are shown
in Table II and III. It can be observed that the difference
between the values for the different algorithms including
proposed approach and basic interpolation is decreased as
scale factor increases. It is due to the fact that many minor
details are lost with increase in the scale factor, the limiting
the image details improvement. It can be observed that the
proposed algorithm results in increased PSNR and SSIM
values, indicating improved quality of super resolved MR
images.
C. Super Resolution in All Three Directions
The MR images from 7T MR scanner are degraded in
all three directions by the same pipeline as mentioned in
experimental settings. While super resolving, the MR image
volume is split into overlapping sub-volumes of 10 in-plane
slices, with overlap of 5 slices. The values for PSNR and
SSIM is computed for the quantitative analysis of resultant
images and are shown in Table IV for scaling factor 2. The
proposed algorithm provide high PSNR, SSIM values justi-
fying the accurate image detail reconstruction in Figure 7.
It can be observed that the boundary between gray matter
and white matter is visible in zoomed window of proposed
algorithm as compared to LRTV algorithm. Moreover the
details which are otherwise blurred in existing algorithms
have been comparably preserved in proposed algorithm.
D. Efficacy on Structural Deformation - Tumor
It is observed that the learning based algorithms tend to
change the deformity. On the contrary, single image based
algorithms transform the details to enhance the information
already present. To emphasize the significance of single
image based algorithms in image processing, we chose an
MR image volume with tumor and improved the resolution
of MR images as shown in Figure 8. It can be observed that
apart from skull outlines, the interior tissue boundaries are
also clear and more localized in images super resolved by
proposed algorithm.
E. Efficacy of super resolved images in segmentation of
tissues
The upsampled images shall aid in better segmentation at
higher resolution and thus can tend towards better applica-
bility in clinical analysis. Thus we evaluate the segmentation
accuracy of super resolved images. We compute the segmen-
tation labels using the FAST software in FSL [9]. The orig-
inal HR images are also segmented for white matter(WM),
gray matter(GM), and cerebrospinal fluid(CSF). These labels
are used as ground truth and we check the similarity of
segmentation labels obtained for MR images reconstructed
by different algorithms. The similarity is quantified by dice
ratio and is shown via box plots in Fig 9. It can be seen
in box plots that the dice ratio for CSF is comparable in
images by LRTV and Proposed method. In WM and GM
the proposed algorithm is relatively better.
F. Performance on MR images of Alzheimer, MCI, CN sub-
jects
To address the performance of proposed approach in case
of unhealthy patients, we use the ADNI data set available
online [?].
G. Super Resolution Fused with Denoising - Healthy Sub-
jects
The performance of super resolution as well as denoising
algorithm in single framework has been computed for up-
scaling factor 2 and 2% Noise. The qualitative results for
the same have been shown in Figure. VI, for the nearest
neighbourhood interpolation, non local mean NLM3D [7],
LRTV [8], proposed approach with reconstruction of only
avergae component with PCA based dictionaries, and pro-
posed appraoch considering the reconstruction of high fre-
quency components while restoration. It can be observed
that the NLM approach lead to loss of minor details which
are preserved in LRTV. The proposed approach with only
average compoenent lead to a smoother image which is a
compolete denoised version of LR image. However, there
are minor details which are lost and thus in the second case,
the high frequencies are also reconstructed compromising on
denoising aspect yet keeping the details preserved justified
with higher SSIM values. The quantitative results for the
same have also been computed using PSNR and SSIM values
for different subjects in Table V.
1) Different Noise Levels: To see the effect of different
noise levels, the single randomly selected volume is degraded
with downsampling factor 2 and the noise added is varied
from 1% to 5% to see the performance on varying levels
of noise. The PSNR and SSIM values are calculated for
the same and mentioned in Table VII. The trend of PSNR
values remains same. However, increase in PSNR values is
decreased as the noise level is increased.
2) Different Noise Distributions:
 R EFERENCES
[1] E. Plenge, D. H. J. Poot, M. Bernsen, G. Kotek, G. Houston,
P. Wielopolski, L. van der Weerd, W. J. Niessen, and E. Meijering,
“Super-resolution methods in mri: Can they improve the trade-off
between resolution, signal-to-noise ratio, and acquisition time?”
Magnetic Resonance in Medicine, vol. 68, no. 6, pp. 1983–1993, 2012.
[Online]. Available: http://dx.doi.org/10.1002/mrm.24187
[2] J. Sun, Z. Xu, and H.-Y. Shum, “Image super-resolution using gradient
profile prior,” in 2008 IEEE Conference on Computer Vision and Pattern
Recognition, June 2008, pp. 1–8.
[3] P. Kaur and A. Sao, “Gradient profile based super resolution of
mr images,” in Medical Image Computing and Computer Assisted
Intervention, 2018, pp. 109–117.
[4] Q. Yan, Y. Xu, X. Yang, and T. Q. Nguyen, “Single image superresolu-
tion based on gradient profile sharpness,” IEEE Transactions on Image
Processing, vol. 24, no. 10, pp. 3187–3202, Oct 2015.
[5] S. Mandal, A. Bhavsar, and A. K. Sao, “Noise adaptive super-resolution
from single image via non-local mean and sparse representation,”
Signal Processing, vol. 132, pp. 134 – 149, 2017. [Online]. Available:
http://www.sciencedirect.com/science/article/pii/S016516841630247X
[6] “https://www.humanconnectome.org/study/hcp-young-
adult/document/1200-subjects-data-release.”
[7] J. V. Manjón, P. Coupé, A. Buades, V. Fonov, D. Louis Collins, and
M. Robles, “Non-local MRI upsampling,” Medical Image Analysis,
vol. 14, no. 6, pp. 784–792, Dec 2010. [Online]. Available:
http://dx.doi.org/10.1016/j.media.2010.05.010
[8] F. Shi, J. Cheng, L. Wang, P. T. Yap, and D. Shen, “LRTV: MR image
super-resolution with low-rank and total variation regularizations,” IEEE
Transactions on Medical Imaging, vol. 34, no. 12, pp. 2459–2466, Dec
2015.
[9] F. Shi, L. Wang, Y. Dai, J. H. Gilmore, W. Lin, and D. Shen, “LABEL:
Pediatric brain extraction using learning-based meta-algorithm,”
NeuroImage, vol. 62, no. 3, pp. 1975 – 1986, 2012. [Online]. Available:
http://www.sciencedirect.com/science/article/pii/S1053811912005307
 Subject Nearest Spline NLM LRTV Proposed Subject Nearest Spline NLM LRTV Proposed
1 28.58 29.95 30.56 32.57 33.73 1 27.27 28.51 29.22 29.60 30.59
2 27.34 28.39 28.53 30.54 31.69 2 25.99 26.97 27.25 27.65 28.59
3 27.76 28.78 28.95 0.00 0.00 3 26.30 27.51 28.10 28.38 29.39
4 27.69 29.03 29.25 31.42 32.70 4 26.93 28.14 28.71 28.96 29.98
5 28.31 29.60 30.07 31.89 33.07 5 26.65 27.72 28.16 28.48 29.49
6 28.03 29.24 29.42 31.61 32.97 6 27.11 28.24 28.59 29.07 30.13
7 28.51 29.73 29.91 32.08 33.34 7 26.07 27.22 27.69 27.96 28.91
8 27.47 28.64 28.87 30.91 32.06 8 25.71 26.84 27.22 27.54 28.62
TABLE II
M EAN PSNR VALUES FOR SUPER RESOLVED IMAGES IN - PLANE DIRECTION BY DIFFERENT ALGORITHMS FOR SCALE FACTOR 2 ( LEFT ) AND 3 ( RIGHT )

Subject Nearest Spline NLM LRTV Proposed Subject Nearest Spline NLM LRTV Proposed
1 0.37 0.41 0.42 0.45 0.50 1 0.47 0.50 0.50 0.57 0.63
2 0.40 0.44 0.45 0.49 0.54 2 0.50 0.54 0.53 0.62 0.67
3 0.38 0.43 0.44 0.00 0.00 3 0.48 0.52 0.51 0.00 0.00
4 0.42 0.47 0.48 0.51 0.56 4 0.52 0.56 0.56 0.64 0.69
5 0.39 0.45 0.45 0.48 0.54 5 0.50 0.53 0.53 0.61 0.67
6 0.41 0.46 0.47 0.50 0.55 6 0.52 0.56 0.55 0.63 0.69
7 0.42 0.47 0.48 0.52 0.57 7 0.53 0.57 0.56 0.66 0.70
8 0.41 0.45 0.47 0.49 0.55 8 0.51 0.55 0.54 0.62 0.68
TABLE III
M EAN SSIM VALUES FOR SUPER RESOLVED IMAGES IN IN - PLANE DIRECTION BY DIFFERENT ALGORITHMS FOR SCALE FACTOR 2 ( LEFT ) AND 3
( RIGHT )

Fig. 6. Illustration of reconstructed results super resolved in in-plane direction with three planes of MR image volume. Left to Right- In first row: Nearest
neighbor interpolated, spline interpolated, NLMD [7], and in second row: LRTV [8], Proposed Algorithm, original HR images are shown.
 Subject Nearest Spline NLM LRTV Proposed Subject Nearest Spline NLM LRTV Proposed
1 31.71 32.15 31.42 33.40 33.81 1 0.42 0.44 0.41 0.43 0.43
2 0 0 0 0 0 2 0.00 0.00 0.00 0.00 0.00
3 31.28 31.59 32.26 33.84 34.65 3 0.26 0.27 0.28 0.31 0.25
4 29.67 29.95 30.55 32.00 32.82 4 0.32 0.33 0.34 0.37 0.32
5 5 0.43 0.45 0.43 0.45 0.37
6 31.44 31.78 32.44 34.24 35.40 6 0.35 0.36 0.37 0.40 0.41
7 31.23 31.70 29.95 31.73 32.02 7 0.46 0.48 0.37 0.39 0.34
8 8
TABLE IV
M EAN PSNR ( LEFT ) AND SSIM ( RIGHT ) VALUES FOR SUPER RESOLVED IMAGES IN ALL THREE DIRECTIONS BY DIFFERENT ALGORITHMS FOR SCALE
FACTOR 2.

(a) Nearest Neighbor (b) Spline image (c) NLM3D image

(d) LRTV (e) Proposed (f) Original HR image

Fig. 7. Illustration of results for super resolution in all three directions by scaling factor 2. Each slice of axial, sagittal, and coronal is shown. Zoomed
version of axial slice is also shown to demonstrate the difference.
 (a) Nearest Neighbor (b) Spline image (c) NLM3D image

(d) LRTV (e) Proposed (f) Original HR image

Fig. 8. Illustration of super resolution results for structural deformity- tumor. Each slice of axial, sagittal, and coronal is shown. Zoomed version of tumor
region is shown separately.

Fig. 9. From from left to right: Box plots of dice ratio computed for segmentation labels for white matter, gray matter, cerberospinal fluid and skull,
computed for images reconstructed by different algorithms
Subjects NN NLM LRTV PROP
PSNR 30.26 30.58 31.43 31.78
1
SSIM 0.904 0.9114 0.9096 0.8719
PSNR 32.64 31.66
2
SSIM
PSNR 30.44 30.84 31.86 32.23
3
SSIM 0.8895 0.8957 0.9115 0.903
TABLE V
Q UANTITATIVE RESULTS FOR DIFFERENT ALGORITHMS FOR LR IMAGE
WITH UPSCALING FACTOR 2 AND 2% ADDED NOISE .
 PSNR=29.67dB, PSNR=29.91dB, PSNR=31.06dB,
SSIM=0.4820, Interpolation SSIM=0.4720, NLM3D [7] SSIM=0.5122, LRTV [8]

PSNR=30.88dB, PSNR=31.14dB, PSNR=inf, SSIM=1, Origi-

SSIM=0.5174, Proposed SSIM=0.5361, Proposed nal HR Image
Approach with denoising Approach with preferred
preferred detail preservation
TABLE VI
S UPER RESOLUTION IN ALL THREE DIRECTIONS BY SCALE FACTOR 2 AND AND DENOISING THE (G AUSSIAN 2%)

Noise Metric NN NLM LRTV Proposed

Level Inter-
pola-
tion
PSNR 30.12 30.5 31.67 32.19
1%
SSIM 0.8778 0.8859 0.9063 0.9002
PSNR 30.26 30.58 31.43 31.78
2%
SSIM 0.904 0.9114 0.9096 0.8719
PSNR 29.75 30.00 30.34 30.45
3%
SSIM 0.7636 0.7607 0.6952 0.585
PSNR 29.54 29.62 29.28 29.34
4%
SSIM 0.7882 0.77 0.6927 0.565
TABLE VII
Q UANTITATIVE RESULTS FOR THE RANDOMLY SELECTED SUBJECT
DEGRADED WITH DOWNSAMPLING FACTOR 2 AND DIFFERENT LEVELS
OF NOISE .