PROCEEDINGS OF SPIE

SPIEDigitalLibrary.org/conference-proceedings-of-spie

Astronomical image restoration and

point spread function estimation with
deep neural networks

Jia, Peng, Wu, Xuebo, Yang, Xiaoshan, Huang, Yi, Cai,

Bojun, et al.

Peng Jia, Xuebo Wu, Xiaoshan Yang, Yi Huang, Bojun Cai, Dongmei Cai,
"Astronomical image restoration and point spread function estimation with
deep neural networks," Proc. SPIE 11203, Advances in Optical Astronomical
Instrumentation 2019, 112030Q (3 January 2020); doi: 10.1117/12.2541083

Event: ANZCOP, 2019, Melbourne, Australia

Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 10 Feb 2020 Terms of Use: https://www.spiedigitallibrary.org/terms-of-use

 Astronomical Image Restoration and Point Spread Function
Estimation with Deep Neural Networks
Peng Jiaa,b,c , Xuebo Wua , Xiaoshan Yanga , Yi Huanga , Bojun Caia , and Dongmei Caia
a
College of Physics and Optoelectronics, Taiyuan University of Technology, Taiyuan, 030024,
China
b
Department of Physics, Durham University, South Road, Durham DH1 3LE, UK
c
Key Laboratory of Advanced Transducers and Intelligent Control Systems, Ministry of
Education and Shanxi Province, Taiyuan University of Technology, Taiyuan, 030024, China

ABSTRACT
From ground-based extremely large telescopes to small telescope arrays used for time domain astronomy, point
spread function plays an important role both for scientific data post-processing and instrument performance
estimation. In this paper, we propose a new method which can restore astronomical images and obtain the
point spread function of the whole optical system at the same time. Our method uses simulated high resolution
astronomical images and real observed blurred images to train a deep neural network (Cycle-GAN). The Cycle-
GAN contains a pair of generative adversarial neural networks and each generative adversarial neural network
contains a generator and a discriminator. After training, one generator (PSF-Gen) can learn the point spread
function and the other generator (Dec-Gen) can learn the deconvolution kernel. We test our method with
real observation data from solar telescope and small aperture telescopes. We find that the Dec-Gen can give
promising restoration results for solar images and can reduce the PSF spatial variation for images obtained by
smaller telescopes. Besides, we also find that the PSF-Gen can provide a non-parametric PSF model for short
exposure images, which would then be used as prior model for PSF reconstruction algorithms in adaptive optics
systems.
Keywords: Point spread function reconstruction, Deep neural network, Cyclegan, Image restoration

1. INTRODUCTION
Ground–based optical observations are affected by many different effects such as the atmospheric turbulence,
thermal or gravity deformations of the optical system. The imaging process of ground based telescopes can be
modelled as:
Img(x, y) = [Obj(x, y) ∗ P SF (x, y)]pixel(x,y) + N oise(x, y), (1)

where Obj(x, y) and Img(x, y) are the original and observed images, P SF (x, y) is the point spread function (PSF)
of the whole optical system, []pixel(x,y) stands for the pixel response function of the detector and N oise(x, y)
stands for noise from the background and the detector. PSF can reflect real-time performance of an astronom-
ical instrumentation and is important for post-processing methods design in many different observation modes,
including: photometry, astrometry, transient detection, image restoration and even spectral data processing.
For ground based telescopes, the PSF has highly spatial and temporal variations and can not be described by
contemporary analytical PSF model. For telescopes with adaptive optic system, PSF reconstruction with forward
calculation is widely used1, 2 and has reached remarkable results.3–8 However, the forward PSF reconstruction
method requires AO telemetry data, which is not an easy task for many telescope. Recent work proposes to use
real observation images to further restrict the results obtained by PSF reconstruction method.9 This method
can give a promising results for images obtained by multi-conjugate adaptive optic system. As more and more
astronomical images are obtained by different telescopes, including high resolution images obtained by space
based or ground based telescope with adaptive optic system and low resolution images obtained by ordinary
ground based telescopes, is it possible to use the properties of these images to design an algorithm to restore
blurred images and obtain the PSF model at the same time?

Further author information: (Send correspondence to Peng Jia)

Peng Jia: E-mail: robinmartin20@gmail.com, Telephone: 86 133 6341 5009
Advances in Optical Astronomical Instrumentation 2019, edited by Simon Ellis, Céline d'Orgeville, Proc. of
SPIE Vol. 11203, 112030Q · © 2020 SPIE · CCC code: 0277-786X/20/$21 · doi: 10.1117/12.2541083

Proc. of SPIE Vol. 11203 112030Q-1

Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 10 Feb 2020
Terms of Use: https://www.spiedigitallibrary.org/terms-of-use
 2. METHOD AND RESULTS
The Cycle-GAN is an unsupervised learning algorithm, which contains a pair of Generative Adversarial Networks
(GAN).10 Different from many supervised neural networks, the Cycle-GAN does not need pairs of images as
training set. Given two sets of images with different properties, one GAN learns the image mapping and the
other GAN learns the inverse mapping. Under the constrain condition that the mapped image after inverse
mapping should be similar to itself and vise versa (cycle consistency loss), the Cycle-GAN can learn the mapping
function between two spaces represented by different images.
In this paper, according to different observation targets, we choose corresponding high resolution images as
reference, which are either obtained by high resolution observation or simulated images. Then we use blurred
images and high resolution images to train the Cycle-GAN. The Cycle-GAN we used in this paper has two
GANs, which contain a generator with 6 convolutional layers and 5 residual blocks and a discriminator with 4
convolutional layers and a sigmoid function as activation layer as shown in figure 1.
We train the Cycle-GAN with 2000 to 6000 iterations in our computer with two Nvidia 1080 Ti GPUs. It costs

Figure 1. The Cycle-GAN contains two GANs and each GAN contains a Generator and a Discriminator. All the
Generators have the same structure and all the Discriminators have the same structure. The Generator which learns the
point spread function is called PSF-Gen and that learns the deconvolution kernel is called Dec-GAN. The structure of
the PSF-Gen is shown in this figure.

around 10 minutes to restore 200 frames of images. The trained Cycle-GAN can give promising results for image
restoration as shown in figure 2.

3. PSF ESTIMATION RESULTS

As the number of layers in DNN becomes larger, the representative ability of DNN becomes better. However,
there is a very high risk that the DNN can remember high resolution images and generate artifacts that may cheat
human-beings. So it is important to know contents that DNN learns. We can obtain the impulse response of
the DNN through inputting an image which only contains a point. We obtain the impulse response of PSF-NET
from our Cycle-GAN which is trained by short exposure solar images from G-band and the results are shown in
figure 3.

Proc. of SPIE Vol. 11203 112030Q-2

Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 10 Feb 2020
Terms of Use: https://www.spiedigitallibrary.org/terms-of-use
 Figure 2. Restored images by the Cycle-GAN. The first and second columns are original and restored real observed solar
images in G-band and H-alpha band. We use several speckle reconstructed images obtained as reference to restore these
images. Images in the third column are original and restored images from a small aperture wide field telescope. Reference
images are Monte Carlo simulated images with Gaussian function with full width half magnitude of 2 pixels as the PSF.
From these figures, we can find that the Cycle-GAN can give promising results.

Figure 3. Pulse response of the PSF-NET. It is the mean PSF of short exposure PSFs in these solar images.

ACKNOWLEDGMENTS
This work is supported by National Natural Science Foundation of China (NSFC)(11503018), the Joint Research
Fund in Astronomy (U1631133) under cooperative agreement between the NSFC and Chinese Academy of Sci-
ences (CAS). PJ is supported by the China Scholarship Council to study at the University of Durham. The solar
images used in this paper were obtained with the New Vacuum Solar Telescope in Fuxian Solar Observatory of
Yunnan Astronomical Observatory, CAS and the wide field night time observation data were obtained with a 80
cm survey telescope in Purple Mountain Observatory, CAS.

Proc. of SPIE Vol. 11203 112030Q-3

Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 10 Feb 2020
Terms of Use: https://www.spiedigitallibrary.org/terms-of-use
 REFERENCES
[1] Veran, J.-P., Rigaut, F., Maitre, H., and Rouan, D., “Estimation of the adaptive optics long-exposure
point-spread function using control loop data.,” J. Opt. Soc. Am. A, Vol. 14, No. 11, p. 3057 - 3069 14,
3057–3069 (Nov. 1997).
[2] Gendron, E., Clénet, Y., Fusco, T., and Rousset, G., “New algorithms for adaptive optics point-spread
function reconstruction,” aap 457, 359–363 (Oct. 2006).
[3] Peter, D., “Point-spread function reconstruction for the ground layer adaptive optics system ARGOS,” in
[Adaptive Optics Systems II], procspie 7736, 77364R (July 2010).
[4] Villecroze, R., Fusco, T., Bacon, R., and Madec, P.-Y., “PSF reconstruction for MUSE in wide field mode,”
in [Adaptive Optics Systems III], procspie 8447, 84475W (July 2012).
[5] Jolissaint, L., Neyman, C., Christou, J., and Wizinowich, P., “Adaptive optics point spread function re-
construction project at W. M. Keck Observatory: first results with faint natural guide stars,” in [Adaptive
Optics Systems III], procspie 8447, 844728 (July 2012).
[6] Witzel, G., Ghez, A., Meyer, L., Britton, M., Fitzgerald, M., Sitarski, B., Campbell, R., and Matthews, K.,
“Modeling anisoplanatism for the laser guide star adaptive optics system at the keck observatory,” SPIE
Proc. 9148-266 (2014).
[7] Martin, O. A., Correia, C. M., Gendron, E., Rousset, G., Gratadour, D., Vidal, F., Morris, T. J., Basden,
A. G., Myers, R. M., Neichel, B., and Fusco, T., “PSF reconstruction validated using on-sky CANARY data
in MOAO mode,” in [Adaptive Optics Systems V], procspie 9909, 99091Q (July 2016).
[8] Gilles, L., Wang, L., and Boyer, C., “Point spread function reconstruction simulations for laser guide star
multi-conjugate adaptive optics on extremely large telescopes,” in [Adaptive Optics Systems VI], 10703,
1070349, International Society for Optics and Photonics (2018).
[9] Beltramo-Martin, O., Correia, C. M., Ragland, S., Jolissaint, L., Neichel, B., Fusco, T., and Wizinowich,
P. L., “PRIME: Psf Reconstruction and Identification for Multiple sources characterization Enhancement.
Application to Keck NIRC2 imager,” arXiv e-prints , arXiv:1903.02772 (Mar 2019).
[10] Zhu, J., Park, T., Isola, P., and Efros, A. A., “Unpaired image-to-image translation using cycle-consistent
adversarial networks,” international conference on computer vision , 2242–2251 (2017).

Proc. of SPIE Vol. 11203 112030Q-4

Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 10 Feb 2020
Terms of Use: https://www.spiedigitallibrary.org/terms-of-use