Natural Image Inpainting through Examplar-Based Approach using local and Non Local Patch Sparser

K John Peter1, Dr. K. Senthamarai Kannan,Dr. S Arumugam,G.Nagarajan

Abstract: This paper focuses on the examplar-based inpainting algorithm through local patch sparser. In this three basic steps involved which include patch selection, local and non local patch distribution and patch inpainting. In the patch selection, a patch on the missing region boundary with the highest priority is selected. Then based on the selected patch local and non local patches were defined using the PDE (Partial Differential Equation) to identify the edges and flat surface. Finally to inpaint a selected patch on the boundary of missing region, we use a sparser based combination of examplars to infer the patch in a framework of sparse representation. This linear combination of patches are regularized by the sparseness prior (regularization) on the combination coefficients. More importantly, the inpainted patches are more consistent with their surrounding textures or structures due to the local patch consistency constraint. This will produce effective result on the natural images with missing textures synthesizes. KeywordsInpainting, Exemplar, Non Local, Exemplar Approach
have focused on the concept of exemplar-based synthesis [5, 6, 7]. In these techniques, a best match sample from the source region is found and copied directly into the target region. The main benefit of techniques based on exemplarbased synthesis is that they can utilize nonlocal information. This is particularly useful in situations where the correct information for filling in a region is located far away from that region. The main drawback is that the nonlocal information is used in a very limited way. By using only the best match sample, the method runs the risk of choosing a sample that is corrupted, or not a perfect match. However, an image with redundant content could have several samples that could be combined to form a more robust estimate of the missing information. Therefore, we propose that improved inpainting quality can be achieved using nonlocal information from multiple samples within the image.

I. INTRODUCTION

econstruction of missing or damaged portions of images is an ancient practice used extensively in artwork restoration. Also known as inpainting or retouching, this activity consists of filling in the missing areas or modifying the damaged ones in a non-detectable way by an observer not familiar with the original images [2]. Applications of image inpainting range from restoration of photographs, films and paintings, to removal of occlusions, such as text, subtitles, stamps and publicity from images. In addition, inpainting can also be used to produce special effects. Traditionally, skilled artists have performed image inpainting manually. But given its range of applications, it would be desirable to have image inpainting as a standard feature of popular image tools such as PhotoShop. Recently, Bertalmio et al [2] have introduced a technique for digital inpainting of still images that produces very impressive results. Their algorithm, however, usually requires several minutes on current personal computers for the inpainting of relatively small areas. Such a time is unacceptable for interactive sessions and motivated us to design a simpler and faster algorithm capable of producing similar results in just a few seconds. Newer approaches to the problem of image inpainting

Recently, the category of exemplar-based inpainting scheme is proposed, in which the visible parts of the image serve as an source set of examples to infer the target regions. That is, the missing data is replicated rather than synthesized from available information. Compared with other kinds of approaches, exemplar- based approach is very effective in reducing the undesired blurring artifacts and applicable to both the small and large image gaps. Harrison [8] first presents an exemplar-based algorithm focusing on the object removal, but in this work the linear structures are often overruled by nearby noises. Next, Drori et al [6] iteratively approximates the missing regions from coarse to fine levels using the principle of self-similarity. They show impressive experimental results but the reported computation time is extremely long (between 83 and 158 minutes for a 384 223 sized image). Although some fast algorithms [7] are developed, the visual quality is accordingly degraded. Criminisi et al [9] exploit a patch- based algorithm, in which the filling order is decided by a predefined priority function to ensure that the linear structures will propagate before texture filling to preserve the connectivity of object boundaries. The performance is compatible to previous techniques and better speed efficiency is obtained. Nevertheless, we will show that the priority function adopted in [9] is not well- defined and becomes very unreliable after numbers of iterations. Further, for different ordinary images, the variations of relative importance between the texture and the structure characteristics have still been ignored. The main contribution of this paper is a novel approach to exemplar-based inpainting using the concept of nonlocal-

means. In this framework, the relative contribution of each sample to the reconstruction of a target pixel is determined using a weighted similarity function and aggregated to form the missing information. Thus, the proposed method makes use of all relevant nonlocal image information. The method can also be integrated into existing exemplar-based inpainting techniques to provide improved visual quality.

This paper is organized as follows: Section II proposes Exemplar Based Approach. Section III proposes Image Inpainting while the experimental result is shown in section IV and conclusion is defined in section V.

First, for the convenience of explanation, similar notations that used in the inpainting literature [3][9] are adopted. The entire image is composed of two disjoint regions: the source region and the target region . The source region is defined to be the visible part and the target region is the missing one. Additionally, represents the pixel set of the target region boundary. 1) Initializing the Target Region, in which the initial missing areas are extracted and represented with appropriate data structures. 2) Computing Filling Priorities, In which the predefined computation are filled. 3) Searching Example and Compositing,in which most simple example are filled easily 4) Updating Image Information, in which the

II. EXEMPLAR-BASED INPAINTING

It is important to pro- vide a brief overview of the concepts underlying exemplar- based inpainting techniques. The first attempts at inpainting propagated image information from the source region to the target region through the process of diffusion. Hence, only information that was local to the target region was used. This heavy reliance on local redundancy is problematic in situa- tions where the missing information in the target region is not characterized by local information. Exemplar-based meth- ods address this issue by using nonlocal information from the image to fill in the target region. Sup- pose we wish to fill in a patch p , centered on pixel p, that is at least partially within the target region .

boundary of the target region and the required information for computing filling priorities are updated.

III. THE INPAINTING ALGORITHM Images may contain textures with arbitrary spatial discontinuities, but the sampling theorem [8] constraints the spatial frequency content that can be automatically restored. Thus, for the case of missing or damaged areas, one can only hope to produce a plausible rather than an exact reconstruction. Therefore, in order for an inpainting model to be reasonably successful for a large class of images the regions to be inpainted must be locally small. As the regions become smaller, simpler models can be used to locally approximate the results produced by more sophisticated ones. Another important observation used in the design of our algorithm is that the human visual system can tolerate some amount of blurring in areas not associated to high contrast edges [9]. Thus, let be a small area to be inpainted and let be its boundary. Since is small, the inpainting procedure can be approximated by an isotropic diffusion process that propagates information from into . A slightly improved algorithm reconnects edges reaching (for instance, using an approach similar to the one described in [12]), removes the new edge pixels from (thus splitting into a number of smaller sub-regions), and then performs the diffusion process as before. The simplest version of the algorithm consists of initializing by clearing its color information and repeatedly convolving the region to be inpainted with a diffusion kernel. is a one-pixel thick boundary and the number of

Fig1: The image represents the simple Examplar based inpainting approach The size of this patch may vary depending on the situations and can be greater than or equal to a single pixel. In exemplarbased inpainting, a neighborhood surrounding p (whose size is greater than or equal to p ) is compared using a similarity measure to other neighborhoods centered around other points within the image (eg., q and r ). For example, the sum of squared differences (SSD) cost function was used in [6]. The patch that minimizes the cost function is selected as the best match and its contents are copied into the part of p that lies inside the target region.

B. Robust image inpainting

iterations is independently controlled for each inpainting domain by checking if none of the pixels belonging to the domain had their values changed by more than a certain threshold during the previous iteration. Alternatively, the user can specify the number of iterations. As the diffusion process is iterated, the inpainting progresses from into . Convolving an image with a Gaussian kernel (i.e., computing weighted averages of pixels neighborhoods) is equivalent to isotropic diffusion (linear heat equation). Our algorithm uses a weighted average kernel that only considers contributions from the neighbor pixels (i.e., it has a zero weight at the center of the kernel). B. Preserving Edges The simplest version of the algorithm can introduce artifacts (noticeable blurring) when crosses the boundaries of high contrast edges (Figure 3 (front left)). In practice, intersections between and high contrast edges are the only places where anisotropic diffusion is

straight lines are shown in Figure 3 (front right).

Fig 3: The image represents simple exemplar based image inpainting.

C.Selecting the component weights For different images, the relative importance of texture and structure characteristics should be carefully considered. For example, in Fig. 5, the target region is filled with four different weight settings of confidence and data terms. Higher weights for the data and the confidence terms are respectively given in the settings 1, 3 and the settings 2, 4. For the image, the outputs with the settings 1, 3 are obviously preferred by end users because the reconstructed tower contours are more reasonable. It is also worthy to notice that, even with similar settings, the results may be visually different such as the presence. There are strong connections between the component weights and the resulting image. The combination of the priority function needs more flexibility to achieve better visual quality according to the image contents. Therefore, we define the robust priority function RP(p) as follows. RP( p) = RC ( p) + D( p) , 0 , 1 , (1)

Fig 2: The image represents Image of degraded circle lattice.

Creating the mask used to specify the regions to be inpainted is the most time-consuming step of the inpainting process, requiring user intervention. Since our algorithm can inpaint an image in just a few seconds, it can be used for interactive construction of tight masks. We exploit this interactivity to implement edge reconnection by defining diffusion barriers, which are boundaries for the diffusion process inside . This accomplishes a result similar to boundary reconstruction and anisotropic diffusion, but without the associated overheads. In practice, a diffusion barrier is a two-pixel wide line segment. As the diffusion process reaches a barrier, the reached pixel has its color set, but the process stops. Figure 3 illustrates the idea, with the clear crossing lines in Figure 3 (back left) representing the inpainting domain. The simple diffusion-based inpainting algorithm produces blurred spots at the intersections between and high contrast edges (see the small circles in Figure 3 (front left)). By appropriately adding diffusion barriers (line segments across the mask in Figure 3 (back right)), the user stops the diffusion process from mixing information from both sides of the mask. The resulting

where and are respectively the component weights of the confidence and the data terms. Note that + = 1 . Based on our observations, the bottleneck of exemplarbased inpainting algorithm lies in the search for best matching example from source regions. Even if we need more computations in the proposed priority function (two multiplications and one addition versus one multiplication), the overall computation is fast enough and very close to that of [9]. Our experiments confirm this claim. Therefore, it may not be reasonable but currently acceptable to decide an optimal setting of the component weights by exhausted computer search. Of course, finding more efficient searching algorithms will be one of future works. IV . EXPERIMENTAL RESULT We conduct some experiments and compare the so-

obtained results with the conventional approach [9. The testing data are carefully selected from both natural and synthetic images.

V. CONCLUSION
This paper introduced a novel approach to exemplar-based in- painting using nonlocal-means. By utilizing nonlocal image information from multiple samples and selecting the samples used based on the underlying image content, it was demonstrated that improved visual quality can be achieved over the approach used in current exemplar-based inpainting techniques.

The Proposed Paper is a simple and fast inpainting algorithm based on an isotropic diffusion model extended with the notion of user-provided diffusion barriers. The results produced by this simple model are, in many cases, comparable to previously known non-linear inpainting models.
Fig 4: The image represents Image InPainting through our Approach.

REFRENCES
[1] M. Bertalmio, G. Sapiro,V. Caselles, and C. Ballester, Image inpainting, in Proc. SIGGRAPH, pp. 417-424, 2000. [2] T. Chan and J. Shen, Non-texture inpainting by curvature-driven diffusions, J. Visual Comm. Image Rep., vol. 4, no. 12, pp. 436-449, 2001. [3] S. Rane, G. Sapiro, and M. Bertalmio, Structure and texture filling-in of missing image blocks in wireless transmission and compression applications, in IEEE Trans. Image Processing, vol 12, no. 3, pp. 296-303, 2003. [4] C. Ballester, M. Bertalmio, V. Caselles, G. Sapiro, J. Verdera, Filling-In by Joint Interpolation of Vector Fields and Gray Levels, in IEEE Trans. Image Processing, vol. 10, no. 8, pp. 1200-1211, 2001. [5] A. Criminisi, P. Perez, and K. Toyama, Object Removal by Exemplar-Based Image Inpainting, Proc. IEEE Intl. Conference on Computer Vision, 2003. [6] A. Criminisi, P. Perez, and K. Toyama, Region Filling and Object Removal by Exemplar-Based Image Inpainting, IEEE Trans. Image Processing, vol. 13, no. 9, pp. 1200-1212, 2004. [7] J. Wu and Q. Ruan, Object Removal By Cross Isophotes Exemplar-based Inpainting, Proc. 18th Intl. Conference on Pattern Recognition, vol. 3, pp. 810-813, 2006. [8] A. Buades, B. Coll, and J. Morel, Nonlocal image and movie denoising, Intl. J. of Computer Vision, (to appear), 2007. [9] M. Mahmoudi and G. Sapiro, Fast image and video denoising via nonlocal means of similar neighborhoods, IEEE Signal Processing Letters, vol. 12, no. 12, pp. 839842, December 2005.

Fig 5: The image represents exemplar based Result

The proposed algorithm shows an improvement in PSNR over the current exemplar-based approach for all of the test images. Samples of inpainted images are shown in Figs. 4-6.

Fig 5: The image represents complete Image Inpainting

[10] M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester, "Image inpainting," in ACM Comput. Graph. (SIGGRAPH 2002), July 2000, pp. 417-424. [11] R. Bornard, E. Lecan, L. Laborelli, and J.-H. Chenot, Missing data correction in still images and image sequences, in ACM Multimedia Conf., 2002, pp. 355-361. [12] H. Grossauer, A combined PDE and texture synthesis approach to inpainting, Lecture Notes in Computer Science, vol. 3022, pp. 214-224, 2004. [13] I. Drori, D. Cohen-Or, and H. Yeshurun, Fragment- based image completion, in ACM Comput. Graph. (SIGGRAPH 2003), 2003, pp. 303-312. [14] S. Borikar, K. K. Biswas, and S. Pattanaik, Fast algorithm for completion of images with natural scenes, CS Technical Report, University of Central Florida, April 2004.