Prototyping Color-based Image Retrieval with MATLAB

Petteri Kerminen1, Moncef Gabbouj2 Tampere University of Technology, Pori, Finland Tampere University of Technology, Signal Processing Laboratory, Tampere, Finland
1

Abstract
Content-based retrieval of (image) databases has become more popular than before. Algorithm development for this purpose requires testing/simulation tools, but there are no suitable commercial tools on the market. A simulation environment for retrieving images from database according histogram similarities is presented in this paper. This environment allows the use of different color spaces and numbers of bins. The algorithms are implemented with MATLAB. Each color system has its own m-files. The phases of the software building process are presented from system design to graphical user interface (GUI). The functionality is described with snapshots of GUI.

In the MuVi-project [1] this kind of tool is under construction. It will cover the properties presented above. Research work on content-based image retrieval has been done in [2 6]. The system, which is presented in this paper, is a simulation environment, where MuVis color content based retrieval has been developed and tested.

2.

System development

MATLAB is an efficient program for vector and matrix data processing. It contains ready functions for matrix manipulations and image visualization and allows a program to have modular structure. Because of these facts MATLAB has been chosen as prototyping software.

2.1

System design

1.

Introduction

Nowadays there are thousands or hundreds of thousands of digital images in an image database. If the user wants to find a suitable image for his/her purposes, he/she has to go through the database until the correct image has been found or use a reference book or some intelligent program. Video on demand (VoD) services also requires an intelligent search system for end-users. VoD systems search methods differ slightly from image databases methods. A reference book is a suitable option, if the images are arranged with a useful method, for example: 1) categories: animals, flags, etc, 2) names (requires a good naming technique) or 3) dates. An experienced user can use these systems as well as textual searches (keywords have to be inserted in a database) efficiently. There are situations when a multi-language system has to be used. There a language independent search systems best properties can be utilized. A tool which is based on the images properties can be made language independent. These properties can be for example color, shape, texture, spatial location of shape etc.

Before any m-files have been written, the system design has been done. A system design for the HSV (hue, saturation and value) color system based retrieval process is presented in Figure 1. Similar design has been done for all used color systems.

tesths27

image_pos

hs27read

dif_hsv

hs27conv

hsquan27
Figure 1: Function chart for HSV color space with 27 bins histogram.

Tesths27 is the main function for this color system and this number of bins. It calls other functions (hs27read, dif_hsv and image_pos) when needed. Each color system has a main function of its own and variable number (2 3) of sub-functions. If there is no need for color space conversion there are 2 functions, otherwise 3 functions on the first branch of the function chart. The function call of the main function is: matches=tesths27(imagen,directory,num) The variable imagen specifies the query images name and path. The directory is a path of the image database and num is a desired number of retrieved images.

directory with an almost similar algorithm as in the case of the query image. The difference is that now there will be a histogram difference calculation between the query images and current images histogram. Finally the image_pos function will be used to put a query image and the desired number of best match images on the display.

2.3

Linking

2.2

Functions

At this moment there are functions implemented for four color spaces: HSV, L*a*b*, RGB and XYZ [7]. Each color space has from 2 to 4 implementations for different numbers of bins. There are altogether 14 main functions. For some color systems it is possible to make these functions dynamic, i.e. dynamic histogram calculation. Every color system / bin combination requires its own histograms and these can be made only with an exhaustive method (pixel by pixel). Histogram calculation takes - 5 minutes per image, each approximately 320240 pixels, depending on the complexity of the color space on 150 MHz Pentium. Thus it is not reasonable to let the user select a bin number freely, especially in the case of large databases. The functions have been named so that the names contain information of the color space used, the purpose of the functions and the number of used bins. Some functions, for example image_pos, have been used by many or all main functions and these functions have not been named as described above. The main function checks, if the function call is correct. If the query images name doesnt contain a path, the function assumes that the image is situated in the database directory. In addition to this, the main function checks, if the query image already has a histogram in the currently used database. If the required histogram is not there, the image read (for example hs27read) function is called. This function also normalizes pixel values and arranges image matrix data to a vector format. After that stage a color space conversion function (if needed) is called. Finally a quantization function builds the histogram with the correct number of bins. The histogram will then be saved into the database directory. If the histogram already exists there, the three previous steps will not be executed. Now the query image has been analyzed. Then the main function will go through all images in the database

It is not possible to use a program before the main function and sub-functions are connected to each other. The main function will be called from the command line or through the graphical user interface, which will be presented later in this paper. In both cases the function call will contain the same arguments. For multi-level search purposes separate main functions have been implemented, but it is possible to utilize normal functions and add one parameter, where the best matches array can be transferred for second a stage comparison function. The main function calls an image read function with the images name. The histogram will be returned to the main function. If a color space conversion is needed, the conversion function will be called from the read function with r, g and b vectors. The histogram will be returned to the calling function. Finally the histogram build function will be called with converted color vectors. This function returns a quantized histogram, which will go through all functions until it achieves the main function. The main function calls the histogram difference function with two histogram vectors and will get a difference value as a response. The difference function uses Euclideandistance calculation, but it can be easily changed to another algorithm due to the modularity of the program. If the difference is smaller than largest difference on a best match table, the current result will be written over the last result on the best match table. After that the table is arranged again in an ascending order of distance. When all the images have been analyzed, the sorted best match table, the number of desired output images, the query images name, the search images path and the database path are transferred to the image_pos function. These values can be transferred into larger components (vectors/containers). Now the program works faster with several input arguments, because there is no need for picking up variables from a container.

2.4

Graphical user interface

The graphical user interface (GUI) is an important part of software development. The designing of the GUI have to solve the following problems: learning time, speed of performance, rate of errors by users, retention over time, and subjective satisfaction [9]. This software is, at the moment, intended to be used only for testing purposes. The most important property of this software is that the results of different test queries can be seen quickly and the results can be saved safely on a disk. Thus the visual layout is not as important as in case of a commercial software product.

In Figure 2 the first screen on GUI is presented. The purposes of the buttons, menus and other components will be presented later. If this software is developed into a commercial product, the menu bar will be disabled in the future and the exit and help buttons will be added on the canvas.

Figure 4: Color system selection from a popup menu. The second menu is disabled because a one-level search is selected. quantized or a more accurate histogram is used in one loop (one color system). Figure 2: GUI before the search image selection. In Figure 3 the search screen is presented just before starting a search. The user is shown a search image, and in this way he/she can be sure that the search will be made with the correct image. In a multi-level search two different color systems / histograms are used. During the first loop the roughly quantized histograms are used and during the second loop the more accurate histograms are utilized for the best matches from the first loop. The color system on the second loop can be either the same as on the first loop or a different one. For queries with one-level search the selection of a second color system is disabled. The user can select the number of retrieved images at the final stage. The software can be linked to many image databases and the user can select a database where the query will be directed. The user can select a search image either from the same database where the query will be directed to (default) or from any directory in his/her PC. The selection will be made with the file open dialog, which is presented in Figure 5. The form can be cleared with the Reset button. A query is executed with the Search button. Finally the results of the search will appear on the screen in a separate window, as presented in Figure 6. Earlier [8] the software opened each image in a separate window and evaluating/saving the results is more difficult than after the improvement. In the top left top corner is the original query image. Below that image the best matches are presented in a descending order of similarity from left to right and from top to bottom. The user can select suitable images for further use with the Copy selected or the Print selected buttons. The New search button closes this form and goes back to the original search form. The Search similar button executes a new search where a query histogram is composed of histograms of the selected images. If the user has selected a larger number than 21 as Number of matching images, the best matches will be shown on multiple screens. The user can browse these pages with the Previous page and Following page buttons.

Figure 3: GUI just before running a query. The results of the query will be presented on the screen in the format which is presented in Figure 6.

3.

Using the software

The first screen has already been presented in Figure 2. The user can choose from pop-up menus (see Figure 4), if the search is made with one a color system or as a multi-level search. In a one-level search a roughly

Figure 5: The query image selection dialog. The language of the dialog depends on the language of the operating system used.

Figure 6: The results of a query will be presented graphically.

4.

Summary

References
[1] Alaya Cheikh F et. al. MUVIS: A System for Content-Based Indexing and Retrieval in Large Image Databases. Proceedings of SPIE Storage and Retrieval for Image and Video Databases, San Jose, 1999,vol. 3656: 98 106. [2] Sawhney H. S., Hafner J. L., Efficient Color Histogram Indexing for Quadratic Form Distance Functions, IEEE Transactions on Pattern Analysis and Machine Intelligence, 1995, vol. 17 no. 7: 729 736. [3] Vellaikal A. et. al. Content-Based Retrieval of Color and Multispectral Images Using Joint Spatial-Spectral Indexing, SPIE vol. 2606: 232 243. [4] Ng V. et. al. Adaptive Histogram Indexing, SPIE vol. 2606: 202 211. [5] Smith J. R., Chang S. F. Automatic Image Retrieval Using Color and Texture, 1995-6, http://www.ctr.columbia.edu/ ~jrsmith/html/pubs/PAMI/pami_final_1.htm. [6] Flickner M. et. al. Query by Image and Video Content: The QBIC System, IEEE Computer 1995 vol. 28 no. 9: 23 32. [7] Jain A. K. Fundamentals of Digital Image Processing, Prentice Hall, 1989: 66 - 71. [8] Kerminen P, Gabbouj M. Image Retrieval Based on Color Matching. Proceedings of FINSIG99, Oulun yliopistopaino, 1999: 89 93. [9] Shneiderman B. Designing the User Interface, Addison Wesley Longman, 1998: 14 16.

The color content-based retrieval requires algorithms, which give visually correct results. Correctly working algorithms can not be chosen before simulations. The software presented in this paper is intended to be used for testing purposes. Some operations will be implemented, if the software is developed into a commercial product. Some modifications are under construction. This software has been used as a testing platform for histogram quantization tests. The modularity of this program makes it possible to take new algorithms as a part of the software in a short time. MATLAB makes quick prototyping possible. A possibility to save figures (search results) directly on a disk is a fulfillment of the programs requirements. After the results have been analyzed visually, the best algorithms will be taken as a part of the final software.

5. Acknowledgements
This work has been founded by the European Union ERDF, the Technology Development Centre Tekes, Alma Media, the Helsinki Telephone Company, Nokia Research Center, the Satakunta High Technology Foundation and Ulla Tuominens Foundation.

Address for correspondence. Petteri Kerminen P.O. BOX 300 FIN-28101 PORI Finland petteri.kerminen@pori.tut.fi