Semantic Inter-Media Fusion Design for a Content-Based Medical Image Retrieval System

Semantic Inter-Media Fusion Design for a Content-Based Medical Image Retrieval System Roxana Teodorescu2,3 and Daniel Racoceanu1,2 1

IPAL-Image Processing Access and Language, UMI CNRS 2955, NUS, UJF, I2R - A*STAR, Singapore [email protected] 2 University of Besan¸con, France 3 ”Politehnica” University from Timisoara, Romania

Abstract. This article treats about a semantic content-based medical image retrieval system (CBMIR), focusing more precisely on the inter-media fusion module between the medical images and the associated medical reports. In the proposed CBMIR, the semantic indexing and fusion are based on the Unified Medical Language System’s (UMLS) Metathesaurus, a very large, multi-purpose, and multi-lingual biomedical and health-related concepts vocabulary. Coherent modeling approach of the indexing, the fusion and the retrieval processes is critical for CBMIR systems quality. We design the semantic fusion approach using the Unified Modeling Language (UML). A set of UML diagrams illustrating the steps of the semantic CBMIR methodology are proposed. The fusion operates before the query processing (retrieval) and works at an UMLS-compliant conceptual indexing level. By evaluating the probabilistic, fuzzy and evidence-based approaches for the fusion and different similarity functions for the retrieval processes, we introduce an analysis of the different approaches, trough the medical image retrieval benchmark of the Cross Language Evaluation Forum (CLEF) 2006.

1

Introduction

In the medical field, digital images are produced in huge quantities and used generally seldom for direct diagnosis and therapy. Even if the introduction of DICOM 4 medical image format standardization and PACS 5 medical information storage and management systems represent important milestones in the medical field, efforts still have to be done for using these standards efficiently and effectively for diagnosis assistance, teaching and research. In the same way that PACS expands on the possibilities of a conventional hard-copy medical image storage system by providing capabilities of off-site viewing and reporting (distance education, telediagnosis) and by enabling practitioners at various physical locations to access the same information simultaneously, (teleradiology), the ContentBased Medical Image Retrieval (CBMIR) opens the gate to the next generation of medical procedures. For instance, the CBMIR systems could provide advanced diagnosis assistance, setup semantic links between the related medical information for improving the patient health care. Furthermore, in the near future, data-mining could be used for research applications, for the medical queries expansion and for all the potential Evidence-Based Medicine (EBM) and Image Based Reasoning (IBR) [5, 14] applications generated by the similarity-based image retrievalFinally, decision support systems in radiology and computer-aided diagnostics for radiological practice need powerful data and metadata management and retrieval [11, 1]. The image-based queries will most likely not be able to replace text-based ones. Nevertheless, they have the potential to be a very good complement to text-based search based on their characteristics. Our work focuses on a general semantic CBMIR system using a web-available medical metathesaurus - UMLS (Unified Medical Language System). We treat the fusion between the conceptual indexes of the medical images and medical reports. This approach present the advantage to use an up to date medical metathesaurus (updates are published many times a year), taking into account many languages, a huge number of biomedical concepts concepts and essential symbolic and statistic relations between them. This structured medical metathesaurus offers the opportunity of homogeneous fusion between UMLS-compliant concepts coming from different medical media (images, reports ...), but also automatic query expansion and rule extraction. This paper is organized like follows: The section 2 summarizes a brief state of the art in the CBMIR, focusing on actual challenges like the semantic gap and the use of complementary medical media. The Section 3 introduces the principle of the UMLS- based CBMIR, system which constitute the framework of this paper. Section 4 introduces the Unified Modeling Language (UML) approach for modeling our CBMIR system. The next section 5 treats the fusion module methodology description, including a comparative study of probabilistic, fuzzy and evidence-based theories methodologies. We consider similarity measures and techniques and apply them to our data, by choosing the most appropriate one. The results obtained on the CLEF 2006 medical image retrieval benchmark are introduced in the section 6. Finally, the main conclusions and the future development directions are presented. 4 5

Digital Imaging and COmmunication in Medicine Picture Archiving and Communication Systems

2

Brief overview of the Content-Based Medical Image Retrieval systems

Content-based image retrieval (CBIR) is the application of computer vision to the image retrieval problem, i.e. the problem of searching for digital images in large databases. ”Content-based” means that the search makes use of the contents of the images themselves, rather than relying on textual annotation or human-imputed metadata. The visual features, used for indexing and retrieval, are classified in [8] into three classes: – primitive features that are low-level features such as color, shape and texture; – logical features that are medium-level features describing the image by a collection of objects and their spatial relationships; – abstract feature that are semantic/contextual features.

Fig. 1. Pre-retrieval fusion in the proposed semantic CBMIR.

The obvious loss of information from image data to a representation by abstract features is called the semantic gap [19] and constitutes nowadays a major research topic in this field. In order to reduce this semantic gap, since general systems have low retrieval performances, specialized retrieval systems have been proposed in literature. Indeed, the more a retrieval application is specialized for a limited domain, the smaller the gap can be made by using domain knowledge [6, 13, 18]. Nonetheless, those concepts for medical image retrieval are limited to a particular modality, organ, or diagnostic study and, hence, usually not directly transferable to other medical applications [15]. In [15] the authors propose a general content-based Image Retrieval in Medical Applications (IRMA). This system aims to develop and implement high-level methods for CBMIR with application to medico-diagnostic tasks on radiologic image archive. Based on a general structure for semantic image analysis that results in 6 layers of information modeling, IRMA is implemented with distributed system architecture suitable for large databases. Until now, this system was only used for basic queries regarding the category of images (modality, orientation, body region, biological system). Another important reference in CBMIR systems is the medGIFT project. Textual information as well as structured information plays a much more important role in the medical field. Pre-treatment of images is equally done. This includes the removal of the background to concentrate research on the main part of the image as well as segmentation of specific objects for retrieval. Efficient and discriminative content-based indexing of the medical image still remains a difficult challenge. Nowadays, classical visual parameters extraction methods have reached a limit, related to the use of the medical knowledge. Integrating the medical domain knowledge into the indexing and retrieval algorithms will certainly constitute one of the main research topic related to the CBMIR in future researches.

3

Use of the Unified Medical Language System for CBMIR

As an attempt to integrate a form of medical knowledge into a CBMIR system, we propose the use of the UMLS (Unified Medical Language System6 ) metathesaurus in order to create a common semantic platform between the medical images and the associated medical reports. The purpose of NLM’s7 (US National Library of Medecine) UMLS is to facilitate the development of computer systems that behave as if they ”understand” the meaning of the language of biomedicine and health. The Metathesaurus part of the UMLS, is a very large, multi-purpose, and multi-lingual vocabulary database that contains information about biomedical and health related concepts, their various names, and the relationships among them. This Metathesaurus is organized by concept or meaning. All concepts are assigned to at least one semantic type from the Semantic Network. This provides consistent categorization of all concepts at the general level represented in this Semantic Network. In order to filter the UMLS concepts and relationships needed for the fusion, we use the Metathesaurus UMLS Knowledge Source. Extracting UMLS Metathesaurus concepts from the medical reports is based on the use of MetaMap8 software. The medical report indexes are represented by corresponding UMLS concepts. For each concept, frequency/inverse frequency indexes of the form tf/idf have been computed. For the medical images, the approach is based on the use of SVMs (Support Vector Machines) to relate the basic features of the whole medical image to a UMLS concept, according to a supervised choice of a learning image set. The indexing corresponds thus to a SVM classification problem, according the OVA (one-vs-all) method. The coefficients obtained for each concept represent a fuzzy confidence degree given by each classifier. The objective of our work is situated further on, focusing on a homogeneous way of fusion of the semantic UMLS indexes of the medical images and medical reports, in order to enhance the performances of the CBMIR system. In order to converge to a simple efficient architecture for an UMLS-based semantic medical image retrieval system, we propose to design it using the UML approach.

4

UML design of the UMLS-based CBMIR system

The Unified Modeling Language is a language for specifying, constructing, visualizing, and documenting the objects of a software system readable by humans and machines. It combines the concepts of the main available Object Oriented (OO) methodologies [4, 10, 17]. OO modeling and design lead to a better understanding of requirements, simpler designs, and more maintainable systems. OO methodology formalized by UML can be used to define the objects involved in the various processes using a series of diagrams specified by user requirements. For a complex project, specialized notational elements are required, as well as consistency between diagrams metamodels [9]. Following the UML approach, we design the main components of the CBMIR system, by focusing on the fusion module. User requirements are initially expressed in terms of use cases. Use Case diagrams are abstract representation of the main processes in a system and the interactions between those processes and external systems or users. For our system, the Use Case diagram contains 4 kinds of users (Figure 2): Administrator, Medical Expert, Student/Teacher/Assistant/Doctor, Medical Researcher and 3 subsystems used for the fusion: the UMLS Metathesaurus, the medical image processing (indexing) service and the text processing (indexing) service Once the Use Case diagram designed, most of the use-cases will become classes. The purpose of the Class diagram is to make the transition between the abstract representation of the Use Case diagram and the implementation of the system. The information retrieved from the subsystems are managed by the InitSys class, which is a business entity, as it performs the data refinement. The Retrieve Data class is a business entity dealing with our own database. The same type of class is represented by the Mine Data class, used for the statistics and research purposes. The UpdateDB class is a business worker, performing updates in our database or in the UMLS. This class is managed by the Connect class and Config class. Query, Fusion and Clustering are also business worker classes, but they perform actions requested by users or by Config. The Config class is an instantiated class used by our system for initializing it, performing the operations for creating the database and the user accounts. This class is used for starting the configuration of the system and is managed by the system’s administrator (Figure 3). The statechart or activity diagram contains any number of internal actions (Figure 4). This diagram contains the actual flow of the data. 6 7 8

UMLS - http : //www.nlm.nih.gov/research/umls/ http : //www.nlm.nih.gov/ http : //mmtx.nlm.nih.gov/

Fig. 2. CBMIR System. Use Case Diagram

5

Fusion module design

The definition of the fusion refers to it like an alliance, a merging process, a combination or a concatenation between elements provided by different sources. The aim of this process is to obtain more precise and accurate data. Steinberg et al.[2] define data fusion as process of combining data to refine state estimates and predictions, suggesting that data fusion contains information fusion and sensor fusion, the difference between the two being made by the process of correlation or estimation used. It’s obvious that the two methods overlap, as some applications can use both techniquesThe Information Fusion includes also the Data Mining. Each of these applications need a database, the resource management, the reason why the Multi-Sensor Integration is included, interacting with these levels of Data fusion[2]. According to those considerations, the main modules of our CBMIR system are: the preprocessing - with the preparation of the data and the link between medical images and medical reports, the fusion layer, and the retrieval layer. Composed by a collection of medical reports and a collection of medical images, our database is a multimedia database one. In the fusion process, the starting point is represented by the data obtained from the image and respectively text processing (indexing) modules. Through the fusion process, we obtain a combination of concepts that we refer to as a medical case indexes. 5.1

Preprocessing

In the clinical practice, a medical case is constituted by one or more medical reports and one or more associated medical images. In our approach, we consider a decomposition in elementary medical cases c formed by one medical report and one associated medical image. The combination of the results of those elementary cases can give a reconstruction of the original medical one, if different. The medical case c will bring thus an indexing from the associated image and respectively medical report:    CU Ij1 , λcimgj CU Ii1 , λctxti1 1 =  ... ...  , Λcimg =  ... ...  c CU Iin , λtxtin CU Ijm , λcimgjm 

Λctxt

Fig. 3. Class diagram

c c with: λctxti = µctxti ∗ νtxt ∗ ωtxt ∗ ϕctxti , l = 1, ..., n il il l l l c c c c λimgjq = µimgjq ∗ νimgjq ∗ ωimgjq ∗ ϕcimgjq , q = 1, ..., m

where: CU I - UMLS concept, µ - fuzzy confidence degree, ν - relative frequency of the concept, ω - spatial localization fuzzy weight and ϕ - data test based feedback or relevance feedback. The spatial localization ω corresponds - for the medical report indexing - to the importance of the section to which the concept belongs. For example, in the < Diagnosis > paragraph of the medical report, the physician synthesized the most important keywords describing the disease - pathology - and the anatomic part. This tag will thus be more important than the < Description > tag, which is more neutral. The feedback ϕ represents the confidence accorded to the extracted concepts. Indeed, for the medical image indexing, some of the SVMs have better performances for a certain type of modality detection, having then naturally a better associated feedback. Those feedback coefficients can come from two different sources: one possibility (used in our application) is to take into account the results from a test dataset classification; another one can come from a relevance feedback process. The indexing confidence degree µ is a fuzzy result directly given by the classifiers (actually SVM - according to the quality of the training dataset) used for the medical image indexing. For the concepts extracted from the text part, the value for this parameter is 1, since those concepts are obviously existing or not. c For the text pre-processing, a text indexing software has been used for calculate the local relative frequency νtxt i of the concept occurrence in the given medical report. If a patch extraction method is used for the image, the local c relative frequency νimg will be computed using the relative weight of this concept versus all the patches of the image. i For the text concepts, this parameter is computed as tf/idf (term frequency /inverse document frequency) or it can be computed as drf (document related frequency). For the image, the value for this parameter is 1, as for one image a global generalization approach has been used (SVM), so the neural system always will give a result.

Fig. 4. State Chart Diagram

5.2

Fusion Methods

There are several fusion methods used, depending on the data that is provided and on the final purpose of the fusion: Low, Intermediate and High Level – Low level fusion - data fusion takes several sources of data and combines them into a new data raw, more informative or more synthetic then any of the original ones [12]; – Intermediate level fusion - feature level fusion combines various features; it can be used to fuze them or to find significant features among the ones extracted in a raw of data. Methods for this type of fusion: e.g. Principal Component Analysis (PCA), Multi-layer Perceptrons (MLP)...; – High level - decision fusion combines decisions coming from several experts; the experts can also return a confidence degree. Methods: voting methods, statistical methods, fuzzy logic based methods ... Study of Probabilistic, Fuzzy and Evidence Based Fusions These types of fusion are used for decision making, when the information is unprecise, uncertain or incomplete. Data fusion imposes in general three major tasks: sensor registration - data gathering, data association and data combination. Typical association techniques are: nearest neighbor search (method which selects the closest item), probabilistic data association (which selects a set of candidates with confidence weights) and multiple hypothesis (association decision making is postponed until enough evidences are collected). Conjunctive vs Disjunctive Fusion Those cases are classified as possibilistic fusions because their sources represent possibilities. Conjunctive fusion is an intersection between fuzzy sets; it assumes the fusion on perfect agrement between the sources - only on intersection the fusion can be done. The minimum function corresponds to this type of fusion, being the most cautious one. Disjunctive fusion is used to fuse sources in conflict, takes one or two sources, but only one of them is considered as being reliable. This type of fusion is based on low or non-overlapping sets. This kind of fusion can be realized with the maximum operator. These two methods of fusion are similar to the probabilistic consensus, but they are extreme models of combination. Operators These fusion techniques have their own operators, depending on the data provided and on their features, as well as on the purpose of the fusion. For our approach we consider the Context Independent Constant Behavior (CICB) [3] operators because they are stable, we don’t need the contextual information or external information. Having noise coming from the text part, we need the stability of this class of operators. Table 2 presents the types of CICB operators [7, 20].

Table 1. Fusion Methods Techniques

Purpose

Possibilistic

Evidence-based

consensus probability distribution

propositions that are not mutually exclusive

Data

Distribution π Measure Π supx∈Ω π(x)=1 ∀(A, B) ∈ ℘(Ω)2

Formulae

Π(A) = supx∈A π(x)

Operations

S Π(A B) = max(Π(A), T Π(B)) N (A B) = min(N (A), N (B))

Behavior

non-associative non-commutative

Basics

Probabilistic membership of a fuzzy set in the possible values Distribution p PMeasure P x∈Ω p(x)=1 ∀(A, B) ∈ ℘(Ω)2

Distribution [0,1] P Measure m X⊆Ω m(X)=1 ∀A ⊆ Ω m(X) = P K X1 T X2 =X m1 (X1 )m2 (X2 ) P P (A) = x∈A p(x) −1 K = P 1 − X1 T X2 =Φ m1 (X1 )m2 (X2 ) ∀ feature vectors g, oi object, classification ϕj moi ,ϕj (Soi ,ϕj L |g) = P (oi , ϕj |g) m(g) = moi ,ϕj (g) ∀singletonsubsetsS oi ,ϕj ∈ Ω P Bel(Soi |g) = ϕj m(Soi ,ϕj |g) associative commutative

S P (A B) = P(A)+P(B) T P (A B) = P(A)*P(B) if independent associative commutative

If we refer to operators as disjunctive and conjunctive, than the conjunctive operators create a consensus between the data imputed, reduces less certain sources by giving more confidence to the source which has the smallest measure. This kind of operators searches for a simultaneous satisfaction of the criterions. The disjunctive operator increases the confidence in the source with the most certain measure, or the one with the biggest value. A compromise operator provides global measure, intermediate between the partial measures given by the sensors. Redundancy can be overcome by fuzzy operators. Wald [16] suggests that Dempster - Shafer applies the best for the decision fusion, Table 2. Context Independent Constant Behavior Operators

Basic concept

Probabilistic

Possibility

probabilities

possible values

Evidence mass function belief

∀(x, y) ∈ [0, 1]2 , E - event T-norm i(x, y) ≤ min(x, y) Ln to be evaluated 2 i=1 mi (A) = 1/(1 − k) P x1 ,x2 information T-conorm ∀(x, y) ∈ [0, 1] , T T i(x, y) ≥ max(x, y) B1 B2 ... Bn =A = from sensors m1 (B1 )m2 (B2 )...mn (Bn ) min(x, y) Operation p(E|x1 , x2 ) = ≤ m(x, y) p(x2 |E)p(x1 |E)p(E) P T k= T p(x1 )p(x2 ) ≤ max(x, y), B1 B2 ... Bn =∅ mean in case of m 6= min, m (B )m (B )...m 1 1 2 2 n (Bn ) independence m 6= max, between sources m(x, y) = m(y, x)

the last level of fusion. The highest level of fusion has as result a decision, whereas our level of fusion, the intermediate level, has as result feature fusion. We use the operators represented in Table 3. 5.3

Retrieval

The fused data are used by next for the retrieval process. The most used one similarity function is the KNN (K Nearest Neighbor) [21]. The similarity degree can be computed in various manners: The first three similarities functions

Table 3. Selected fusion operators Operators type T-conorms

Operators Reason MAP (our application) max the smallest T-conorm 25.54% min(1,x+y) saturation as soon as (x+y)>1 25.59% T-norms min the biggest T-norm 23.66% max(0,x+y-1) not discriminant 23.60% favour class with highest confidence mean (x+y)/2 24.89% biggest value for decision symmetrical sums σ0 associative 23.54%

Table 4. Similarity measures Measure

Purpose

Cosine

VSM

Similarity operator P (ωq,t ∗ωd,t ) simcosine (q, d) = qP t∈q∩d P t∈q

Jaccard

VSM P t∈q

Dice

VSM

sim P Jaccard (q, d) = (ω ∗ω ) Pt∈q∩d 2 q,tPd,t

2 + ωq,t

t∈d

ωd,t −

t∈q∩d

t∈d

2 ωd,t

ωq,t ∗ωd,t

sim P Dice (q, d) = (ωq,t ∗ωd,t ) P t∈q∩d P 2 2 2×

t∈q

Co-occurrence VSM Minkovski Histogram Intersection

2 × ωq,t

ωq,t + ω t∈d d,t −α(Dist(x,y)−1)

distance-based D(x, y) = e simV SM (P DS1 (ω1 ); P DS2 (ω2 )) = cos(V1 , V2 ) VSM P distance-based simM inkovski (I, J) = ( i |fi (I) − fi (J)|p )1/p PN min(fi (I),fi (J)) distance i=1P simHistogram (I, J) = N ( fi (J)) resolution i=1 p simQuadratic (I, J) = (FI − FJ )T A(FI − FJ ) where A is a similarity matrix and Fi Quadratic Form distance and Fj are vectors that list all the pentries simM ah (I, J) = (FI − FJ )T C −1 (FI − FJ ) Mahalanobis distance where C is the covariance matrix of feature vectors P (I) simKL (I, J) = i fi (I) log( ffii(J) ) Kullback-Leibler distance P simCN (λq , λd ) = t∈q∩d Amax(λq,t , λd,t ) Fuzzy Similarity Measure distance where A is the similarity matrix of feature vectors

(Table 4) are representative for text documents, since based on tf/idf. After the fusion, the indexes meaning has evolved. The co-occurrence is interesting to use since it includes the distance factor also, placing the data according to the distance between co-occurring items. The VSM function applies to pseudo-documents. In our case, the fused image and text information could be considered as a kind of pseudo-document, but the vector must contain all the possible combinations of items. We consider also the distance-based similarities, used for image similarity, where the Quadratic form and the Mahalanobis distance take into account the similar pairs of bins. We call the last function presented in the table 4, Fuzzy Similarity Method (FSM), a mixed similarity function based on the fuzzy operators that takes into account the similarity matrix as well. We introduce this function as result of a mix between min and + fuzzy operators, the first synthesize the information from common concepts from the query and medical cases, since the second one gathers the results of the first one. In our study, we apply a selection of similarities measures represented in the Table 5.

Table 5. Similarities Measures used Similarity measure Cosine document retrieval Dice document retrieval

Formula P (ωq,t ∗ωd,t ) simcosine (q, d) = qP t∈q∩d P t∈q

2 × ωq,t

t∈d

MAP (our application)

2 ωd,t

sim P Dice (q, d) = (ωq,t ∗ωd,t ) P t∈q∩d P 2 2 2×

t∈q

ωq,t +

t∈d

17.17%

ωd,t

VSM simV SM (P DS1 (ω1 ); P DS2 (ω2 )) = cos(V1 , V2 ) pseudo-documents P Minkovski-Form simM inkovski (I, J) = ( i |fi (I) − fi (J)|p )1/p image retrieval P simCN (λq , λd ) = t∈q∩d Amax(λq,t , λd,t ) FSM takes into account where A is the similarity matrix of the similar items feature vectors

6

18.53%

15.59% 0.8% 25.32%

Results on CLEF medical image retrieval benchmark

We apply our approach on the medical image collection of CLEF 9 Cross Language Image Retrieval track. This database contanins four public datasets (CASImage, MIR, PathoPic, PEIR 10 ) containing 50000 medical images with the associated medical report in three different languages. In the pre-processing phase we have conducted a comparative study on the ω (spatial localization fuzzy weight) and ϕ (data test based feedback or relevance feedback)parameters, using small variations around a theoretically suitable structure, composed by the sum fusion operator (simple, commutative, associative and balanced technique) and the Fuzzy Similarity Function (FSM) for the similarity. The results using the probabilistic and fuzzy fusion techniques are presented in the last column of the Table 3. The evidence-based techniques were not used according to our previous analysis, this category being more appropriate for the decision fusion level. From our experiment, we can deduct that the T-norms perform the best results on CLEF 2006 queries/database. Retrieval process apply the Cosine, Dice, Minkovsky, VSM and the FSM. The results so obtained are presented in the last column of the Table 5. We remark that the best approach is given by the FSM function, a mixture between classic fuzzy logic operators. Comparative results with other mixed automatic CBMIR approaches recorded in CLEF 2006 medical image retrieval track, are given in table6 . Without using any filtering, our method is situated at the second place in this hierarchy, after a mixed approach using a multiple filtering technique. Considering that the tests on the automatic text retrieval are around 22,55 % of MAP and the automatic image retrieval around 6,41 % of MAP for the same indexes, the fusion applied here seems to be effective since it gives a result near to the sum of those elementary results. Table 6. Results on CLEF 2006 Run MAP [%] R-prec [%] IPAL-IPAL Cpt Im.eval 30.95 34.59 OUR SYSTEM 25.59 29.65 UB-UBmedVT2.eval 20.27 22.25 RWTHi6-EnFrGePatches 16.96 20.78

7

Conclusion

Some characteristics shown by Dubois and Prade [7] on the multimedia data features are represented by lack of standard structure, heterogeneity of formats, self describing information, inadequacy of textual descriptions, multiplicity 9 10

CLEF - Cross Language Evaluation Forum - www.clef − campaign.org/ www.casimage.com, gamma.wustl.edu/home.html, alf 3.urz.unibas.ch/pathopic/intro.html, peir.path.uab.edu

of data types and source database, spatial and temporal characteristics. In consequence, all fusions methods still remain strongly related to the accuracy and the structure of the initial input data. Nevertheless, the richness of the UMLS metathesaurus and the results on CLEF medical image retrieval benchmark, open interesting perspectives for futures developments related to semantic clustering and query expansion, in order to improve the robustness, the efficiency and the effectiveness of medical image retrieval systems. Improving the off-line and on-line computation time, especially for the medical image indexing, will use the grid computing facilities. Such a grid enabled medical image retrieval system will be studied in the frame of the ONCOMEDIA (ONtology and COntext related MEdical image Distributed Intelligent Access)11 ICT Asia international project, in order to improve the performances of the CBMIR systems.

Acknowledgment This work has been done with the support of ONCO-MEDIA (ONtology and COntext related MEdical image Distributed Intelligent Access) ICT Asia project: URL - http : //www.onco − media.com/. We would like to thank our colleagues from IPAL - Caroline Lacoste, Nicolas Vuillemenot, Le Thi Hoang Diem and Jean-Pierre Chevallet - for providing us the text and images separate indexes used for the experiments.

References 1. H. Abe, H.and MacMahon, R. Engelmann, Q. Li, J. Shiraishi, S. Katsuragawa, M. Aoyama, T. Ishida, K. Ashizawa, C. E. Metz, and K. Doi. Computer-aided diagnosis in chest radiography: Results of large-scale observer tests at the 1996-2001 rsna scientific assemblies. RadioGraphics, 23(1):255–265, 2003. 2. Franklin E.White Alan N.Steinberg, Christopher L.Bowman. Revisions to the jdl data fusion model. Joint Directors of Laboratoires, 1997. 3. Isabelle Bloch. Information combination operators for data fusion: A comparative review with classification. IEEE Transactions on Systems,Man, and Cybernetics- Part A:Sytsems and Humans, 26(1):52 – 67, January 1996. 4. G. Booch. Object-oriented analysis and design with applications. Addison-Wesley Professional; 2 edition, 1994. 5. A. A. T. Bui, R. K. Taira, J. D. N. Dionision, D. R. Aberle, S. El-Saden, and H. Kangarloo. Evidence-based radiology. Academic Radiology, 9(6):662–669, 2002. 6. W.W. Chu, F. C. Alfonso, and K.T. Ricky. Knowledge-based image retrieval with spatial and temporal constructs. IEEE Transactions on Knowledge and Data Engineering, 10:872–888, 1998. 7. Florence Sedes Didier Dubois, Henri Prade. Fuzzy logic techniques in multimedia database querying: A preliminary investigation of the potentials. IEEE Transactions on Knowledge and Data Engineering, 13(3):383–392, May/June 2001. 8. Graham M. E. Eakins, J. P. Content-based image retrieval. Technical Report JTAP-039, JISC Technology Application Program, Newcastle upon Tyne, 2000. 9. Watson M. Harmon P. Understanding UML - The Developers Guide -With a web-based application in Java. Inc. San Francisco, 1998. 10. I. Jacobson, M. Ericson, and A. Jacobson. The Object advantage. Business Process reengineering with Object technology. Addison-Wesley Professional; 1st edition, 1994. 11. C. E. Kahn. Artificial intelligence in radiology: Decision support systems. RadioGraphics, 14:849–861, 1994. 12. Weyman J. Kokar M., Tomasik J. Data vs. decision fusion in the cathegory theory framework. 2001. definition and comparation between the two fusion techniques. 13. P. Korn, N. Sidiropoulos, C. Faloutsos, E. Siegel, and Z. Protopapas. Fast and effective retrieval of medical tumor shapes. IEEE Transactions on Knowledge and Data Engineering, 10(6):889–904, 1998. 14. C. LeBozec, M.-C. Jaulent, E. Zapletal, and P. Degoulet. Unified modeling language and design of a case-based retrieval system in medical imaging. In Proceedings of the Annual Symposium of the American Society for Medical Informatics, Nashville, TN, USA, 1998. 15. T. M. Lehmann, M.O. Gld, C. Thies, B. Fischer, K. Spitzer, D. Keysers, H. Ney, M. Kohnen, H. Schubert, and B. B. Wein. Content-based image retrieval in medical application. Methods of Information in Medicine, 43(4):354–361, 2004. 16. L.Wald. Definitions and terms of reference in data fusion. International Archive of Photogammetry and Remote Sensing, 32, June 1999. 17. J. Rumbaugh, M. Blaha, W. Premerlani, and W. Lorensen. Object-oriented modeling and design. Prentice Hall, 1991. 18. C.-R. Shyu, C. E. Brodley, A. C. Kak, A. Kosaka, A. M. Aisen, and L. S. Broderick. ASSERT: A physician-in-the-loop content-based retrieval system for HRCT image databases. Computer Vision and Image Understanding, 75:111–132, 1999. Special issue on content-based access for image and video libraries. 19. A. W. M. Smeulders, M. Worring, S. Santini, A. Gupta, and R. Jain. Content-based image retrieval at the end of the early years. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22:1349–1380, 2000. 20. Sun Shou-Qian Li Zhong-Yang Tang Yong-Chuan, Pan Zhi-Gheng. Probability measures compatible with the fuzzy dempster - shafer belief structure. Proceedings of the International Conference on Machine Learning and Cybernetics, pages 2705–2709, november 2003. 21. Jinaguo Xiao Xiaojun Wan, Jianwu Yang. Document similarity search based on mainflod-ranking of texttiles. Information Retrieval Technology, pages 14–25, October 2006. similarities measures for text files based on tf/idf. 11

ONCO-MEDIA includes medical and academic partners from Singapore, France, Switzerland, Japan, Philippine, China and Taiwan. URL - http : //www.onco − media.com/