J-Net System: a new paradigm for Artificial Neural Networks applied to diagnostic imaging Enzo Grossi
Massimo Buscema
Bracco SpA Medical Department
Semeion Research Center Of Science of Communication Via Sersale 117, 00128 Rome, Italy
[email protected]
Via E. Folli, 50, 20134 Milan, Italy
[email protected] Abstract - In this paper we present a new unsupervised artificial adaptive system, able to extract features of interest in digital imaging, to reduce image noise maintaining the spatial resolution of high contrast structures and the expression of hidden morphological features. The new system, named J-Net, belongs to the family of ACM systems developed by Semeion Research Institute. J-Net is able to isolate in an almost geological way different brightness layers in the same image. These layers seem to be invisible to the human eye and for the other mathematical imaging system. This ability of the J-Net can have important medical applications. Two examples of application are reported: the first in digital subtraction angiography for arterial stenosis diagnosis and the second in Multi-slice CT for lung cancer early detection and evolution prediction. Keywords: Image processing, Artificial Neural Networks, Active Connections Matrixes
I. INTRODUCTION Significant progress in the development of machine vision and image processing technology has been made in the past few years in the medical field in conjunction with improvements in computer technology [1-9]. With the introduction of multi-slice spiral CT scanners, the number of images of body organs, like the lung for example, is steadily increasing and it is critical to develop fast, accurate algorithms that require minimal to no human interaction to identify emergent features of interest. Artificial neural networks (ANNs) can overcome some of these difficulties by interpreting images quickly and effectively. ANNs are composed of numerous processing elements (PEs) arranged in various layers, with interconnections between pairs of PEs [10, 11, 12]. They are designed to emulate the structure of natural neural networks such as those of a human brain. For most ANNs, PEs in each layer are fully connected with PEs in the adjacent layer or layers, but are not connected to other PEs in the same layer. The PEs simulate the function of the neurons in natural neural networks, while the interconnections between them mimic the functions of dendrites and axons. There have been many applications of ANNs reported for the interpretation of images in medicine. The main problems in many image processing applications still are the abundance of features and the difficulty of coping with concomitant variations in position, orientation and scale. This clearly indicates the need for more
978-1-4244-2352-1/08/$25.00 ©2008 IEEE
intelligent, invariant feature extraction and feature selection mechanisms [13]. In a recent review about the role of ANNs in medical decision support with digital imaging the authors concluded that ANNs can play a role in image processing, although it might be a role as a supporting tool rather than a major one [14, 15, 16, 17]. The Active Connection Matrix (ACM) is a new unsupervised artificial adaptive system developed by Semeion Research Institute [18]. The system is able to automatically extract features of interest (e.g. edges, tissue differentiation, etc.) from digital images when activated by original non linear equations. ACM systems copy with the features selection problems in digital imaging: ACM activation allows the reduction of image noise while maintaining the spatial resolution of high contrast structures and the expression of hidden morphological features. The general philosophy and mathematical background of ACM systems are described in the paper of this special session “Four Models for four medical applications” by Massimo Buscema. II. MEDICAL APPLICATIONS OF J-NET, A NEW ACM SYSTEM The aim of the current study is to describe potential applications of J-Net System, a specific system of ACM family, as a support for accurate diagnosis in digital subtraction angiography of popliteal artery and for differentiating between benign and malignant pulmonary nodules identified by MDCT scanners. J-Net considers each image as a Active Connections Matrix, where each node is linked to its nearest neighbours, trough adaptive weights. During the processing phase J-Net modifies its global weights matrix and the state of each node (pixel), according to a specific cost function, considering only the assigned image. J-Net Image Processing enhances possible differences in the burden of the lesion through a deterministic algorithm which extracts morphological (not noisy) features hidden to human eyes. The detailed notions of equations and function of J-Net system is described in the paper of this special session “Four Models for four medical applications” by Massimo Buscema. Before deciding to apply J-Net system to medical images we have assessed its ability to extract the outline of an image in few cycles in different toy models with noisy images,
comparing with standard software available. The results have been surprisingly remarkable [19,20]. IIA FIRST DISEASE.
EXAMPLE OF APPLICATION:
POPLITEAL
ARTERY
The popliteal artery is a relatively short vascular segment but is affected by a unique set of pathologic conditions. These conditions, which may be common throughout the arterial system or exclusive to the popliteal artery, include atherosclerosis, popliteal artery aneurysm, arterial embolus, trauma, popliteal artery entrapment syndrome, and cystic adventitial disease. The clinical manifestations, imaging appearances, and treatment options associated with these pathologic conditions differ significantly. Consequently, the radiologist should be familiar with these conditions to direct imaging for accurate diagnosis and treatment and to prevent loss of limb. The J-Net system has demonstrated its ability to detect internal stenosis which appeared to be invisible to the surgeon even in the case of particularly sophisticated images, such as the digital subtraction angiography:
Figure 1b. First 12 cycles of J-Net;
Figure 1a. Digital subtraction angiography: popliteal artery (Galeazzi Hospital, Milan);
As is visible in figure 1b, J-Net system just after few cycles points out the existence of a stenosis ( red circle), not easily visible in the original image in figure 1a, which was subsequently confirmed by the interventional radiologists at surgical table. The stenosis and the detailed skeleton of arterial structure is clearly visible also in figure 1c which correspond to the final image obtained after convergence.
Figure 1c. J_Net after the convergence.
In the previous J-Net elaborations the values of the initial image were scaled down between -1 and +1; so, the factor α of the Eq (0) was implicitly equal to zero. This factor determines the sensitivity threshold of the system to the brightness of the image. Table I summarizes what has been noted up to this point:
TABLE I: RATIO BETWEEN THE THRESHOLD AND THE SCALE DOWN OF THE UNITS
Threshold α = -1.0 α = -0.9 α = -0.8 α = -0.7 α = -0.6 α = -0.5 α = -0.4 α = -0.3 α = -0.2 α = -0.1 α = 0.0 α = +0.1 α = +0.2 α = +0.3 α = +0.4 α = +0.5 α = +0.6 α = +0.7 α = +0.8 α = +0.9 α = +1.0
Scale down
∈[-2.0,0.0] u ∈[-1.9,+0.1] u ∈[-1.8,+0.2] u ∈[-1.7,+0.3] u ∈[-1.6,+0.4] u ∈[-1.5,+0.5] u ∈[-1.4,+0.6] u ∈[-1.3,+0.7] u ∈[-1.2,+0.8] u ∈[-1.1,+0.9] u ∈[-1.0,+1.0] u ∈[-0.9,+1.1] u ∈[-0.8,+1.2] u ∈[-0.7,+1.3] u ∈[-0.6,+1.4] u ∈[-0.5,+1.5] u ∈[-0.4,+1.6] u ∈[-0.3,+1.7] u ∈[-0.2,+1.8] u ∈[-0.1,+1.9] u ∈[-0.0,+2.0] u
The same image can be elaborated by J-Net with a different threshold α independently. The pictures which were developed with the different threshold α will highlight the final images with different pictures. Different values of α operate on the assigned image through a scanning of the different bright intensities. In cases in which the intensity of brightness in a medical picture is quite proportional to the activity of the pathology in question, it is possible to use different J-Net scanning in order to detect a temporal order of development of the pathology itself. IIB. SECOND
able to distinguish between those cases in which the bright oscillations of the background represent simple noise and those in which oscillations represent a just- traced- model of a picture. The scanning of the picture operated by J-Net through the variation of the threshold α has been demonstrated to be able to point out some kinds of development of lung's cancer one or two years earlier. In order to verify this assumption we have used research and the picture published by a group of researchers in 2000 in the well known scientific journal [26]. We took 2 pairs of lungs cancer images from this research. Each pair shows the cancer as it was when the first CT was done (Time 0) and one or three years later (Time 1). The researchers report that in the Time 0 the picture did not allow for an accurate and differential cancer diagnosis. Two experiments were carried out with this pair of images: the first experiment elaborated the first image (Figure 2a) with different positive values of α . This was designed to increase the sensitivity of J-Net at each elaboration and to discover hidden background patterns. The second experiment elaborated the second image ( fig. 2b)
Figure 2a. Type B adenocarcinoma in 74-year-old man. Initial CT scan shows localized 22 x 22 mm ground-glass opacity in left lower lobe of lung.
EXAMPLE OF APPLICATION: LUNG CANCER
Also in lungs cancers it can be supposed that the different bright intensity in the computerised tomography (CT) reflects the areas where the cancer is more active. In the case of malignant cancer the most peripheral parts can appear to be dark like the background the human eye, while actually they should present “light shadows” of brightness, indicating the explorative and diffusive strategies of the cancer itself. These micro variations of brightness can be so thin that other analysis algorithms could easily classify them as “noise” and erase them. The J-Net system, on the contrary, seems to be
Figure 2b. Type B adenocarcinoma in 74-year-old man. CT scan obtained 349 days after initial CT scan shows an increase in size (to 23 x 26 mm) and the development of vascular convergence. Attenuation at the center of the tumor is slightly increased
Figure 2b.1 α = 0.0
Figure 2b.2 α = -0.1
Figure 2a.4 α = 0.3
Figure 2b.3 α= -0.2
Figure 2b.4 α = -0.3
Figure 2a.6 α = 0.5
Figure 2b.5 α= -0.4
Figure 2b.6 α = -0.5
Figure 2a.1 α = 0.0
Figure 2a.2 α = 0.1
Figure 2a.3 α = 0.2
Figure 2a.5 α = 0.4
Figure 2a.1 documents a very clear photograph of the outline of the cancer which is contained in the source Figure (Fig 2a). The outline of the cancer is increasingly modified from Fig. 2a.2 to Fig 2a.5 apparently without a known reason with respect to the original figure (Fig 2a). It is, however, to be noted that the outline of the Figures 2a.5 and 2 a.6 are very similar to the shape that the cancer will have in the same patient one year later (see Fig 2b). In the second experiment, one year later, only the image of the cancer was developed (Fig 2b). This time, the different JNet developments were carried out decrementing the values of α . This was designed to make J-Net less sensitive to certain brightness intensities at every elaboration, as if we wanted to go back in time.
It can be noted that the outline of the cancer in Figure 2b.1 is similar to the outline of the cancer in Figure 2a.5. The outline of the cancer in Figure 2a.1 can be practically superimposed upon the outline of the cancer in Figure 2b.6 in the same way,. The transformation of the cancer shape at time 0 during the JNet elaboration having α = 0.0 (present time) and having α = 0.4 (possible future) is clearly visible:
Figure 2 a.1: Time 0 – present -
α = 0.0
The J-Net system developed the image at time 0 (Fig 3a) with different kind of positive values of α :
Figure 2 a.5: Time 0 – possible future - α = 0.4
Figure 3a.1 α = 0.0
Figure 3a.2 α = 0.1
Figure 3a.3 α= 0.2
Figure 3a.4 α = 0.3
Figure 3a.5 α= 0.4
Figure 3a.6 α = 0.5
The same experimental procedure was used with a second pair of images from the same patient approximately 3 years later with a different type of cancer:
Figure 3a. Type C adenocarcinoma in a 64-year-old man. The initial CT scan shows a tiny solid nodule (arrow) with minimal adjacent ground-glass opacity (15 x 10 mm) in the right middle lobe.
The elaboration of the cancer shape in Figures 3a.5 and 3a.6 appear to be an accurate prognosis of the spread of the same cancer three years later. The superimposition of the real cancer shape at time 1 (3 years later) upon the two prognoses of the J-Net system, having two independent parameters of α on the image, scanned 3 years earlier (see further) confirms a specific ability of the system. The J-Net system can isolate and highlight informative models existing on the initial image with such thin and specific levels of brightness that they appear as noisy oscillations to other algorithms and are invisible to the human eye.
Figure 3b. Type C adenocarcinoma in a 64-year-old man. This CT scan, obtained 1137 days after the initial CT scan, shows an increase in size (25 x 20 mm) and vascular convergence. The area of solid attenuation is also increased.
Figure 3c = Fig 3b + 3a.5 Prognosis of the cancer shape made by J-Net with α = 0.4
Figure 3d=Fig 3b + 3a.6: Prognosis of the cancer shape made by J-Net with α = 0.5 [The images were manually superimposed without trying to register or to stretch them. An imperfect registration results because of the images' different sizes of the, taken at two different times].
These experiments clearly and visibly document that the J-Net system is able to read some brightness variations into the image at Time 0, which are very difficult to see. These thin brightness variations appear to delineate the cancer’s pattern of spread that will occur at time 1. III. DISCUSSION A.
A medical perspective In principle, cancer cells can spread within the body by different mechanisms, such as direct invasion of surrounding tissues spread via the blood vascular system (hematogenous metastasis) and spread via the lymphatic system (lymphatic metastasis).
The metastatic spread of tumor cells is responsible for the majority of cancer deaths, and with few exceptions, all cancers can metastasize. Clinical findings have long suggested that by providing a pathway for tumor cell dissemination, tumorassociated lymphatics are a key component of metastatic spread. Despite its clinical relevance, surprisingly little is known about the mechanisms leading to the spread via the lymphatics. It is not known, however, whether pre-existing vessels are sufficient to serve this function, or whether tumor cell dissemination requires de novo lymphatic formation (lymphangiogenesis) or an increase in lymphatic size. Lymphangiogenesis has traditionally been overshadowed by the greater emphasis placed on the blood vascular system (angiogenesis). In recent years four separate research groups provide direct evidence that two recently-cloned members of the vascular endothelial growth factor (VEGF) family, VEGF-C and VEGF-D are not only important regulators of lymph vessel growth (lymphangiogenesis) in vivo but also enhance lymphatic metastasis [27,28]. The lymphatic system is important for tissue fluid balance regulation, immune cell trafficking, edema, and cancer metastasis, yet very little is known about the sequence of events that initiate and coordinate lymphangiogenesis. We know very well that angiogenesis occurs in embryonic development, wound healing, and tumor growth through a polypeptide growth factor- related sprouting process; (e.g.. VEGF-A and angiopoietin-2). The primary physiological driving force for blood angiogenesis is oxygen concentration, which is directly correlated with the primary function of the blood vasculature. Indeed, a number of growth factors including VEGF-A and erythropoietin are expressed under the influence of the hypoxia inducing factor-I, an oxygen-sensitive transcription factor. The primary function of the lymphatic system, in contrast to the blood circulation, is to maintain interstitial fluid balance and provide lymphatic clearance of interstitial fluid and macromolecules, thereby sustaining osmotic and hydrostatic gradients from blood capillaries through the interstitium and stimulating convection for interstitial protein transport. In a recent paper a group of authors employing a new model of skin regeneration using a collagen implant in a mouse tail has shown that: 1. interstitial fluid channels form before lymphatic endothelial cell organization and 2. (2) lymphatic cell migration, vascular endothelial growth factor-C expression, and lymphatic capillary network organization are initiated primarily in the direction of the lymphatic flow. These data suggest that interstitial fluid channelling precedes and may even direct lymphangiogenesis (in contrast to blood angiogenesis, in which fluid flow proceeds only after the vessel develops). A novel and robust model is introduced for correlating molecular events with functionality in lymphangiogenesis. In exploring one possible mechanism for these events, the authors observed that in the upstream region of interstitial flow, MMP activity is increased, which could lead to preferential cell migration in the direction of the flow. They also documented that increased expression of the LEC mitogen VEGF-C occurs primarily in upstream regions of the
CDE, which (with subsequent unidirectional transport) could then enhance cell migration and proliferation even further in the direction of the flow. Thus, interstitial flow may represent an important transport mechanism to help guide growth and organization of a developing lymphatic capillary network. The J-Net system's ability to document the neo-formed network of interstitial flow channels preceding the lymphangionenesis surrounding the malignant tumour makes it possible to anticipate very reliably what the tumour mass, which will growth alongside these direction lines, .will be in the future. B. A mathematical perspective The most important features of the J-Net system are the followings: 1. The J-Net isolates a closed picture and background in each image figure on the basis of a specific bright intensity (factor α ) in few cycles. 2. When factor α changes, J-Net selects different figures and background, "as if" it could read the bright intensity as “different kinds of frequencies”. 3. At the end of its evolution J-Net has filled up the inside of the figures or the background (according to which part is the brightest) with waves, which have started to propagate from the isolated figures. The shape of each wave will be homologated with the shape of the figure working as its source. The destructive and constructive differences between these waves create the skeleton of the figures. 4. The J-Net evolves with different values of the parameter α scans different parts of the same image, having different brightness. In the case of lung cancers detected by a CT it seems that J-Net is able to read both the past and future history of the cancer itself, due to the different bright tracks left by the cancer and to those revealing its diffusion patterns. This is possible because J-Net is able to isolate in an almost geological way different brightness layers in the same image. These layers seem to be invisible to the human eye and for the other mathematical imaging system. These features of the J-Net system, which still have to be examined more closely are being presented in an early research stage. J-Net's operation seems to result from some of their mathematical characteristics described in the paper of Buscema in this session. REFERENCES [1] [2] [3] [4] [5] [6] [7] [8] [9]
Davies E., Machine vision: theory, algorithms and practicalities. Academic Press, London, 1990. Gonzalez R and Woods R, Digital image processing, Addison Wesley, Reading,1992. Haralick R and Shapiro L, Computer and robot vision, Vol. 1. Addison-Wesley, Reading, 1992 Horn B, Robot vision, MIT Press, Cambridge,1986. Marr D., Vision. Freeman, San Francisco, 1982. Vernon D (1991), Machine vision. Prentice-Hall, 1992 Boyle R, Thomas R, Computer vision: A first course, Blackwell Scientific Publications, Cambridge, 1988. CANDY- Multilayer CNN Simulator © Analogical and Neural Computing Laboratory, MTA-SzTAKI, Budapest, Hungary, 2003. th Jahne B., Digital Image Processing, (5 revised and extended edition), Springer-Verlag, 2003
[10] [11] [12] [13] [14] [15] [16]
[17]
[18]
[19] [20] [21] [22] [23]
[24]
[25]
[26]
[27] [28]
Haykin, S. Neural Networks. A comprehensive foundation. Macmillan College Publishing Company, Inc, New York, 1994. Kartalopoulos, S.V. Understanding Neural Networks and Fuzzy Logic. Basic Concepts and Applications. New York, NY: the Institute of Electrical and Electronics Engineers, Inc, 1996. Kasabov, N.K., Foundations of Neural Networks, Fuzzy Systems, and Knowledge Engineering., Cambridge, MA: The MIT Press. 1996. Egmont-Petersen, M. De Ridder, D. and Handels, H., "Image processing using neural networks -- a review", Pattern Recognition, vol. 35, no. 10, 2002, pp. 2279-2301. Chua L.O., Roska T., Cellular neural networks and visual computing. Foundations and applications. Cambridge University Press, 2002. Harrer H., Nossek Josef A., “Discrete-time cellular neural networks”, International Journal of circuit theory and application, Vol. 20, no. 5, 1992, pp. 453-467. Harrer H, Nossek Joseph A. “Skeletonisation: a new application for discrete-time cellular neural networks using time-variant templates”, in. Proceedings IEEE International, 10-13 May, San Diego, in Circuits and Systems, 1992,6:2897-2900. Schamschula M P, Johnson J L, Inguva R, "Image Processing with Pulse Coupled Neural Networks”, The Second International Forum on Multimedia and Image Processing, World Automation Congress, Maui, 2000. Buscema, P.M. Sistemi ACM e imaging diagnostico. Le immagini mediche come matrici attive di connessioni [ACM Systems and Diagnostic Imaging. Medical Images as Active Connections Matrices, in Italian], Springer-Verlag Italy, 2006. Buscema M ACM: Active Connection Matrix, v. 10.0, Semeion Software #30, (2003-2006), Roma. Buscema M (2007), ACM Batch, v. 2.0, Semeion Software #33, Roma Hansen T. and Neumann H. “A simple cell model with dominating opponent inhibition for a robust image processing”, in Neural Networks 17, 2004, pp-647–662. MATLAB, The Language of Technical Computing, ver 7.1, MathWorks Inc., (1984-2005). Amendolia S.R., Bisogni M.G., Bottigli U., Ceccopieri A., Delogu P., Fantacci M.E., Marchi A., Marzulli V.M., Palmiero M., Stumbo S., “The Calma Project: a CAD tool in breast radiography”, in Nuclear instruments and method in physics research, A460,2001, pp. 107112,. Delogu P., Fantacci M.E., Masala G.L., Oliva P., Retico A., Documents D8.2: Evaluation of new developments for CADe and report; D8.4: Preliminary tests on MammoGrid/CADe and report, in www.mammogrid.vitamib.com, 2004-2005. Fantacci M.E., Bottigli U., Delogu P., Golosio B., Lauria A., Palmiero R., Raso G., Stumbo S., Tangaro S., Search for microcalcification clusters with the Calma CAD station, in SPIE 4684:1301-1310. Aoki T., Nakata H., Watanabe H., Nakamura K., Kasai T., Hashimoto H., Yasumoto K., Kido M., “Evolution of peripheral lung adenocarcinomas : CT findings correlated with histology and tumor doubling time”, in American journal of roentgenology, o vol. 174, n 3, 2000, pp. 763-768. Plate, K. H., From angiogenesis to lymphangiogenesis, Nature Medicine: 7,2001, pp. 151-152. Boardman K.C., Swartz M.A, Interstitial flow as a guide for lymphangiogenesis. Circulation Research. April 18,; 92(7): 2003, pp. 801-8.
