Cross-Cultural Diagnostic Applicability of the Pictogram Test

This article was downloaded by:[Leontieva, Luba] On: 25 February 2008 Access Details: [subscription number 790913618] Publisher: Routledge Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

Journal of Personality Assessment Publication details, including instructions for authors and subscription information: http://www.informaworld.com/smpp/title~content=t775653663

Cross-Cultural Diagnostic Applicability of the Pictogram Test Luba Leontieva a; Julia Rostova b; Roy Tunick c; Sergey Golovko d; John Harkulich e; Robert Ploutz-Snyder f a Department of Psychiatry, State University of New York, b The Third Psychiatric Hospital, St. Petersburg, Russia c Department of Counseling Psychology, West Virginia University, d Regional Medical Diagnostic Center "Behterev,", St. Petersburg, Russia e St. Joseph's Hospital Outpatient Mental Health Center, Syracuse, New York f Center for Outcomes Research & Evaluation, State University of New York,

Online Publication Date: 01 March 2008 To cite this Article: Leontieva, Luba, Rostova, Julia, Tunick, Roy, Golovko, Sergey, Harkulich, John and Ploutz-Snyder, Robert (2008) 'Cross-Cultural Diagnostic Applicability of the Pictogram Test', Journal of Personality Assessment, 90:2, 165 - 174 To link to this article: DOI: 10.1080/00223890701845286 URL: http://dx.doi.org/10.1080/00223890701845286

PLEASE SCROLL DOWN FOR ARTICLE Full terms and conditions of use: http://www.informaworld.com/terms-and-conditions-of-access.pdf This article maybe used for research, teaching and private study purposes. Any substantial or systematic reproduction, re-distribution, re-selling, loan or sub-licensing, systematic supply or distribution in any form to anyone is expressly forbidden. The publisher does not give any warranty express or implied or make any representation that the contents will be complete or accurate or up to date. The accuracy of any instructions, formulae and drug doses should be independently verified with primary sources. The publisher shall not be liable for any loss, actions, claims, proceedings, demand or costs or damages whatsoever or howsoever caused arising directly or indirectly in connection with or arising out of the use of this material.

Downloaded By: [Leontieva, Luba] At: 19:56 25 February 2008

Journal of Personality Assessment, 90(2), 165–174, 2008 C Taylor & Francis Group, LLC Copyright
ISSN: 0022-3891 print / 1532-7752 online DOI: 10.1080/00223890701845286

Cross-Cultural Diagnostic Applicability of the Pictogram Test LUBA LEONTIEVA,1 JULIA ROSTOVA,2 ROY TUNICK,3 SERGEY GOLOVKO,4 JOHN HARKULICH,5 AND ROBERT PLOUTZ-SNYDER6 1

Department of Psychiatry, State University of New York The Third Psychiatric Hospital, St. Petersburg, Russia 3 Department of Counseling Psychology, West Virginia University 4 Regional Medical Diagnostic Center “Behterev,” St. Petersburg, Russia 5 St. Joseph’s Hospital Outpatient Mental Health Center, Syracuse, New York 6 Center for Outcomes Research & Evaluation, State University of New York 2

The aim of this study was to determine whether the Pictogram Test (PT; Vygostsky, 1960) a test of logical memory that was developed in Russia, is applicable for diagnosing and understanding thought disturbances in schizophrenia within an English-speaking population. Participants included 62 patients with schizophrenia or schizoaffective disorders and 80 community nonpatients from Russia and the United States. We used correlational analysis to organize PT variables into Concrete (CI), Attribute (AI), and Geometric (GI) indexes. Discriminant function analysis revealed that only AI and GI had significant discriminatory power. These indexes correctly classified 91% of English-speaking and 86% of Russian-speaking participants as either patients or nonpatients. Controlling for age and education, multivariate analysis revealed that patients had significantly lower AI and significantly higher GI scores relative to nonpatients, and those differences were similar across languages. These results indicate that the PT can discriminate between patients with schizophrenia or schizoaffective disorders and nonpatients, which suggests that characteristics of the PT could be used to understand the mechanism of logical thinking in patients and nonpatients.

Thought distortions and impaired patterns of association have been hallmarks of schizophrenia since the disorder was first defined by Bleuler (1911/1950). Bleuler believed that the confusion found in schizophrenic patients was a manifestation of extreme association disorder. Arieti (1955) called schizophrenic thinking paleologic (primitive) as opposed to Aristotelian logic, which uses secondary process cognition. Arieti considered paleologic thinking to be the precursor of delusions. Vygotsky (1934) identified the roots of schizophrenic thinking in flaws of abstract logic. Goldstein (1959) later developed the idea that patients with schizophrenia cannot form abstract concepts; their thinking becomes maladaptively concrete. Distortions of association and logical thinking have been described as the major characteristics of schizophrenia (Hemsley, 1992; Watson, 1973; Whitaker, 1973). Impairment of logical memory has been found in people who later developed schizophrenia (Brewer et al., 2005). Research has shown that verbal memory dysfunction occurs early in the course of schizophrenia (e.g., Hill, Beers, Kmiec, Keshavan, & Sweeney, 2004). Logical memorization is a part of the ideational component of the thinking process. The ideational component of thinking addresses how the translations of inputs from the environment become conceptualized and used (Exner, 1991). Conceptualization involves the meaningful organization of information, which constitutes the core of psychological activity. From these conceptualizations, decisions and deliberate behaviors evolve. Difficulty in organizing information could be due to deficits in associative memory. When such deficit exists, incoming information cannot be efficiently retained, sorted, and used Received March 21, 2005; Revised February 8, 2007. Address correspondence to Luba Leontieva, State University of New York, Upstate Medical University, Department of Psychiatry, 750 East Adams Street, Syracuse, NY 13210; Email: [email protected]

to form cognitive concepts and make decisions. Faulty concepts may develop such as in the case of delusional ideas. Taking into account that logical memory impairment is one of the hallmarks of schizophrenia thinking disturbances and that it occurs earlier in the course of the disorder, understanding how faulty logic develops may lead to treatment strategies for overcoming its often devastating consequences. Instruments that are sensitive to how people develop associations, logical connections, and meanings are needed to increase understanding of thought distortion in schizophrenia. The Pictogram Test (PT; Vygostsky, 1960) delves into the development of associations during mediated memorization and consequently, into the organization of logical memory. This test had not been translated into English nor used with Englishspeaking population, although it is commonly used as part of psychological assessments in Russia, some Western European countries, and Cuba. Finding cross-cultural similarities could open the way for the use of this instrument with the Englishspeaking population of the United States. The cross-cultural adaptation of the PT will broaden the knowledge of tests that investigate mediated logical memorization and thinking process. The PT was first developed in the 1920s as a technique of mediated logical memorization and was derived from Vygotsky’s picture sorting technique (as cited in Leont’ev, 1928). The test involves mediated memorization of 16 abstract words and phrases (targets) for 1 hr. Each target is announced by the examiner one at a time; after each target is announced, examinees are asked to draw small pictures (pictograms) that will help them recall the targets 1 hr later. There is no time limit for drawing, and artistic quality is not judged. After 1 hr, examinees receive their drawings back and are asked to recall targets in any order while looking at their drawings. They are asked to explain how each picture helped to memorize the target. Interpretation involves analyzing drawings and explanations of their selections

165

Downloaded By: [Leontieva, Luba] At: 19:56 25 February 2008

166 as well as the percentage of correctly recalled words. The rationale behind the PT is that logical memorization through the use of drawings can have meaningful connections with the targets to be memorized. Examinee choice of drawings and the explanation why they chose them reveals associations that develop between the meaning of the target and the picture drawn for it. Birenbaum (1934) was the first who sought to identify the PT patterns among diagnostic groups of patients with schizophrenia, organic brain damage, affective disorders, and neuroses. Birenbaum noticed that patients with schizophrenia produced either very concrete drawings with many unnecessary details that distracted them from recall later or overly abstract, schematic drawings without any connection to the target. Other diagnostic groups did not have such qualities in their pictograms. Rubinstein (1970) outlined common features in the PT of patients with schizophrenia: meaningless drawings (i.e., geometrical shapes) and cumbersome drawings with many irrelevant details that distract from target recall. Longinova (1972) conducted a descriptive study of 160 patients with various forms of schizophrenia and normal controls. Patients with schizophrenia had a smaller percentage of adequate drawings compared to nonpatients. Later, normative data (150 patients with schizophrenia, other diagnostic groups, and over 400 nonpatients) was collected for the PT and a manualized approach to scoring and interpretation was developed (Khersonsky, 1988, 2000). Khersonsky (1988, 2000) has reported that patients with schizophrenia formed very peculiar connections between target words and drawings. Their associations were frequently based on subordinate meanings of targets. They incorporated too much of the target, were too cumbersome or concrete for effective memorization, or drew pictures that had no clear connection to the target (Khersonsky, 1979, 1981, 1988). Studies of the PT conducted in other cultures have shown similar results for patients with schizophrenia. Perez-Ramirez, Miqueles-Estevez, and Hernandez-Melendez (1985) found in their Cuban sample that mediated memory is the most disturbed in patients with schizophrenia compared to neurotic and demented patients. Patients with schizophrenia used overrationalization, bizarre associations, could not explain their choice of drawings, and had poor recall. Another Cuban study investigated PT of 21 patients with paranoid schizophrenia and found that patients had either very concrete or too abstract, geometrical drawings with no relation to targets (Calvo-Montalvo & Villar-Diaz, 1988). One recent study (Klasik, 2005) conducted in Poland examined the PT of 30 patients with schizophrenia and 30 healthy controls and found that patients were not able to keep the limits of an abstract concept of targets while fixating on irrelevant information that was not important for memorization. Thus, predominately descriptive research that is available on the PT has demonstrated that patients with schizophrenia have characteristics (i.e., overly concrete drawings with subordinate meanings, overly abstract drawings with no meanings, bizarre associations, and poor recall) of their pictograms that discriminate them from nonpatients or other diagnostic groups. Few psychological tests examine associative or logical memory. Logical Memory Tests I and II from the Wechsler Memory Scale (as cited in Lezak, 1995) concentrate on logical memorization but do not analyze how participants memorize the information and build their logical connections. The examinee produces an end result in the form of a memorized story, but the test does not indicate why the story was memorized as it was or

LEONTIEVA ET AL. what helped in the memorization process. The Whitaker Index of Schizophrenia Thinking (Whitaker, 1973) examines logical connections in thinking but again does not address the specific associations of the test taker. The PT requires that examinees connect the material to be remembered with an external representation (the drawing), thereby developing meaningful associations for memorization. The examination of these associations reveals the logic underlying an examinee’s thinking process. Patients with schizophrenia did not make effective use of their associative or logical memory for memorization. They became distracted by their associations, and consequently, their memorization was less effective. The analysis of associations made by patients with schizophrenia will help researchers and clinicians identify specific distortions in the thinking pattern and process of these patients. Detection of these distortions of thinking will aid in diagnosing schizophrenia and can also provide a record of patients’ thinking during the course of treatment. The aim of this study was to determine whether the PT could be used to differentiate participants with diagnoses of schizophrenia or schizoaffective disorder from nonpatients cross-culturally. All prior studies on the PT have utilized descriptive approaches to the data rather than the more sophisticated statistical procedures we employed in this study. Moreover, there are no formal validity and reliability studies on the PT, so this study initiated an investigation of the psychometric properties of the PT.

METHOD A total of 142 participants were involved in the study; 72 were native Russian speakers in Russia and 70 were native English speakers in the United States. Sociodemographic and clinical characteristics of study participants are summarized in Table 1.

Patients Groups The diagnosis of schizophrenia in the patients’ chart was the criterion for inclusion. We did not include patients who were too unstable or disorganized to complete the assessment. Russian-speaking patients. Study staff recruited 31 participants diagnosed with schizophrenia or schizoaffective disorder from their inpatient units at a state psychiatric hospital. Diagnoses of schizophrenia and schizoaffective disorder were confirmed by chart reviews that included results of a comprehensive psychological evaluation. Chart diagnoses were recorded using the International Classification of Diseases, 10th Revision (World Health Organization, 1994) and were made by the hospital’s staff psychiatrists. Psychological evaluations (i.e., Minnesota Multiphasic Personality Inventory; as cited in Berezin, Miroshnikov, & Rozhanets, 1976; Rorschach Inkblot test; as cited in Besbalko, 1978) recorded in patient charts were made by the hospital’s staff psychologists. English-speaking patients. 31 participants diagnosed with schizophrenia or schizoaffective disorders were recruited from the outpatient clinic of a large medical school and inpatient units at a state psychiatric hospital. Seven participants were outpatients (23%) and 24 were inpatients (77%). Outpatients were recruited during their visits to the medication clinic. Inpatients were approached on the units. Diagnoses of schizophrenia and schizoaffective disorder were confirmed by review of patient

Downloaded By: [Leontieva, Luba] At: 19:56 25 February 2008

THE PICTOGRAM TEST

167 TABLE 1.—Sociodemographic description of study participants. Nonpatients

Age, M (SD) [min, max] Sex, M (SD) Male Female Education, M (SD) No. years [min, max] Nationality/ethnicity, M (SD) Russian Othere White African American Students, M (SD) Schizophrenia type, M (SD) Paranoid Simple/undifferentiated Catatonic Schizoaffective Duration of illness, M (SD) [min, max] Number of hospitalizations, M (SD) [min, max] History of alcohol abuse/dependence, M (SD)

Patients

Russian Speakinga

English Speakingb

Russian Speakingc

English Speakingd

27 (9) [19,49]

28 (6) [21,54]

40 (8) [22, 52]

40 (10) [19,56]

21 (51) 20 (49)

18 (46) 21 (54)

14 (45) 17 (55)

21 (68) 10 (32)

15 (2) [11,17]

18 (2) [15,23]

13 (2) [9, 16]

13 (2) [7, 19]

35 (85) 6 (15) 16 (39)

9 (26) 22 (69) 2 (5) 33 (85)

29 (94) 2 (6) 27 (87) 4 (13) 27 (87) 3 (10) 1 (3) 1 (3) 15 (9) [1, 29] 11 (6) [1, 18] 6 (20)

21(68) 3 (10) 6 (19) 14 (9) [3, 38] 8 (7) [1, 26] 11 (36)

Note. N = 142. n = 41. b n = 39. c n = 31. d n = 31. e Ukrainian, Byelorussian, Armenian, Korean, Tatar, Bashkir (for Russian-speaking participants, primary language was Russian); Indian, Turkish, Pakistani, Brazilian, Venezuelan (for English-speaking participants, primary language was English). a

charts, which included psychiatric evaluations made by staff psychiatrists according to Diagnostic and Statistical Manual of Mental Disorders (4th ed. [DSM–IV]; American Psychiatric Association, 1994) criteria. Diagnoses for 10 patients had been confirmed using the Structured Clinical Interview for DSM–IV Axis I Disorders (SCID–I; First, Spitzer, Gibbon, & Williams, 1997), whereas diagnoses for 8 patients had been confirmed by the Schedules for Clinical Assessment in Neuropsychiatry (World Health Organization, 1999) conducted by psychiatrists. Russian and English-speaking patients were not significantly different in age, F (1, 61) = 0.00, p > .05, Cohen’s d = 0.00; years of education, F (1, 61) = .97, p > .05, Cohen’s d = 0.13; duration of schizophrenia, F (1, 61) = 0.20, p > .05, Cohen’s d = 0.06; and number of hospitalizations, F (1, 61) = 3.75, p > .05, Cohen’s d = 0.24. There were more patients with schizoaffective diagnosis and past history of alcohol abuse/dependence among English speakers than in the Russian-speaking group (19% vs. 3% and 36% vs. 20 %, respectively).

Nonpatients Groups Russian-speaking nonpatients. A total of 41 participants were university students and hospital workers who responded to advertisements posted by the study investigator. English-speaking nonpatients. A total of 39 participants were university and medical school students recruited through advertisements and personal contacts. Sociodemographic characteristics of both Russian and English speaking nonpatients are presented in Table 1. Russianand English-speaking nonpatients were not significantly differ-

ent in age, F (1, 79) = 0.13, p > .05, Cohen’s d = 0.04. Englishspeaking nonpatients had more years of education than Russianspeaking nonpatients, F (1, 79) = 87.99, p < .05, Cohen’s d = 0.73. Students made up a larger percentage of English-speaking nonpatients than Russian-speaking nonpatients (85% vs. 39%; p < .05, Fisher’s exact test). Patients and nonpatients in both language groups were significantly different in age, F (1, 141) = 57.86, p < .05, Cohen’s d = 0.54 and years of education, F (1, 141) = 68.13, p < .05, Cohen’s d = 0.57. The mean age of patients was 39 ± 9 years versus 28 ± 8 years for nonpatients. Mean years of education for patients was 13 ± 2 years versus 16 ± 3 years of education for nonpatients.

Procedure All participants provided informed consent. Tests were administered in each participant’s native language. The Beck Depression Inventory–II (BDI–II; Beck, Steer, & Brown, 1996) was administered first followed by the PT drawing phase. After the PT drawing phase, participants began completion of the Minnesota Multiphasic Personality Inventory–2 (MMPI– 2; Hathaway & McKinley, 1991). After 1 hr of the MMPI–2, the test was interrupted, if necessary, and the retrieval phase of the PT was conducted. Participants then finished the MMPI–2. Assessment sessions lasted from 11/4 hr to several hours; typical sessions lasted about 2 hr. L. Leontieva and J. Rostova were involved in data collection; both had training in administration of the measures and were experienced in working with patients with schizophrenia. All assessments were conducted in private rooms, free from distractions and noise. All participants were

Downloaded By: [Leontieva, Luba] At: 19:56 25 February 2008

168 debriefed; those requesting their test results were provided with follow-up sessions. Nonpatients were offered a chance to win $25 as an incentive for their participation in the study. Participants were told that if they provided sincere responses to the MMPI–2, their names would be included in a drawing for one of four $25 prizes. Patients were not offered monetary incentives for their participation in the study. They were asked to help with the study, and it was explained to them that the information gained could be helpful for other patients in the future.

Measures The BDI–II, MMPI–2, and the PT were the measures in the study. In this article, we present only the results of the PT. The PT. The PT (Khersonsky, 1981, 1988, 2000) was translated from Russian into English by three independent translators. All three translators had philology degrees specializing in Russian language and were fluent in Russian and English. When items were translated as different synonyms, the best suited synonym was selected and the others were provided in parenthesis for clarification in case the word closest in meaning was not understood. The English translation of the test was translated back into Russian by a translator blind to the original Russian version. The back translator held a philology degree and was fluent in Russian and English. The back translation was a close approximation to the original Russian version of the PT, as some Russian words cannot be translated directly to English. Some examples of target words are: happy holiday, illness, and friendship. Please contact L. Leontieva for more information on the PT. Standardized Instructions for the Pictogram Test Examiners followed standardized instructions for the PT and recorded responses verbatim. In each case, the examiner says I am going to check how you memorize words and phrases. I will say words and short phrases that you are to memorize. After I say each word, you are to draw something that can help you to memorize that word. This is not a drawing test; so do not worry about how to draw. Just draw something that will help remind you about this word after one hour. You cannot write words or letters.

If the examinee asks, “Can I put numbers?,” say, “It’s up to you.” If the examinee asks what to draw, say, “Draw whatever you want that will help you to memorize the word.” During the retrieval, examinees are asked to give a brief explanation about why they had constructed the drawings for each target. Thus, the drawings and explanations of how they related to the targets were considered in scoring.

Scoring the PT The PT scoring and variables description is presented in Table 2. We followed the format outlined by Khersonsky (1981, 1988, 2000) in organizing variables into three groups: major, special features, and graphical features. The major variables deal with what was drawn, its level of abstraction, its meaning to the examinee, the originality and adequacy of the drawing, and how well the target was recalled. Special features variables included specific characteristics of the drawings such as repetitions of the same drawing, associations by sound, and so forth. Graphical features variables included position of drawings, characteristics

LEONTIEVA ET AL. of their size, and pencil pressure. Following is an example of scoring of the PT response according to the major variables. For the target phrase “happy holiday,” the participant drew fireworks, explaining that fireworks frequently happened during holidays. This drawing was scored attribute selection, formal, items, standard, and adequate (AS, F, I, St, A). If a person correctly retrieved the word after 1 hr, a score of 1 was given to the drawing; for synonym retrieval a score of 0.5 was given. All 16 drawings were scored in such manner. Special and graphical features applied to the whole PT protocol (either present or absent). After all 16 drawings were scored; their scores on major variables were summarized.

Data Analysis We employed a number of different statistical procedures to explore the PT variables in this study. The unit of analysis was the PT protocols, which represented the summary scores for the PT variables (i.e., all concrete variables in the protocol were summed up, all attribute selection variables were summed up, etc). First, we examined the correlation matrix with the PT major variables. We combined variables that had significant positive correlations and related to each other in a meaningful way (i.e., human scenes and concrete) into three indexes (Concrete index [CI], Attribute index [AI], and Geometric index [GI]) as described in the Results section. We performed a discriminant function analysis (DFA) to investigate how the three indexes discriminate among the classification groups (patients vs. nonpatients). After identifying discriminatory indexes, we submitted their scores to a multivariate analysis of covariance (MANCOVA) to compare scores of patients versus nonpatients and Russian- versus English-speaking participants. Cohen’s effect sizes d and phi coefficients are reported in conjunction with significance p values. We performed Fisher’s exact test to compare how patients and nonpatients were different on the special and graphical features of the PT. All analyses utilized a two-tailed alpha of .05. RESULTS Understanding of the target words. Russian-speaking participants understood all of the target words used in the PT. However, there was some misunderstanding of target words among English-speaking participants. In the English-speaking nonpatients group, two participants did not know the meaning of the word animosity, and three participants were unsure of the meaning of the word deception. In the English-speaking patients group, 15 participants were unsure about the meaning of animosity, and four participants were unsure about the word deception. Unknown words were clarified during the administration of the PT to English-speaking participants. If the clarifications used were correctly retrieved during the retrieval phase, they received the same credit as the original target word. Reliability of the PT. PT data for all participants was scored by L. Leontieva. An independent, qualified rater who was blind to participant group assignments also scored 20 randomly selected PT protocols (5 from each study group). L. Leontieva and S. Golovko each had training in PT scoring protocols, expertise with the test manual (Khersonsky, 2000), experience in PT administration and scoring, and were proficient in Russian and English. We selected the intraclass correlation coefficients

Downloaded By: [Leontieva, Luba] At: 19:56 25 February 2008

THE PICTOGRAM TEST

169 TABLE 2.—Pictogram Test (PT) scoring and variables descriptions.

Level of abstraction (single score per drawing)

Personal meaning (single score per drawing) Content (multiple scores could be assigned per drawing) Frequencya (single score per drawing) Adequacy (present or absent per drawing) Productivity of memorization Special features (present or absent per drawing)

Graphical features (present or absent in the entire PT protocol)

Concrete—Assigned for scenes or photographical drawings Attribute selection—Assigned when the drawing was selected according to the principle of belonging (i.e., drawings of a Christmas tree for “happy holidays”) Metaphorical symbols—Assigned when an abstract word was depicted through drawings that had a connection with the target word via metaphor, proverb, or art (for “Happiness” a crossed out watch, which shows that “when people are happy they do not keep track of time”) Geometric symbols—Assigned for geometrical figures Formal—No personal meaning Personalized P—drawings of participant, scene with participant, or part of participant’s body Personalized S—when drawings had some association with subject (S) Human scenes, humans, human parts, small human parts, items, animals, plants, landscape & nature, art & architecture, geometric forms, and grammatical symbols Standard—Drawings seen in at least 20% of the PT protocols Repetitive—Drawings seen in 4% to 19% of the PT protocols Original—Drawings seen in 3% or less of the PT protocols Assigned when the drawing was suitable for memorization of the target word; adequacy was judged by the criteria that the adequate drawing should be succinct, logically explainable, and not overly abstract Score of 1 for exact retrieval of the target and 0.5 for approximate (synonym) retrieval of the target word Nonformalized drawings including association by sound (e.g., for target word “animosity” participant drew animal + city), multiple drawings (many unrelated drawings for one target word), empty symbols (geometric forms that had no connection to the target word), and undifferentiated drawings (drawings that a participant could not understand or connect to the target word; i.e., scribbles) Shock reactions including refusal to draw and refusal to explain Repetitions including typical stereotypy (repetitive drawing of human figures), atypical stereotypy (repetitive drawings of any figures except humans), and perseverations (repetition of the same drawing for different target word with the same explanation) Other including use of letters (despite instructions not to), the whole system (incorporation of previous drawings into new ones), and spontaneous verbalization Finishing including part only (unfinished drawings), duplication (repetition of same drawing in a given pictogram), change (erasing or blackening the drawing), and return (returning back to and elaborating on previous drawings) Systematization of the drawings on the page, including Numbering, Punctuation, and Compartmentalization Positions of all 16 drawings on the page including horizontal, vertical, on edge, reverse, central, and chaotic Size of the drawings including macrographia, micrographia, enlargement, constriction, local size change, and nonsystematic size Line pressure including weak, pressured, oscillating, interrupted, overlapping, and tremulous

a

Based on frequencies in this sample and Russian norms (Leontieva, 2005; Khersonsky, 2000).

(ICC) for 23 major PT variables as the most appropriate interrater reliability measure because the unit of analysis was the entire PT protocol and not individual drawings. All ICCs were .91 or above, and all were significant (p < .01). Both raters reached 100% agreements in scoring of special and graphical features of the PT. Coefficient alpha for productivity of memorization of the PT was .90, which showed high internal consistency of the measure for memorization. Split-half reliability analysis of productivity of memorization yielded equal-length Spearman–Brown coefficient of .89. Thus, when equally divided by odd/even numbers, two parts of the PT had a high correlation.

variables. Attribute drawings tended to be succinct, impersonal, used by many, and tended to be remembered. The GI consisted of geometric symbols, geometric forms, and original. Unadjusted means and standard deviations for each index are presented in Table 4. When comparing patients versus nonpatients across languages, unadjusted AI and GI were significantly different (p < .05, Cohen’s d = 1.92 and 0.76, respectively). Patients were not different from nonpatients on CI (p > .05, Cohen’s d = 0.09). The three indexes were also statistically significantly related (AI and GI r = –.57; AI and CI r = –.30; GI and CI r = –.35; all ps < .05).

Relationships between the PT variables. Table 3 displays correlations among 18 PT test variables. Five variables (animals, plants, landscape and nature, art and architecture, and grammatical symbols) were not included into the matrix due to their low count. Variables that had significant positive correlations, were conceptually relevant, and had meaningful relationships with each other were summed to create three indexes: CI, AI, and GI. CI consisted of concrete, human scenes, humans, and Person P variables. Concrete drawings tended to be scenes with people or pictures of persons, and they might depict an examinee (Person P). The AI consisted of items, human parts, small human parts; standard, adequate, formal; and productivity of memorization

DFA. To determine how the three indexes discriminate patients from nonpatients, a one-way DFA was performed. In this analysis, we determined the number of participants correctly classified as patients with schizophrenia and nonpatients as a function of each of the indexes and their prediction coefficients. The cutoff values used to determine whether participant was a patient or nonpatient were as follows: .969 for nonpatients and –1.250 for patients. DFA indicated that only AI and GI contributed significantly to discrimination between patients and nonpatients. The AI was the most discriminating index (Wilks’ λ = 0.524, p < .001) and had the highest correlation with the discriminant function (r = .86), accounting for 74% of

170 .22∗ .07 −.29∗ .28∗ .13 .02 .08

.29∗ −.24∗ −.17∗ −.51∗ −.32∗ .36∗ .23∗

.29∗ .05

.38∗ −.04 −.37∗ .41∗ .36∗

−.23∗ .29∗ .15 .64∗ .41∗ −.20∗ −.27∗

−.19∗ −.51∗

−.14 .01 .14 −.19∗ −.22∗

−.05 −.07 .09 −.17∗ −.18∗ −.03 .12

.04

−.53∗ .02

.00

−.24∗

−.83∗

Note. Conceptually relevant correlations are underlined and bold. p < .05 (two-tailed).

∗

Attribute selection Metaphorical symbols Geometric symbols Formal Person P Person S Human scenes Humans Human parts Small human parts Items Geometric forms Standard Repetitive Original Adequacy Productivity of memorization −.20∗ −.06 .25∗ −.26∗ −.13

−.17∗ .93∗

.09 −.14 −.05 −.34∗ −.23∗ −.18∗ .10

.27∗ .06 −.32∗ .26∗ .16

.10 .06

−.61∗ −.95∗ .02 −.10 .14 −.04

−.20∗ −.05 .25∗ −.26 −.18

−.13 −.15

.36∗ .13 .21∗ −.19∗ −.01

−.22∗ −.04 .27∗ −.18∗ −.09

−.04 .00

−.08 .03 −.11 .05

.29∗ .02 −.31∗ .31∗ .27∗

−.05 −.33∗

.11 −.22∗ −.24∗

.05 −.17∗ .04 −.16 −.21∗

−.26∗ −.20∗

−.23∗ −.24∗

.29∗ −.14 −.22∗ .20∗ .16

−.07 −.21∗

−.08

−.08 .13 .00 .02 .05

.10 .16

.14 .05 −.18∗ .29∗ .32∗

−.15 −.19∗ −.09 .26∗ −.25∗ −.12

−.40∗ −.78∗ .70∗ .59∗

−.24∗ .10 .03

−.80∗ −.65∗

.88∗

Small Attribute Metaphorical Geometric Human Human Human Geometric Concrete Selection Symbols Symbols Formal Person P Person S Scenes Humans parts Parts Items Forms Standard Repetitive Original Adequacy

TABLE 3.—Correlations between 18 major Pictogram Test variables.

Downloaded By: [Leontieva, Luba] At: 19:56 25 February 2008

Downloaded By: [Leontieva, Luba] At: 19:56 25 February 2008

THE PICTOGRAM TEST

171

TABLE 4.—Descriptive statistics for unadjusted Attribute Index (AI), Geometric Index (GI), and Concrete Index (CI). Diagnosis Status AI scores

Nonpatients Total Patients

GI scores

Total Nonpatients Total Patients

CI scores

Total Nonpatients Total Patients Total

Language

M

SD

Russian speaking English speaking

71.0 71.8 71.4 54.5 49.1 51.8 4.6 4.9 4.6 8.2 8.6 8.4 12.6 13.5 13.0 12.4 15.0 13.7

7.5 8.2 7.8 11.6 13.5 12.8 3.9 3.6 3.8 7.2 4.8 6.1 7.8 6.9 7.3 6.7 8.1 7.5

Russian speaking English speaking Russian speaking English speaking Russian speaking English speaking Russian speaking English speaking Russian speaking English speaking

Note. English-speaking nonpatients, n = 39; patients, n = 31. Russian-speaking nonpatients, n = 41; patients n = 31.

the shared variance in the difference between patients and nonpatients (correlation with discriminate function squared). The GI, although significant, was less discriminatory than the AI (Wilks’ λ = 0.876, p < .001); it has a lower correlation with the discriminate function (r = –.34), accounting for 12% of shared variance in the difference between patients and nonpatients. The CI was not statistically significant (Wilks’ λ = 0.998, p > .05, r = –.04). Classification statistics determined the accuracy with which the indexes classified participants as patients versus nonpatients. The indexes correctly classified 88% of originally grouped cases as patients or nonpatients. Of the 62 patients with schizophrenia, 51 (82%) were correctly classified as patients by the PT indexes. A total of 11 patients (18%) were misclassified; they had diagnoses of schizophrenia/schizoaffective disorder but were classified by the PT indexes as nonpatients. Of nonpatients, 74 (93%) were correctly classified as nonpatients; 6 (7%) were misclassified, that is, they were not patients but they were classified as patients by the PT indexes. To verify that DFA function scores work equally well in both language groups, we conducted the subgroup analysis comparing the rates and types of misclassifications by language (see Table 5). DFA correctly identified 91% of Americans and 86% of Russians (p = .43) as either patients or nonpatients. Furthermore, in the few occasions when misclassifications were made, the types of missclassifications (false positives or false negatives) were similar in both language groups: 4 (67%) and 6 (60%) of the misclassifications were false negatives in the American and Russian groups, respectively.

MANCOVA. We used this analysis to evaluate the effect of diagnoses and language on AI and GI indexes using age and years of education as covariates. The CI index was not included into this analysis because it did not discriminate significantly in DFA, and there was no significant difference between patients and nonpatients on this index when we compared unadjusted means. In MANCOVA, AI and GI were entered as dependent variables, diagnosis status and language as indepen-

TABLE 5.—Comparisons of diagnostic accuracy from DFA by language.

Distribution of Overall Classifications

English speaking Russian speaking

Incorrect Classification 6 (9%)

Correct Classification 64 (91%)

10 (14%)

62 (86%)

Fisher’s Two-Tailed Exact p Values

Phi Coefficient

.43

−.08

.86

.02

1.00

–.07

Distribution of Correct Classifications

English speaking Russian speaking

Correctly Classified as Patients 27 (42%)

Correct Classified as Nonpatients 37 (58%)

25 (40%)

37 (60%)

Distribution of Incorrect Classifications

English speaking Russian speaking

Incorrectly Incorrectly as Patients 4 (67%)

Incorrect Incorrect as Nonpatients 2 (33%)

6 (60%)

4 (38%)

Note. DFA = discriminant function analysis.

dent variables, and age and education level as covariates to adjust for potential confounding effects of age and education on our comparisons of AI and GI scores. Not surprisingly, the results demonstrate significant effects for indexes, F (1, 136) = 27.05, p < .001, Cohen’s d = 0.89 and diagnosis, F (1, 136) = 24.70, p < .001, Cohen’s d = 0.85. However, these effects were qualified by a significant Diagnosis × Index interaction effect, F (1, 136) = 28.12, p < .001, Cohen’s d = 0.90, suggesting that differences between patients and nonpatients were also dependent on index (see Figure 1). We further compared patients to nonpatients within each index (using Bonferroni corrections for multiple comparisons) and found that patients had significantly higher GI scores (p < .01, Cohen’s d = .54) and significantly lower AI scores (p < .001, Cohen’s d = 1.13) than nonpatients (see Figure 1).

FIGURE 1.—Patients vs. non-patients on Attribute and Geometric indices (Bonferroni coorected for multiple comparisons).

Downloaded By: [Leontieva, Luba] At: 19:56 25 February 2008

172

LEONTIEVA ET AL.

TABLE 6.—Comparisons of patients to nonpatients on special features present in the Pictogram Test. Special Features Association by sound Multiple drawings Empty symbols Undifferentiated drawings Refusal to draw Refusal to explain Typical stereotypy Atypical stereotypy Use of letters The whole system Spontaneous verbalization Drawing part only Duplication Change Return

Nonpatients n (%)a

Patients n (%)b

3 (4%) 0 2 (3%) 0 0 1 (1.3%) 55 (70%) 29 (36%) 5 (6%) 1 (1.3%) 3 (4%) 0 2 (3%) 1 (1.3%) 2 (3%)

1 (1.6%) 3 (5%) 12 (19%) 2 (3.2%) 3 (5%) 37 (60%) 32 (52%) 29 (47%) 12 (19%) 1 (1.6%) 7 (11%) 2 (3.2%) 0 3 (5%) 6 (10%)

Fisher’s Exact Test Phi p Value∗ Coefficient .63 .08 < .01 .19 .08 < .01 .06 .23 .02 1.0 .10 .19 .51 .32 .08

–.06 .17 .28 .14 .17 .66 –.17 .11 .20 .02 .15 .14 –.11 .11 .15

a n = 80. b n = 62. *p < .05 indicated in bold (two-tailed).

The main effect for language and all interaction effects involving language were nonsignificant, suggesting that after adjusting for age and education, AI and GI scores follow a similar pattern for English- and Russian-speaking individuals. Tables 6 shows statistics comparing patients to nonpatients on special features variables of the PT. Patients had significantly more empty symbols, use of letters, and refusal to explain than nonpatients. Table 7 displays statistics comparing patients to nonpatients on graphical features variables of the PT. Patients had significantly more on edge, chaotic, macrographia,

TABLE 7.—Comparisons of patients to nonpatients on graphical features present in the Pictogram Test. Graphical Features Numbering Punctuation Horizontal Vertical On edge Reverse Central Chaotic Macrographia Micrographia Enlargement Nonsystematic size change Weak line pressure Pressured lines Compartmentalization Local size change Oscillating lines Interrupted lines Overlapping lines Tremulous lines Constriction

Patients n (%)b

40 (50%) 1 (1.3%) 44 (55%) 34 (43%) 1 (14%) 0 65 (81%) 2 (3%) 2 (3%) 0 0 0 0 0 11 (14%) 0 0 0 0 0 0

18 (29%) 2 (3%) 38 (61%) 15 (24%) 24 (39%) 1 (2%) 26 (42%) 9 (15%) 8 (13%) 3 (5%) 1 (2%) 1 (2%) 3 (5%) 5 (8%) 5 (8%) 2 (3%) 0 0 0 0 0

n = 80. b n = 62. p < .05 indicated in bold (two-tailed).

a ∗

Nonpatients n (%)a

Fisher’s Exact Test Phi p Value* Coefficient .02 .58 .50 .03 < .01 .44 < .01 .01 .02 .08 .44 .44 .08 .01 .42 .19

–.21 .07 .06 –.19 .29 .10 –.41 .22 .20 .17 .10 .10 .17 .22 –.09 .14

and pressured variables than nonpatients. Nonpatients had significantly more numbering, vertical, and central variables than patients.

DISCUSSION In this study, we were the first to examine the cross-cultural diagnostic applicability of the PT and some of it psychometric properties. The English version of the PT was successfully administered to patients with schizophrenia/schizoaffective disorders and nonpatients in the United States. Only 2 of the 16 PT target words were not in the vocabularies of some Englishspeaking participants. Scoring of the PT was conducted with excellent interrater reliability. Memorization variable on the PT yielded high internal consistency. Correlational analysis of the PT variables suggested reducing the number of major variables into three statistically manageable and conceptually meaningful indexes: the CI, AI, and GI. These indexes reflected participants’ use of intermediate concepts to memorize abstract materials. The CI was reflected in photographic-like drawings of people, usually engaged in some activity, not always helpful in retrieval. The AI was reflected in more concise drawings according to the principle of belonging (i.e., Christmas tree for “happy holidays”) and was helpful in retrieval. The GI was reflected in meaningless drawings and associated with poor recall. The results of DFA indicated that AI was the strongest discriminator of the patient group. This indicates the importance of this index when considering logical thinking disturbance in schizophrenia based on the PT. Our analysis supports the value of building economical and logically connected intermediate concepts for the most productive memorization of abstract materials. Schizophrenia patients lacked this quality in their PTs (low AI). This suggests that logical memorization and appropriate intermediate concept building represented a challenge for these patients and support the previous descriptive findings (Khersonsky, 1979, 1981, 1988; Klasik, 2005). GI was significant in its separation of patients from nonpatients. Patients with schizophrenia tended to use geometric forms as mediators, which did not help them with memorization. Similar to previous studies descriptive findings (CalvoMontalvo & Villar-Diaz, 1988; Khersonsky, 1979, 1988; Rubinstein, 1970), the results of this study support the conclusion that patients did not utilize their associative memory productively. Their associations tended to be imprecise, and they often used meaningless symbols when presented with abstract concepts. AI and GI are most important when using the PT in diagnosing schizophrenia and schizoaffective disorders. A lack of attribute drawings and high number of geometric symbols are representative of associative thinking disturbances in patients with these conditions. PT drawings of schizophrenia participants have decreased numbers of economical images, increased selection of unconventional geometric drawings, poor utility for memorization of targets, empty symbols that are not explained, and a lack of productive organization for the drawings. Nonpatients tended to use succinct drawings, more conventional associations for memorization, logical connection with targets, good memorization utility, and systematic organization of drawings on the page. The CI did not demonstrate discriminatory power in our analysis. This finding is not entirely consistent with previous

Downloaded By: [Leontieva, Luba] At: 19:56 25 February 2008

THE PICTOGRAM TEST descriptions of pictograms that have been produced by persons with schizophrenia (Khersonsky, 1979, 1988). Earlier research has indicated that patients’ pictograms tended to be concrete and overattached to literal interpretation of the targets, which did not help in memorizations. These findings indicate that concrete drawings are not specific for patients with schizophrenia. Qualitative analysis of the CI is needed to determine what degree of concreteness impedes the memorization process. Simultaneous comparisons of patients to nonpatients across both languages revealed that Russian- and English-speaking patients and nonpatients were similar in AI and GI. Analysis of classification statistics of AI and GI revealed that these indexes classified equally well patients from nonpatients in both languages. Thus, PT produced similar results in both language groups, indicating its cross-cultural similarities in schizophrenia patients and nonpatients. Comparative analysis of the special features of the PT across languages revealed that patients had more meaningless intermediate concepts (empty symbols) and more refusals to explain than nonpatients. These may indicate the difficulties that patients had in constructing appropriate drawings for memorization. Patients tended to produce meaningless drawings that did not help them to recall the targets or drawings that they could not explain at all. Consistent with prior findings (Khersonsky, 1979, 1988), empty symbols represented a hallmark of difficulties in logical thinking. More frequent use of letters by patients (despite PT instructions) than by nonpatients could indicate difficulty in following instructions or oppositional tendencies. Comparative analysis of graphical features of the drawings across languages revealed the tendency of patients to place their drawings chaotically or on the edge of the page. They also had significantly less numbering. These tendencies may indicate a lack of organizational strategy as patients drew their pictograms. Organizational strategies such as positioning all 16 drawings systematically on the page and numbering the drawings might be helpful in recall. Patients tend not to employ such strategies. Summarizing diagnostic applicability of the PT, it is possible to conclude that this test is applicable in the English language as well as Russian. Further, the AI and GI can discriminate patients with schizophrenia from nonpatients. The PT can indicate areas of associative memory deficit in schizophrenia patients and thereby provide insight into their logical thinking disturbances. Another test used for defining distortions of thought processes is the Rorschach Inkblot Technique (Exner, 1991). The PT is less labor intensive than the Rorschach in both scoring and interpretation. Additional studies using the PT could help in the development of normative data for various clinical populations, which would serve to expand the knowledge of these patients groups.

Implications The PT results suggest that participants with schizophrenia had difficulty making logical connections between target words and the drawings they made for each target. These results are consistent with recent findings (Hill et al., 2004) that schizophrenia patients have difficulties organizing information into cognitive sets. More research data is needed on the use of the PT in assessment batteries for diagnosing schizophrenia. As Hunsley and Meyer (2003) recommended, studies using regression analysis can be employed to investigate if the addition of the PT

173 can improve prediction on the criterion (i.e., schizophrenia diagnosis) and, consequently, present evidence of the incremental validity of the PT in test-informed clinical references. Understanding the thought disturbances of patients with schizophrenia allows for the design of therapeutic strategies to ameliorate these disturbances. The PT is a relatively straightforward means of characterizing areas in which patients could best realize practical gains from appropriate therapies. Specifically, teaching patients how to memorize abstract information more efficiently could help them apply what they learn in therapy to daily life. The PT can also be administered to persons without a psychiatric diagnosis. Our study provided normative results for 80 nonpatients. These results showed whether a test taker is methodical, accurate, precise, and whether he or she uses clear and logical constructs for productive memorization. With further research, the PT might be used for preemployment evaluations for positions requiring high levels of memory productivity and organization. The results of this test could be used to provide valuable recommendations for improvement of memory and for shaping thought in productive and logical ways.

Limitations In this study, we used a somewhat homogenous nonpatient sample as a control group. Many in this group were students. Although we selected nonpatients who reported that they had never been diagnosed with a psychiatric disorder, formal interviews, for example, SCID, were not conducted to verify the absence of diagnoses. In the patients’ group, there was a small percentage of people with a history of past alcohol abuse. Alcohol is known to have a negative impact on memory functioning. In this data analysis, we did not examine patients’ medication type and the severity of current symptoms. Patients whose symptoms were well controlled by medications could be potentially misclassified as nonpatients. Future Research A next step for PT research would be an investigation of its construct validity. Gathering PT data from other diagnostic groups (i.e., affective, personality, and substance abuse disorders and traumatic or organic brain injuries) would define the discriminative boundaries of PT interpretation. Comparing PT data from patients with schizophrenia and other diagnostic groups will help to understand the differences these patients have on this test. A comprehensive English version of the PT manual is needed to help structure scoring and interpretation of this test. The English version of the PT used in this study could be revised to prevent difficulties in understanding of the targets by test takers with limited vocabularies. Studies comparing schizophrenia patients’ pretreatment and posttreatment results from the PT could indicate whether their thinking becomes more logical and efficient after interventions. Further insight into associative memory functioning in patients with schizophrenia could be gained from an investigation of performance on the PT versus tests of direct word list memorization. ACKNOWLEDGMENTS We thank Jason Weimer and Drs. Aboraya, Cutlip, Adel, and Mogge for facilitating data collection in the United States. We thank Svetlana Voskoboinikova, Thomas Campbell, Konstantin

Downloaded By: [Leontieva, Luba] At: 19:56 25 February 2008

174

LEONTIEVA ET AL.

Lyavdansky, and Rebecca Mestemacher for their help with translation of the test. We greatly appreciate Paul W. Gately for his invaluable editing of the final version of this article. Finally, we thank Drs. Marinelli, Walls, DeLo, and Morgan for their comments, suggestions, and guidance. Research findings are based on the doctoral dissertation of Luba Leontieva, who was awarded the Certificate of Merit for outstanding student research from The Division of International Psychology of the American Psychological Association, July 2004. This article was presented at the 114th Annual Convention of the American Psychological Association (New Orleans, LA, August 2006).

REFERENCES American Psychiatric Association. (1994). Diagnostic and statistical manual of mental disorder (4th ed.). Washington, DC: Author. Arieti, S. (1955). Interpretation of schizophrenia (2nd ed.). New York: Basic Books. Beck, A. T., Steer, R. A., & Brown, G. K. (1996). Manual for the Beck Depression Inventory–II. San Antonio, TX: Psychological Corporation. Berezin, F. V., Miroshnikov, M. P., & Rozhanets, R. V. (1976). Metodika mnogostoronnego issledovania lichnosti [The multiscale personality test]. Moscow: Meditsina. Bespalko, I. G. (1978). Normative data for the Rorschach method. In Psikologicheskie metody issledovaniya kichnosti v klinike. Leningrad, Russia: Meditsina. Birenbaum, G. V. (1934). The formation of metaphors and conventional word meanings in the presence of pathological changes in intelligence. In Novoe v izychenii ob apraxii, agnozii i afazii [New developments in the study of apraxia, agnozia, and aphasia]. (pp. 147–164). Moscow: Izdatel ’stvo AN SSSR. Bleuler, E. (1950). Dementia praecox or the group of schizophrenias. New York: International Universities Press. (Original work published 1911) Brewer, W. J., Francey, S. M., Wood, S. J., Jackson, H. J., Pantelis, C., Phillips, L. J., et al. (2005). Memory impairments identified in people at ultra-high risk for psychosis who later develop first-episode psychosis. American Journal of Psychiatry, 162, 71–78. Calvo-Montalvo, N., & Villar-Diaz, I. (1988). Exploracion de las funciones analitico-sinteticas en pacientes esquizofrenicos [Exploration of analyticsynthetic functions in schizophrenic patients]. Boletin-de-Psicologia-Cuba, 11, 57–70. Exner, J. E. (1991).The Rorschach: A Comprehensive System: Vol. 2: Interpretation (2nd ed.). New York: Wiley. First, M. B., Spitzer, R. L., Gibbon, M., & Williams, J. B. W. (1997). Structured Clinical Interview for DSM–IV Axis I Disorders—Clinical version. Washington, DC: American Psychiatric Press. Goldstein, K. (1959). Concerning the concreteness in schizophrenia. Journal of Abnormal and Social Psychology, 59, 146–148. Hathaway, S. R., & McKinley, J. C. (1991). MMPI–2 Manual for administration and scoring. Minneapolis: University of Minnesota Press. Hemsley, D. R. (1992). Disorders of perception and cognition in schizophrenia. Revue Europeenne de Psychologie Applique, 42, 105–114. Hill, S. K., Beers, S. R., Kmiec, J. A., Keshavan, M. S., & Sweeney, J. A. (2004). Impairment of verbal memory and learning in antipsychotic-na¨ıve patients with first-episode schizophrenia. Schizophrenia Research, 68, 127–136.

Hunsley, J., & Meyer, G. J. (2003). The incremental validity of psychological testing and assessment: Conceptual, methodological, and statistical issues. Psychological Assessment, 15, 446–455. Khersonsky, B. G. (1979). Klinicheskoe znachenie “pustoi simvoliki” v pictogramme [The clinical relevance of “empty symbolism” in pictograms]. Zhurnal nevropatologii i psikhiatrii imeni S. S. Korsakova, 12, 1743–1744. Khersonsky, B. G. (1981). Pictogramma. Problema lichnostnogo podhoda k interpretazii dannuh issledovania [The pictogram: A problem of personality]. Zhurnal nevropatologii i psikhiatrii imeni S. S. Korsakova, 12, 1832–1838. Khersonsky, B. G. (1988). Method pictogram v psihodiagnostike psuhicheskih zabolevanii [The pictogram method in diagnostic of psychiatric disorders]. Moscow: Zdorov’ia. Khersonsky, B. G. (2000). The method of pictogram in psychodiagnostic. St. Petersburg, Russia: Sensor. Klasik, A. K. (2005). Short term memory disturbances in paranoid schizophrenia—Results of assessments with pictogram. Wiadomosci Lekarskie, 58, 170–173. Leont’ev, A. N. (1928). Oposredovannoe zapominanie u detei s nedostatochnum i boleznenno-izmenennum intellektom [Indirect memorizing in children with insufficient or distorted intellect]. Voprosu Defectologii, 4, 15–27. Leontieva, L. (2005). Establishing diagnostic applicability and validation of the Pictogram Test, Dissertation Abstracts International: Section B. The Physical Sciences and Engineering, 65(12B), 6659, US: Univ Microfilms International. Retrieved from https://eidr.wvu.edu/etd/document data. eTD? documentid=3528. Lezak, M. D. (1995). Neuropsychological assessment (3rd ed.). New York: Oxford University Press. Longinova, S. V. (1972). “Pictogramma” kak metod issledovania rasstroistv mushlenia pri schizophrenii [The “Pictograms” as an assessment method of distortion of thinking in schizophrenia]. Zhurnal nevropatologii i psikhiatrii imeni S. S. Korsakova, 11, 1679–1684. Perez-Ramirez, D., Miqueles-Estevez, A. E., & Hernandez-Melendez, E. (1985). Estudio patopsicologico de las alteraciones de la actividad mnesica en pacientes psiquiatricos [Psychopathological study of memory disturbances in psychiatric patients]. Boletin-de-Psicologia-Cuba, 8, 10–27. Rubinstein, S. Ia. (1970). Pictogram. In Experimental’nue metodiki patopsihologii. Prakticheskoe rukovodstvo [Experimental methods in pathopsychology. A practical guide] (pp. 160-171). Moscow: Medicina. Vygotsky, L. S. (1934). Thought in schizophrenia. Archives of Neurology and Psychiatry, 31, 1063–1077. Vygotsky, L. S. (1960). Razvitie vushih psuhicheskih funkzii [Development of higher cortical functions]. Moscow: APN RSFSR. Watson, C. G. (1973). Abstract thinking deficit and autism in process and reactive schizophrenics. Journal of Abnormal Psychology, 82, 399– 403. Whitaker, L. C. (1973). Whitaker Index of Schizophrenia (WIST): Manual. Los Angeles, CA: Western Psychiatric Services. World Health Organization. (1994). Megdunarodnaia Klassifikazia Boleznei, 10 peresmotr. Klassifikazia psihicheskih i povedencheskih passtroistv. Klinicheskie opisania I ukazania po diagnostike [Classification of psychiatric and behavioral disorders: Clinical descriptions for diagnosing]. In International Classification of Diseases (10th rev.) St. Petersburg, Russia: Author. World Health Organization. (1999). Schedules for clinical assessment in neuropsychiatry, Version 2.1 (SCAN): Interview. In World Health Organization: Assessment, classification and epidemiology. Geneva, Switzerland: Author.