Developing a comprehensive scale to assess college multicultural programming

Journal of Counseling Psychology 2014, Vol. 61, No. 1, 133–145

© 2014 American Psychological Association 0022-0167/14/$12.00 DOI: 10.1037/a0035214

Developing a Comprehensive Scale to Assess College Multicultural Programming Brent Mallinckrodt, Joseph R. Miles, Tripti Bhaskar, Nicole Chery, Gahee Choi, and Mi-Ra Sung

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

University of Tennessee A barrier to assessing effectiveness of multicultural programming is lack of a relatively brief instrument to measure the wide range of intended outcomes. A frequent goal of programming is to increase cultural empathy, but this is rarely the only intended outcome. We conducted focus groups of campus administrators, student affairs staff, and undergraduate instructors who identified a full range of racial/ethnic multicultural competencies that undergraduates should possess. An 84-item pool generated from these focus groups was combined with the 31-item Scale of Ethnocultural Empathy (SEE; Wang et al., 2003). These 115 items, together with instruments used to gauge concurrent validity, were administered to White undergraduate students in introductory psychology courses at the midpoint (n ⫽ 602) and end (n ⫽ 676) of fall semester. Exploratory factor analysis suggested 6 subscales for the Everyday Multicultural Competencies/Revised SEE (EMC/RSEE): (a) Cultural Openness and Desire to Learn; (b) Resentment and Cultural Dominance; (c) Anxiety and Lack of Multicultural Self-Efficacy; (d) Empathic Perspective-Taking; (e) Awareness of Contemporary Racism and Privilege; and (f) Empathic Feeling and Acting as an Ally. Item response theory principles guided final selection of subscale items. Analyses suggested good factor stability, reliability, and discriminant validity of the 48-item EMC/RSEE in these undergraduate samples. EMC/RSEE subscales were not strongly correlated with a measure of impression management and were significantly associated with measures of Openness to Diversity Challenge, and Universal-Diverse Orientation. Keywords: diversity programming, multicultural competencies, item response theory Supplemental materials: http://dx.doi.org/10.1037/a0035214.supp

In response to increasing diversity in the United States, many colleges and universities have invested considerable resources in multicultural education and programming (Howard-Hamilton, Cuyjet, & Cooper, 2011). Multicultural education takes many forms, including formal courses specifically designed to deliver multicultural content, and efforts to infuse multicultural content into existing courses (Adams, Bell, & Griffin, 2007). Multicultural programming includes workshops targeted at a general campus audience; for example, a workshop may be designed to increase awareness of inequalities, reduce prejudicial attitudes, or develop skills to combat discrimination and reduce bias. In contrast to serving a broad general audience, another type of multicultural programming is training targeted at specific groups, for example, residence hall paraprofessional peer counselors (Elleven, Allen, & Wircenski, 2001).

A growing body of research suggests that college multicultural programming is associated with a variety of positive outcomes, including increased empathy, intergroup understanding, communication skills, awareness about structural power relations, critical thinking skills, student engagement and motivation, and political involvement (Dessel, 2010; Gurin, Dey, Hurtado, & Gurin, 2002; Gurin, Dey, Gurin, & Hurtado, 2004; Gurin, Nagda, & Lopez, 2004; Nagda, Gurin, Sorensen, & Zúñiga, 2009). Although these studies have provided valuable information about multicultural interventions, researchers have been compelled to rely on batteries of single-purpose measures or a variety of unstandardized assessment tools and methods. The lack of a single, comprehensive, reliable, and valid measure with which to assess the wide range of intended outcomes of undergraduate multicultural programming has prevented educators and researchers from directly comparing outcomes of multicultural education and programming across interventions and studies. Answering the questions of “what works, for whom, and when” with respect to multicultural programming has been hampered by the lack of a common measure. Across the diverse range of multicultural programming, a frequent goal is to increase participants’ empathy for others who are culturally different. Ethnocultural empathy is a specific type of empathy for others whose racial/ethnic background differs from one’s own. Wang et al. (2003) drew from a model developed by Ridley and Lingle (1996) to conceptualize ethnocultural empathy as including components of (a) intellectual empathy, the cognitive ability to comprehend another’s cultural perspective; (b) emotional

Brent Mallinckrodt, Joseph R. Miles, Tripti Bhaskar, Nicole Chery, Gahee Choi, and Mi-Ra Sung, Department of Psychology, University of Tennessee. Tripti Bhaskar is now in independent practice, Tallahassee, Florida. We gratefully acknowledge the assistance of Richard Saudargas in this project. We also thank Patrick R. Grzanka and Joseph G. Ponterotto for their helpful feedback on earlier versions of this article. Correspondence concerning this article should be addressed to Brent Mallinckrodt, Department of Psychology, University of Tennessee, 1404 Circle Drive, Room 305, Knoxville, TN 37996. E-mail: [email protected] 133

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

134

MALLINCKRODT ET AL.

empathy, a capacity to feel affect in the present similar to another’s emotional experience; and (c) a capacity to communicate empathic understanding to another. Wang et al. developed the Scale of Ethnocultural Empathy (SEE) to assess this construct. Because this type of empathy is frequently a goal of campus multicultural programming, the SEE has been used to evaluate the effectiveness of these interventions. The SEE has demonstrated good validity and reliability in undergraduate samples, and a growing number of multicultural programming studies have used it as the principal measure of outcomes (cf. Phillips, 2012; Rasoal, Jungert, Hau, Stiwne, & Andersson, 2009). However, ethnocultural empathy is not the only goal of many multicultural interventions. A more comprehensive multidimensional measure is needed to capture a more complete range of outcomes. Thus, the general purpose of the current project was to expand the SEE to develop a multidimensional self-report instrument that could be used to assess the effectiveness of campus ethnic/racial diversity and multicultural programming efforts aimed at a broad undergraduate audience. Because increased empathy is so frequently an intended outcome of multicultural programs, we conceptualized this project fundamentally as an effort to revise and expand the SEE. Therefore, we chose as a working title for the new measure: Everyday Multicultural Competencies/Revised Scale of Ethnocultural Empathy (EMC/RSEE). We recognized that the goals of “typical” programs can be quite varied, so our intention was to develop an instrument with a subscale structure likely to capture the most important generic goals for this broad range of interventions. Howard-Hamilton et al. (2011) recently identified three clusters of undergraduate multicultural programming goals: (a) culturally relevant knowledge (e.g., knowledge of one’s own cultural identity, knowledge of the cultures of others), (b) multicultural skills (e.g., self-reflection, perspective-taking, intergroup communication), and (c) diversityrelated attitudes (e.g., pride in one’s own culture, belief that discrimination is unjust, belief that intergroup interactions enhance quality of life). This tripartite organizing scheme closely parallels the three domains of multicultural competence identified in training mental health professionals (Sue, Arredondo, & McDavis, 1992; Sue et al., 1982), although the target participants in these programs are general populations of college students. Thus, the theoretical framework of (a) knowledge, (b) skills, and (c) attitudes/awareness served as a useful starting point for developing an instrument to assess a broad range of multicultural programming. We adopted a ground-up inductive approach that relied on focus groups of college administrators, student affairs personnel, and instructors to identify specific knowledge, skills, and attitudes/awareness that undergraduate students should develop and that are typical outcome goals of multicultural programming. Second, we asked participants in these focus groups, together with other experts, to help generate an item pool to expand the 31-item SEE to capture this broader range of goals. Finally, we devised a data collection plan to solicit two large samples of undergraduates, separated by an interval of about six weeks, for the purpose of factor analysis, instrument development, and estimation of reliability and validity for the EMC/RSEE. We could not know in advance what new dimensions might emerge, so as a starting point we chose two of the same measures for exploration of convergent and divergent validity that Wang et al. (2003) chose in developing the SEE.

Because our measure was conceived as an extension of the SEE, we expected that any new factors that closely paralleled previous SEE subscales would show a similar pattern of significant correlations. The construct of universal-diverse orientation is defined as “an attitude of awareness and acceptance of both the similarities and difference that exist among people” (Miville et al., 1999, p. 291) and is believed to be essential for effective multicultural interactions. Individuals who hold this cluster of attitudes recognize commonalities of human experience that form a basis for empathy and mutual connection while, at the same time, they recognize and value profound differences in experience rooted in ethnicity, race, gender, and other social identities. The original 45-item instrument used to assess this construct is the Miville-Guzman Universality Diversity Scale (MGUDS; Miville et al., 1999). The shortened 15-item version used in this study (MGUDS-S; Fuertes, Miville, Mohr, Sedlacek, & Gretchen, 2000) consists of three subscales: (a) Diversity of Contact, which reflects predominantly behavioral intentions and commitment to participate in diverse social and cultural activities; (b) Relativistic Appreciation, which captures the core construct of simultaneously valuing both differences and similarities and which the authors believe is primarily a cognitive component; and (c) Comfort with Differences, which is primarily an affective component involving feeling comfortable in close contact with others who are culturally different. Consequently, we anticipated that any EMC/RSEE factor that tapped strong themes of overt behavioral intentions would be significantly correlated with Diversity of Contact, factor(s) tapping attitudes of acceptance and valuing of diversity would be significantly correlated with Relativistic Appreciation; and factor(s) tapping aspects of affect and empathy would be significantly correlated with Comfort with Closeness. The second measure used by Wang et al. (2003) that we used in this study to assess divergent validity was the Impression Management (IM) subscale of the Balanced Inventory of Desired Responding (BIDR; Paulhus, 1984). There is controversy in the personality assessment literature about the appropriateness of using scales like the IM subscale of the BIDR in instrument development. For example, a recent literature review argued that there is very little empirical evidence supporting an interpretation of IM as a form of defensive self-presentation (Uziel, 2010b). Instead, theory and research (Holden & Fekken, 1989; Uziel, 2010a) suggest that instruments such as the IM subscale of the BIDR actually are best considered measures of an adaptive form of interpersonal sensitivity in which self-control in public settings is a central feature. From this perspective, individuals who score high on a measure like the IM subscale of the BIDR have a higher tendency to value social harmony and agreeableness than those low in “impression management,” but this tendency does not prompt high scoring persons to intentionally misrepresent themselves (Uziel, 2010b). From this perspective, a significant correlation between counseling trainees’ self-assessments of their multicultural counseling skills and social desirability (as found by Teel, 2003) might be interpreted as evidence of interpersonal sensitivity, rather than as deceptive or defensive self-presentation. Thus, it is possible that the interpersonal sensitivity component of IM may be an essential component of multicultural skills. Acknowledging this controversy, we nevertheless included the IM subscale because information about correlations between this measure and subscales that

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

EVERYDAY MULTICULTURAL COMPETENCIES

emerge for the EMC/RSEE are likely to be useful for interpreting meaning of the new measures. Finally, we included a third measure, one not used by Wang et al., the Openness to Diversity/Challenge Scale (ODCS; Pascarella, Edison, Nora, Hagedon, & Terenzini, 1996). This brief, eight-item scale is often used as an outcome measure in campus multicultural programming to assess general openness to cultural and values diversity in one’s college environment. We chose the ODCS not only because of its widespread use but also because, as our item pool was developed through the focus group process, it became increasingly clear that negative dimensions were likely to emerge from the large number of these items being generated. Focus group informants, especially those working in residence halls, described how a portion of their efforts were directed at decreasing racist beliefs, and counteracting closed-minded, antidiversity attitudes. Consequently, we expected that any EMC/RSEE factor that emerged to capture the closed and negative attitudes toward diversity that programming sometimes seeks to reduce would be significantly, negatively correlated with ODCS scores. Worthington and Dillon (2011) pointed out that the particular multicultural counseling competencies that are effective in a given context are quite likely to vary, depending on the race and ethnicity of both the counselor and client. A single measure developed with only one type of dyad in mind (e.g., White counselors and clients of color) is unlikely to be valid when used with other types of dyads. Thus, in an effort to appropriately narrow the focus for our new scale, development of items for the EMC/RSEE was limited to the context of White students, on a predominantly White campus. Although psychologists always need to be mindful of not including the experiences of people of color, particularly in multicultural education and programming, it is also problematic to assume that the experiences of those from privileged groups (e.g., White people) in multicultural programs will be the same as those from marginalized groups (e.g., people of color). For example, White people have unique social and emotional experiences related to race and racism (Spanierman & Hepner, 2004), and those from marginalized social identity groups and those from privileged social identity groups have been found to have different experiences in multicultural interventions (Miles & Kivlighan, 2012). Therefore, although we collected data from a broad sample, we retained for analysis only data from students who reported their ethnic/racial identification as “White, European American,” and who checked no additional race/ethnicity. Although the decision to focus on White students restricts the generalizability of our results and range of potential uses for the new scale, students from the dominant White U.S. culture are frequently a focus for campus multicultural programming. Future studies will examine the validity and reliability of the measure with students of color. A final goal of this study arose from the belief that instrument development methods based on item response theory (IRT) are underutilized in counseling psychology research. We hoped to demonstrate the value of these methods in this project. Specifically, IRT criteria were used to (a) screen out items with high differential item functioning for women versus men, (b) evaluate and fine-tune performance of the Likert-type response scale, and (c) identify a set of items to retain for subscales with the highest capability for detecting differences between individuals across the fullest possible range of the underlying construct.

135 Method

Generating an Item Pool We began by conducting three focus groups with (a) key campus administrators responsible for diversity programming (Provost, Associate Provost, and Assistant Director of Residence Life), (b) 17 residence hall peer counselors, and (c) eight counseling psychology doctoral students involved in undergraduate teaching. We used the same core stimulus questions for each focus group derived from Sue et al.’s (1982) multicultural competencies model: Imagine a bright, ambitious undergraduate senior graduating from a large public university. Considering the range of multicultural environments, both in the U.S. and abroad, that this student is likely to experience in a productive career, please discuss the following three questions: (a) What are the attitudes, personal awareness, and ways of thinking that this student should possess to function effectively? (b) What are the skills that this student must acquire to function effectively? (c) What bases of knowledge must this student have to function effectively?

Follow-up questions asked for anecdotes and exemplars of multicultural sensitivity and openness to diversity, as well as instances of insensitivity and intolerance in daily interactions between undergraduate students. Participants were asked initially to identify domains and overall themes, and then invited to generate items to tap these concepts. Each of the focus group interviews was audio recorded. A coding team of one faculty member and eight graduate students independently read the transcripts and generated lists of domains and items. (These eight students were the same team that comprised the third focus group of undergraduate instructors.) Collating these lists with the items that focus group members suggested themselves resulted in a set of 15 domains. The domains were grouped into a Knowledge Cluster (three domains: e.g., “Our daily life in the U.S. is affected by other cultures,” “Recognizing White privilege”), a Skills Cluster (five domains: e.g., “Skills for communicating and connecting,” “Monitoring one’s own behavior so as not to be offensive”), and an Attitudes Cluster (six domains: e.g., “American culture is the best culture, the ‘normative’ culture, and others are inferior,” “Openness to new experiences”). All domain labels, together with sample items are shown in Table S1 in the online supplemental materials. The team generated items for each domain, but many items were deemed appropriate for tapping more than one domain. For example, several items negatively worded for the “Recognizing White privilege” domain (e.g., “In America everyone has an equal opportunity for success”) seemed to serve well as positive indicators for the “Colorblind attitudes” domain. An initial pool of 219 items was generated. Many of these items also tapped aspects of cultural empathy. After removing items redundant with the 31 SEE items and other redundant items, and editing to increase clarity, an initial pool of 84 items1 was generated for the Everyday Multicultural Competencies domain of 1 The original pool contained six items to tap knowledge of specific cultures, for example, “Deaf people have their own culture that is different from cultures of the hearing world.” Initial factor analysis showed these items tended to form one- and two-item factors. They were dropped from subsequent analyses.

136

MALLINCKRODT ET AL.

the new instrument. Judgments about item clarity were made collectively by the team without additional pilot testing.

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

Participants Undergraduate students in five sections of an introduction to psychology course at a large, public, predominantly White, Southeastern U.S. university completed surveys at Time 1, approximately the midpoint of fall semester. The same five sections plus three others were solicited to provide data at Time 2, 6 weeks later, 1 to 2 weeks before final exams. A total of 819 students provided data at Time 1, and 928 did so at Time 2. Of these students, 13 (1.6%) at Time 1 and 31 (3.3%) at Time 2 had missing data for more than 10% of the items included in factor analyses. These cases were excluded. Three validity check items (e.g., “Please code a seven for this item”) suggested that 107 (13%) students at Time 1 and 119 (13%) at Time 2 exhibited a random or inattentive pattern of responding. Data from these students were deleted. Regarding ethnic identification, 97 (14%) students at Time 1 and 102 (13%) at Time 2 reported an identification other than “White, European American” or did not answer this question. These data were also excluded from further analyses. Of the 602 remaining students who provided useable data at Time 1, 61% (n ⫽ 365) were women, and 39% (n ⫽ 237) were men. Their mean age was 18.56 years (SD ⫽ 1.51, range ⫽ 18 –37). Of the 676 students who provided useable data at Time 2, 60% (n ⫽ 404) were women, 40% (n ⫽ 270) were men, and two did not indicate their sex. Their mean age was 18.78 years (SD ⫽ 2.18, range ⫽ 18 –38).

Measures Scale of Ethnocultural Empathy (SEE; Wang et al., 2003). The SEE was designed to assess empathy for persons whose racial and ethnic background differs from one’s own. The 31 items form four subscales: (a) Empathic Feeling and Expression (15 items; e.g., “When I hear people make racist jokes, I tell them I am offended even though they are not referring to my racial or ethnic group”), (b) Empathic Perspective Taking (seven items; e.g., “It is easy for me to understand what it would feel like to be a person of another racial or ethnic background other than my own”), (c) Acceptance of Cultural Differences (five items; e.g., “I feel irritated when people of different racial or ethnic backgrounds speak their language around me” reverse keyed), and (d) Empathic Awareness (four items; e.g., “I am aware of how society differentially treats racial or ethnic groups other than my own”). Participants used a 6-point Likert-type scale, which was slightly modified for this study with the following anchors: 1 (strongly disagree), 2 (moderately disagree), 3 (slightly disagree), 4 (slightly agree), 5 (moderately agree), and 6 (strongly agree). Higher scores indicate greater ethnocultural empathy. In a sample of undergraduates, Wang et al. (2003) reported internal reliability (Cronbach’s alpha) for the four subscales of .90, .79, .71, and .74, respectively. Retest reliabilities (2-week interval) were .76, .75, .86, and .64, respectively. For the total SEE scale Wang et al. reported internal reliability of .91 and retest reliability of .76 (2-week interval). They reported evidence of validity in the form of significant correlations in samples of undergraduates with measures of general empathic perspective-taking, and with universal-diverse orientation (Miville et al., 1999). Higher SEE scores were associated

with having more racially/ethnically diverse family members and friends, and having attended a more diverse high school. In this study, the internal reliability (coefficient alpha) in Time 1 data for the four subscales were .89, .69, .77, and .72, respectively; and in Time 2 data were .88, .71, .82, and .74, respectively. Miville-Guzman Universality Diversity Scale—Short Form (MGUDS-S; Fuertes et al., 2000). The MGUDS-S is a 15-item version of the 45-item MGUDS (Miville et al., 1999), which was developed to assess an attitude of awareness and acceptance of the commonality and diversity that exist in ethnic and racial groups other than one’s own. Like the parent scale, the MGUDS-S, also has a three-factor structure: (a) Diversity of Contact (e.g., “I would like to join an organization that emphasizes getting to know people from different countries.”), (b) Relativistic Appreciation (e.g., “I can best understand someone after I get to know how he/she is both similar to and different from me.”), and (c) Comfort with Differences (e.g., “It’s really hard for me to feel close to a person of another race”). Higher scores indicate a greater appreciation of similarities and differences in individuals who are ethnically and culturally different. Participants use a 6-point, fully anchored Likert-type scale, ranging from 1 (strongly disagree) to 6 (strongly agree). The original MGUDS was positively correlated with healthy narcissism, empathy, positive attitudes toward feminism, androgyny, and positive aspects of African American and European American racial identity; and negatively correlated with dogmatism and homophobia (Miville et al., 1999). Fuertes et al. (2000) reported that the MGUDS-S was significantly correlated (r ⫽ .77) with the parent scale in a sample of undergraduates. Confirmatory factor analyses supported the three-factor structure of the MGUDS-S, with Cronbach’s alphas for the three subscales of .82, .59, and .92, respectively. For the 15-item total scale score, alpha ⫽ .77. Wang et al. (2003) reported that all three MGUDS-S subscales significantly correlated with all four SEE subscales. In the present study, internal reliabilities (coefficient alpha) for subscales of the MGUDS-S at Time 1 were .79, .82, and .81, respectively, and at Time 2 were .81, .86, and .81, respectively. Openness to Diversity/Challenge Scale (ODCS; Pascarella et al., 1996). The ODCS is an eight-item measure developed to assess individuals’ acceptance of being challenged by different ideas, values, and cultural perspectives, as well as their appreciation of racial and cultural diversity (e.g., “I enjoy having discussions with people whose ideas and values are different from my own”). Respondents use a fully anchored, 5-point Likert-type scale, ranging from 1 (strongly disagree) to 5 (strongly agree). Higher scores indicate greater openness to cultural diversity and challenge. Pascarella et al. (1996) reported internal consistency reliability of the scale with college students ranged from .83 to .84. Support for validity is evident in significant positive correlations of ODCS scores with student participation in cultural workshops, studying abroad, and greater interaction with diverse peers (Pascarella et al., 1996). In the current study the internal reliabilities (coefficient alpha) for this scale at Time 1 and Time 2 were .89 and .90, respectively. Balanced Inventory of Desired Responding (Paulhus, 1984). Only the 20-item Impression Management (IM) subscale was used in the current study. The IM is used to assess social desirability bias and the degree to which a respondent consciously presents inflated descriptions of her- or himself to please others (e.g., “I never take things that don’t belong to me”). The items use a fully

EVERYDAY MULTICULTURAL COMPETENCIES

anchored, 7-point response scale ranging from 1 (not at all true) to 7 (very true). The scoring scheme recommended by Paulhus (1984) was used in this study. Only responses of 6 or 7 were counted and assigned one point each. Thus, total scores range from 0 to 20, with higher scores indicating a greater tendency to respond in a more socially desirable way. The coefficient alpha for the IM in the present study was .78.

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

Procedure At both Times 1 and 2, survey packets were distributed during class. Students received extra credit toward their course grade for participating on either occasion. Given total enrollment in these sections of introductory psychology, the estimated research compliance rate for useable data was approximately 74% at Time 1 and 62% at Time 2. Students in the five repeated measure class sections were encouraged to provide data on both occasions, but not required to do so. They self-generated a code name to permit collating data, while protecting their anonymity. Of the 602 students who provided useable Time 1 data, 326 (54%) participated at Time 2 to provide a repeated measures sample. For the purposes of exploratory factor analysis (EFA) and confirmatory factor analysis (CFA) random selection of Time 1 versus Time 2 data was used so that the 326 retest subjects provided only one set of data, and the EFA and CFA subsamples remained completely independent. There were 952 independent cases for factor analysis, 276 Time 1 only students, 350 Time 2 only students, and 326 retest students randomly assigned to provide either 163 Time 1 and or 163 Time 2 data sets. These data were randomly allocated to an EFA sample (n ⫽ 585) that permitted at least five cases per item, with the remainder assigned to the CFA holdout sample (n ⫽ 367).

Results Exploratory Factor Analysis (EFA) The 84 Everyday Multicultural Competences items developed for this study were grouped together with the 31 items of the SEE for factor analysis. The EFA sample permitted 5.09 cases per item. We used SPSS Version 20.0 missing values regression procedure to estimate missing values for cases with less than 10% missing data. (Recall that cases with ⬎ 10% missing data were deleted.) The Kaiser-Meyer-Olkin measure of sampling adequacy was .94, suggesting that the 115-item matrix exhibited much more than sufficient covariation. A parallel analysis was conducted using syntax developed for SPSS by O’Connor (2000) to suggest the appropriate number of factors for extraction (Fabrigar, Wegener, MacCallum, & Straham, 1999; Worthington & Whittaker, 2006). Results from principal axis estimation of 1,000 simulated samples suggested that 19 factors should be extracted. We used squared multiple correlations on the diagonal (e.g., principal axis) in these estimates because it is the extraction method Worthington and Whitaker (2006) suggested for EFA. However, Buja, and Eyuboglu (1992) warned that principal axis parallel analysis tends to overestimate the number of factors to extract. Therefore, a parallel analysis using principal components extraction was conducted. It indicated that eight factors should be extracted. Following the recommendations of Worthington and Whitaker (2006), an EFA was performed using principal axis extraction with

137

oblique (direct oblimin) rotation to allow for correlated factors. The eight-factor solution resulted in extraction of two small factors of two items and three items. Therefore, both seven-factor and six-factor solutions were also inspected. The six-factor solution yielded more readily interpretable factors. Table 1 shows that the first six factors accounted for a total of 38.68% of the cumulative variance after extraction. Following recommendations of Worthington and Whitaker, an item was assigned to a factor if its loading was ⬎.32 on that factor and did not load within an absolute value of .15 on any other factor. This selection procedure resulted in retention of 82 items. Factor 1 ⫽ 25 items, Factor 2 ⫽ 22 items, Factor 3 ⫽ nine items, Factor 4 ⫽ six items, Factor 5 ⫽ 10 items, and Factor 6 ⫽ 10 items. The complete 115-item ⫻ six-factor pattern matrix is shown in Table S2, with item wordings shown in Table S3. The EFA was run again, this time imposing a six-factor solution on the selected 82 items. Only one item “changed allegiance” by loading more strongly on a different factor. It was dropped from Factor 3, leaving 81 items (22 from the SEE and 59 from the new item pool). Clearly, an 81-item instrument would be too cumbersome for the intended purposes of the EMC/RSEE. Our goal was to reduce factors to six to 10 items each, and to examine all items for poor conceptual fit and undesirable psychometric characteristics. Item 64, “Members from minority groups use many excuses to explain their laziness,” was eliminated from Factor 2 because we were concerned that many respondents would find this item too offensive. For the same reason we deleted Item 17, “Seeing people from different cultures makes me feel somewhat disgusted,” from Factor 3. The traditional method to reduce the length of a scale suggested by instrument development experts (cf. Worthington & Whittaker, 2006) has been to retain items with the highest factor loadings. Other experts (cf. Doucette & Wolf, 2009) assert that IRT should be used to generate additional empirical selection criteria. There are many of approaches to IRT, which can be roughly grouped according to the number of estimated parameters. The Rasch approach is a one-parameter IRT model (1-pl), which has the advantage of yielding a single score that reflects positioning along the hypothesized latent measurement construct (Doucette & Wolf, 2009). Note that three- and four-parameter approaches are also available to allow assessment of respondent guessing and careless responding. These models are used primarily by educational psychologists in performance test construction. The two-parameter model (2-pl) adds an item-discrimination dimension that can be useful in psychological assessment (de Alaya, 2009) but was not

Table 1 Exploratory Factor Analysis Eigenvalues and Total Variance Explained in 6-Factor Solution

Factor

Eigenvalue

% of variance

1 2 3 4 5 6

27.43 5.87 5.17 4.11 3.18 2.31

23.86 5.11 4.50 3.57 2.77 2.01

Note.

N ⫽ 585.

Cumulative % of variance

Cumulative % of variance after extraction

23.86 28.96 33.46 37.03 39.80 41.81

23.73 27.97 31.93 34.95 37.21 38.68

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

138

MALLINCKRODT ET AL.

used in present study. In contrast to 2-pl models, the 1-pl Rasch model used in this study has the desirable feature of specific objectivity, which allows the assumption of an invariant ranking of item difficulty independent of the abilities of individual test-takers (as well as an invariant ranking of person ability independent of the difficulty of items). The 2-pl estimates of item discrimination are always sample dependent, whereas the 1-pl Rasch model estimates of item difficulty carry an assumption of sample independence (Fox & Jones, 1998). Finally, the simplicity and parsimony of the Rasch model permits analyses of whether an instrument’s response scale functions as intended. Rasch model principles were used to assist in the final selection of items to retain for subscales, using WINSTEPS 3.80.0 software (Linacre, 2009). Each subscale was examined as an independent instrument. First, items were screened for disordered thresholds, that is, a failure of pairs of neighboring points on the Likert-type response scale to discriminate respondents appropriately. Of the remaining items, three were excluded for this reason. Next, items were screened for differential item functioning (DIF) with respect to sex. Significant DIF in this context means that for the item in question, a significantly higher amount of the underlying construct is required for one sex to cross the threshold of .50 probability for endorsing this item than the other sex. In large samples, levels of DIF that may be statistically significant for an item might nevertheless make only a trivial contribution to possible sex bias for the subscale as a whole (Osterlind & Everson, 2010). Thus, we set the threshold for DIF significance at less than p ⬍ .0005 using the Mantel-Haenszel ␹2 test. Six items were excluded for this reason. A third screen checked for the level of “underfit,” which can be considered an indication of poor fit for an item with the assumed unidimensional structure of a subscale. Nine items were excluded on this basis. After these screens, 61 of the 79 items remained. The final IRT screen selected items to provide the best spread of difficulty, in an effort to maximize sensitivity of the subscales in discriminating among individuals across the full range of scores on the latent construct of interest (Bond & Fox, 2007; de Alaya, 2009). Difficulty refers to the quantity of the latent construct that an individual must possess corresponding to a .50 probably of answering a specific item in the scored direction. The term is a legacy from the initial development of IRT methods to calibrate items on tests of ability. For a psychological construct such as depression, an item like “Once in a while I feel sad” would probably have low difficulty, whereas “Nearly every day I consider a plan for killing myself” would have high difficulty. A subscale with many low-difficulty items can make relatively finegrained discriminations among individuals who rank low on the construct but can make only gross distinctions among individuals at the high-score end of the continuum. Unless instrument developers can anticipate that users of a scale will be relatively more interested in a particular restricted range of scores, the wisest course is to retain items in a wide range of difficulty to maximize sensitivity across the widest possible range of scores, or “bandwidth” for each subscale (Embretson & Reise, 2000). In our screen for difficulty, pairs of items with approximately equivalent difficulty for a subscale were examined, with one member of each pair discarded to shorten the subscale. After the first three screens, the 28 items remaining on Factors 3– 6 provided a good spread of difficulty, so all these items were retained. However, 14 items remained for Factor 1, and 19 for Factor 2. For each factor, the

most and least difficult items were retained to preserve the maximum possible bandwidth. Within these extremes, one member of each pair of items closely spaced in difficulty was eliminated until both subscales were reduced to 10 items each. Mallinckrodt, Miles, and Recabarren (2013) have provided a detailed practical guide for using IRT criteria in the final stages of item selection. Note that only six of the 10 items included for final selection in Factor 1, and six of the final 10 items selected for Factor 2 were among the 10 highest loading items in the EFA, that is, would have been selected using traditional methods for shortening a subscale. The traditional “highest loading” method would have resulted in retention of two items for Factor 1 (m22 and m27) with highly significant sex DIF, and a more constricted bandwidth of item difficulty than the IRT-assisted method used in this study. Finally, Rasch model screening was applied to evaluate performance of the Likert response scale at the level of total subscale scores. Results suggested crossed category Andrich thresholds, that is, a failure of a pair of neighboring responses for the subscale as a whole to differentiate respondents. This, in turn, suggests that the response scale contains too many categories, some of which do not contribute useful variance to the total scale score (Bond & Fox, 2007). Collapsing categories with crossed Andrich thresholds reduces error variance. Thus, the responses strongly agree and moderately agree were combined for Factor 2 (coding, 6 ⫽ 5). The 48-item EMC/RSEE is shown in Table 2. A fully formatted version of the scale is reproduced in the online supplemental materials. Scoring instructions for the EMC/RSEE are provided in the table notes. To name the factors we developed the preliminary labels shown in Table S4. To refine these labels, we engaged consultants who were student affairs professionals on our campus (n ⫽ 3) or national experts in diversity programming, all of whom are doctoral level researchers (n ⫽ 6). They were randomly assigned to one of two teams, each presented with the clusters of items and given a different factor labeling task. Team A was asked to make a selection from among 12 choices that best fit each cluster of items. In contrast, each member of Team B was asked to generate his or her own label in fill in a blank following each set of items. Results of both tasks are shown in supplemental Table S4. Responses to Task A suggested that the preliminary label for Factor 1 was most in need of revision, because only three of the five consultants selected our label. Responses to the open-ended rating Task B suggested a new label for Factor 1, and more minor modifications to the other labels. Based on our consultants’ responses, we selected the following labels: (a) Cultural Openness and Desire to Learn, (b) Resentment and Cultural Dominance, (c) Anxiety and Lack of Multicultural Self-Efficacy, (d) Empathic Perspective-Taking, (e) Awareness of Contemporary Racism and Privilege, and (f) Empathic Feeling and Acting as an Ally.

Confirmatory Factor Analysis Wirth and Edwards (2007) warned that ordered-categorical data (e.g., Likert-type items), when used as single-item indicators, violate many of the necessary assumptions of confirmatory factor analysis (CFA), including a linear relationship between observed and latent variables, and continuous normal distribution of outcomes. Floyd and Widaman (1995) recommended using item parcels in CFA instead of individual items, especially for subscales with more than five to eight items, due to problems introduced

EVERYDAY MULTICULTURAL COMPETENCIES

139

Table 2 Everyday Multicultural Competencies/Revised Scale of Ethnocultural Empathy Final Items Selected

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

Item no.

Item

Corrected total subscale with item correlationa

m65. m50. m61. m31. m12. m07. m06. m73. m38. m40.

Factor 1: Cultural Openness and Desire to Learn (10 items, ␣ ⫽ .92) I think it is important to be educated about cultures and countries other than my own. I welcome the possibility that getting to know another culture might have a deep positive influence on me. I admire the beauty in other cultures. I would like to work in an organization where I get to work with individuals from diverse backgrounds. I would like to have dinner at someone’s house who is from a different culture. I am interested in participating in various cultural activities on campus. Most Americans would be better off if they knew more about the cultures of other countries. A truly good education requires knowing how to communicate with someone from another culture. I welcome being strongly influenced by my contact with people from other cultures. I believe the United States is enhanced by other cultures.

.77 .75 .73 .72 .70 .62 .69 .66 .63 .62

m10. m35. m66. m54. s10. m71. m85. m01. m60. m18.

Factor 2: Resentment and Cultural Dominance (10 items, ␣ ⫽ .85) Members of minorities tend to overreact all the time. When in America, minorities should make an effort to merge into American culture. I do not understand why minority people need their own TV channels. I fail to understand why members from minority groups complain about being alienated. I feel irritated when people of different racial or ethnic backgrounds speak their language around me. Minorities get in to school easier and some get away with minimal effort. I am really worried about White people in the U.S. soon becoming a minority due to so many immigrants. I think American culture is the best culture. I think members of the minority blame White people too much for their misfortunes. People who talk with an accent should work harder to speak proper English.

.62 .58 .58 .57 .57 .55 .55 .51 .51 .50

m19. m14. m28. m04. m95. m75. m09.b

I I I I I I I

s19. s28.b s31.b s06. s02.b

Factor 3: Anxiety and Lack of Multicultural Self-Efficacy (7 items, ␣ ⫽ .77) feel uncomfortable when interacting with people from different cultures. often find myself fearful of people of other races. doubt that I can have a deep or strong friendship with people who are culturally different. really don’t know how to go about making friends with someone from a different culture. am afraid that new cultural experiences might risk losing my own identity. do not know how to find out what is going on in other countries. am not reluctant to work with others from different cultures in class activities or team projects.

Factor 4: Empathic Perspective-Taking (5 items, ␣ ⫽ .69) It is easy for me to understand what it would feel like to be a person of another racial or ethnic background other than my own. It is difficult for me to put myself in the shoes of someone who is racially and/or ethnically different from me. It is difficult for me to relate to stories in which people talk about racial or ethnic discrimination they experience in their day to day lives. I can relate to the frustration that some people feel about having fewer opportunities due to their racial or ethnic backgrounds. I don’t know a lot of information about important social and political events of racial and ethnic groups other than my own.

m15.b m25.b

Factor 5: Awareness of Contemporary Racism and Privilege (8 items, ␣ ⫽ .79) The U.S. has a long way to go before everyone is truly treated equally. For two babies born with the same potential, in the U.S. today, in general it is still more difficult for a child of color to succeed than a White child. I can see how other racial or ethnic groups are systematically oppressed in our society. Today in the U.S, White people still have many important advantages compared to other ethnic groups. I am aware of how society differentially treats racial or ethnic groups other than my own. I am aware of institutional barriers (e.g., restricted opportunities for job promotion) that discriminate against racial or ethnic groups other than my own. Racism is mostly a thing of the past. In America everyone has an equal opportunity for success.

s21.b s15. s03. s26. s16.b

I I I I I

m39. m34. s20. m20. s25. s07.

Factor 6: Empathic Feeling and Acting as an Ally (8 items, ␣ ⫽ .81) don’t care if people make racists statements against other racial or ethnic groups. get disturbed when other people experience misfortunes due to their racial or ethnic background. am touched by movies or books about discrimination issues faced by racial or ethnic groups other than my own. share the anger of people who are victims of hate crimes (e.g., intentional violence because of race or ethnicity). rarely think about the impact of a racist or ethnic joke on the feelings of people who are targeted.

.61 .57 .52 .51 .50 .41 .34 .55 .53 .44 .37 .33

.58 .56 .53 .51 .50 .46 .45 .38 .58 .59 .57 .53 .49 (table continues)

MALLINCKRODT ET AL.

140 Table 2 (continued)

Item no. s30. s22. s11.

Corrected total subscale with item correlationa

Item When I hear people make racist jokes, I tell them I am offended even though they are not referring to my racial or ethnic group. When I see people who come from a different racial or ethnic background succeed in the public arena, I share their pride. When I know my friends are treated unfairly because of their racial or ethnic backgrounds, I speak up for them.

.48 .48 .48

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

Note. m ⫽ item generated specifically for this study, s ⫽ item taken from the Scale of Ethnocultural Empathy. N ⫽ 952. a Correlations between an item and the subscale total score with that item omitted. b Reverse key.

through correlated errors between pairs of items with similar wording. Alhija and Wisenbaker (2006) conducted a simulation study comparing the approaches and reported that item parcels performed better than individual indicators. The main argument in favor of single item indicators in CFA is that parcels might mask multidimensionality (Worthington & Whittaker, 2006). Because IRT based methods were used to screen items for multidimensionality, and because of the disadvantages of single item indicators, in this study three multiple-item parcels were constructed as indicators of each factor. The CFA was conducted using Mplus Version 6.12 (Muthén & Muthén, 2010) on the holdout sample (n ⫽ 367). In addition to the ␹2 likelihood ratio of exact model fit, we examined three approximate fit indices suggested by Kline (2011): the comparative fit index (CFI), the root-mean-square error of approximation (RMSEA), and the standardized root-mean-square residual (SRMR). Hu and Bentler (1999) suggested that CFI values ⬎.95; SRMR values ⬍.08; and RMSEA values ⬍.06 are desirable. Customarily, the 90% confidence interval (CI) for RMSEA is also reported. Browne and Cudeck (1993, p. 144), described RMSEA values ⬍.05 as consistent with a “close fit,” in the range .05–.08 as “reasonable error of approximation,” and ⬎.10 as unacceptable. However, ␹2 tends to be significant in models with large sample sizes, high correlations between variables, and nonnormal distributions—all of which are characteristic of these data. Table 3 shows that all loadings of parcels on the six latent variables were statistically significant. Table S5 presents correlations between the six latent variables. A maximum likelihood estimator was used for this CFA. The test of exact model fit was significant, ␹2 ⫽ 292.686 (df ⫽ 120), p ⬍ .0001; but approximate fit indices suggested a reasonable fit of the model to the data, CFI ⫽ .950, SRMR ⫽ .047, and RMSEA ⫽ 0.063 (90% CI: 0.054 – 0.072).

Reliability and Validity Table 2 reports that internal reliability (coefficient alpha) in the independent data sample (n ⫽ 952) ranged from .92 (Factor 1) to .69 (Factor 4). Table 4 reports test–retest reliabilities (n ⫽ 326, 5-week interval) that ranged from .80 for Factor 6, to .62 for Factor 3. Two subscales had retest reliability ⬍ .70, Factor 3, Anxiety and Lack of Multicultural Self-Efficacy, and Factor 4 Empathic Perspective-Taking. The constructs these subscales measure might be expected to change during the last half of a semester that for many students was their first in college. Among students 20 years or older (who were likely to be at least juniors), retest reliability of Factor 3 and 4 was .72 and .78, respectively. Table 4 also shows

that only one pair of subscales at the same time point were correlated ⬎ .50, indicating that each subscale assesses a separate construct. Consequently, we recommend against using a total scale score for the EMC/RSEE. Table 5 reports correlations relevant to discriminant validity. At both Time 1 and Time 2, EMC/RSEE subscale correlations with impression management scores were very modest (all rs ⬍ .18). Specific pairs of correlations also supported the conceptual underpinning of the factors. As expected, the MGUDS-S Diversity of Contact subscale was correlated at both Time 1 and Time 2 most strongly with Factor 1, which contains items tapping behavioral intentions and commitment to participate in campus activities. MGUDS-S Relativistic Appreciation is primarily a cognitive factor. Although no EMC/RSEE subscale captures a purely cognitive dimension, as expected, the strongest correlations were with Factor 1, which contains three items assessing knowledge and education. MGUDS-S Comfort with Differences is conceptualized as primar-

Table 3 Standardized Indicator Loadings on Latent Variables in Confirmatory Factor Analysis Indicator Factor 1 Parcel Parcel Parcel Factor 2 Parcel Parcel Parcel Factor 3 Parcel Parcel Parcel Factor 4 Parcel Parcel Parcel Factor 5 Parcel Parcel Parcel Factor 6 Parcel Parcel Parcel Note.

Loading

SE

Loading/SE

p

1 2 3

.919 .906 .871

.012 .013 .015

79.109 72.430 56.823

.000 .000 .000

1 2 3

.784 .706 .812

.027 .032 .026

28.905 22.255 31.663

.000 .000 .000

1 2 3

.710 .746 .740

.034 .032 .032

20.917 23.439 23.064

.000 .000 .000

1 2 3

.676 .806 .605

.041 .039 .044

16.620 20.699 13.875

.000 .000 .000

1 2 3

.752 .875 .773

.028 .022 .027

26.436 39.716 28.565

.000 .000 .000

1 2 3

.884 .695 .685

.022 .032 .033

39.584 21.912 20.586

.000 .000 .000

N ⫽ 367.

EVERYDAY MULTICULTURAL COMPETENCIES

141

Table 4 Everyday Multicultural Competencies/Revised Scale of Ethnocultural Empathy Correlations Time 2 Time 1

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

Factor Factor Factor Factor Factor Factor M SD

1 2 3 4 5 6

M

SD

Factor 1

Factor 2

Factor 3

Factor 4

Factor 5

Factor 6

4.39 3.22 2.31 2.95 3.81 4.12

0.85 0.78 0.75 0.86 0.82 0.80

.77ⴱⴱ ⫺.49ⴱⴱ ⫺.47ⴱⴱ .22ⴱⴱ .18ⴱⴱ .49ⴱⴱ 4.37 0.87

⫺.46ⴱⴱ .77ⴱⴱ .43ⴱⴱ ⫺.26ⴱⴱ ⫺.32ⴱⴱ ⫺.46ⴱⴱ 3.28 0.80

⫺.40ⴱⴱ .42ⴱⴱ a .62ⴱⴱ ⫺.16ⴱⴱ ⫺.08 ⫺.33ⴱⴱ 2.43 0.80

.31ⴱⴱ ⫺.23ⴱⴱ ⫺.21ⴱⴱ a .66ⴱⴱ .12ⴱ .26ⴱⴱ 3.09 0.81

.34ⴱⴱ ⫺.39ⴱⴱ ⫺.15ⴱⴱ ⫺.06 .71ⴱⴱ .30ⴱⴱ 3.83 0.81

.57ⴱⴱ ⫺.48ⴱⴱ ⫺.34ⴱⴱ .23ⴱⴱ .27ⴱⴱ .80ⴱⴱ 4.05 0.78

Note. Correlations below the diagonal are Time 1 subscale intercorrelations (n ⫽ 602); correlations above the diagonal are Time 2 subscale intercorrelations (n ⫽ 676); correlations on the diagonal are Times 1 and 2 test–retest correlations (n ⫽ 326). Retest interval was approximately five weeks. Factor 1 ⫽ Cultural Openness and Desire to Learn; Factor 2 ⫽ Resentment and Cultural Dominance; Factor 3 ⫽ Anxiety and Lack of Multicultural Self-Efficacy; Factor 4 ⫽ Empathic Perspective-Taking; Factor 5 ⫽ Awareness of Contemporary Racism and Privilege; Factor 6 ⫽ Empathic Feeling and Acting as an Ally. a Among the subset of students 20 years old or older, test–retest reliability for Factor 3 ⫽ .72, and Factor 4 ⫽ .78. ⴱ p ⬍ .05. ⴱⴱ p ⬍ .01.

racial/ethnic diversity and multicultural programming outcomes. A recent review of the “fatal flaws” that contributed to rejection of manuscripts submitted to the Journal of Counseling Psychology (Hoyt & Mallinckrodt, 2012) emphasized that, in instrument development research, it is vital to generate an initial pool of items that adequately samples the intended conceptual domain. If key aspects of the construct(s) of interest are not tapped by prospective items, validity of the resulting measure will be compromised. Hoyt and Mallinckrodt recommend using focus groups of persons intimately familiar with the construct as one way to help ensure a breadth of items are generated and that items are worded in the colloquial language of potential respondents. For these reasons, we used three focus groups to generate an initial item pool. A rough indicator of construct validity of the EMC/RSEE is that 13 of the 15 overlapping content domains generated through this process are

ily an affective subscale. As expected, this subscale was correlated most strongly (and negatively) at Time 1 and 2 with Factor 3 that taps anxiety. We expected Openness to Diversity/Challenge to have the strongest correlation (in absolute value) with dimensions of the EMC/RSEE that emerged to tap negative and close-minded attitudes toward diversity. Contrary to expectations, correlations with Factor 2, Resentment and Cultural Dominance, were not higher than correlations with other factors. Overall, the pattern of correlations shown in Table 5 supports the expected pattern for convergent and divergent validity of the EMC/RSEE, except that correlations for the ODCS did not show a single peak of strong correlations.

Discussion The primary goal of this project was to develop a relatively brief self-report measure for assessing a broad range of undergraduate

Table 5 Convergent and Divergent Validity EMC/RSEE factor Variable

1

2

3

4

5

6

Impression Management MGUDS⫺S Diversity of contact MGUDS⫺S Relativistic appreciation MGUDS⫺S Comfort with differences Openness to Diversity/Challenge

.10ⴱ .74ⴱⴱ .68ⴱⴱ .51ⴱⴱ .76ⴱⴱ

Time 1 (N ⫽ 602) ⫺.12ⴱⴱ ⫺.43ⴱⴱ ⫺.37ⴱⴱ ⫺.54ⴱⴱ ⫺.48ⴱⴱ

⫺.07 ⫺.39ⴱⴱ ⫺.42ⴱⴱ ⫺.67ⴱⴱ ⫺.39ⴱⴱ

.08ⴱ .35ⴱⴱ .17ⴱⴱ .20ⴱⴱ .26ⴱⴱ

.03 .23ⴱⴱ .33ⴱⴱ .19ⴱⴱ .30ⴱⴱ

.17ⴱⴱ .47ⴱⴱ .48ⴱⴱ .47ⴱⴱ .48ⴱⴱ

Impression Management MGUDS⫺S Diversity of contact MGUDS⫺S Relativistic appreciation MGUDS⫺S Comfort with differences Openness to Diversity/Challenge

⫺.01 .58ⴱⴱ .41ⴱⴱ .44ⴱⴱ .77ⴱⴱ

Time 2 (N ⫽ 676) ⫺.05 ⫺.35ⴱⴱ ⫺.11ⴱ ⫺.44ⴱⴱ ⫺.40ⴱⴱ

.00 ⫺.38ⴱⴱ ⫺.27ⴱⴱ ⫺.52ⴱⴱ ⫺.41ⴱⴱ

.05 .28ⴱⴱ .07 .16ⴱ .32ⴱⴱ

⫺.01 .22ⴱⴱ .16ⴱ .12ⴱ .30ⴱⴱ

.10ⴱ .33ⴱⴱ .28ⴱⴱ .36ⴱⴱ .47ⴱⴱ

Note. EMC/RSEE ⫽ Everyday Multicultural Competencies/Revised Scale of Ethnocultural Empathy; MGUDS ⫽ Miville-Guzman Universality Diversity Scale—Short Form (Fuertes, Miville, Mohr, Sedlacek, & Gretchen, 2000). Factor 1 ⫽ Cultural Openness and Desire to Learn; Factor 2 ⫽ Resentment and Cultural Dominance; Factor 3 ⫽ Anxiety and Lack of Multicultural Self-Efficacy; Factor 4 ⫽ Empathic Perspective-Taking; Factor 5 ⫽ Awareness of Contemporary Racism and Privilege; Factor 6 ⫽ Empathic Feeling and Acting as an Ally. ⴱ p ⬍ .05. ⴱⴱ p ⬍ .01.

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

142

MALLINCKRODT ET AL.

clearly represented in items retained for the final version of the scale (see online supplemental materials). Items from only two domains, “Our daily life in the U.S. is affected by other cultures” and “Monitoring one’s own behavior so as to not be offensive,” were not retained after the EFA and IRT-assisted selection process. As discussed below in regard to individual factors, the clusters of Knowledge, Skills, and Attitudes were also well represented among the 48 final items. However, it should be noted that the items written to tap specific cultural knowledge (e.g., “Different cultures have different norms about taking turns in a conversation”) did not function as intended, and tended to create one- and two-item factors and were deleted from final analyses. Instead, the domain of multicultural knowledge is assessed by the EMC/RSEE through items that tap personal awareness (“I am aware of how society differentially treats racial or ethnic groups other than my own”) and attitudes toward learning (“I think it is important to be educated about cultures and countries other than my own”), rather than assessing specific cultural knowledge. Programming aimed at increasing specific cultural knowledge may need to assess these outcomes with ad hoc content-based knowledge exams in much the same way that mastery of other college course content is tested. Turing now to the specific factors, at least one result of this study confirms the advantages of focus groups, because it is exceedingly unlikely that our research team alone would have generated the items that eventually formed EMC/RSEE Factor 2, Resentment and Cultural Dominance. Our team had been focused exclusively on attitudes, skills, and knowledge that diversity programming is designed to promote. However, the residence hall staff focus group drew our attention to the importance of negative, prejudicial attitudes that their programming efforts target for reduction. In addition to promoting positive change, these staff members were sometimes confronted in their day-to-day work by students who hold the negative attitudes captured by the items of Factor 2. Thus, fortunately, the EMC/RSEE includes one subscale to assess attitudes that some programs seek to reduce. A further advantage of Factor 2 is that it can be used to assess unintended negative consequences or “backlash” effects of multicultural programming. Not all students support diversity-affirming, critical multicultural policies and programs in higher education. For example, a growing body of research has found that individuals who most strongly endorse a color-blind racial ideology are less likely to support affirmative action programs (Oh, Choi, Neville, Anderson, & Landrum-Brown, 2010), less likely to support the discontinuation of racialized athletic mascots (Neville, Yeung, Todd, Spanierman, & Reed, 2011), and more likely to perceive the campus climate (both generally, and for racial and ethnic minority students) at their university positively (Worthington, Navarro, Loewy, & Hart, 2008). Therefore, those who endorse a color-blind ideology may enter multicultural programs with more negative expectations or attitudes than those lower in color-blind ideology. Interestingly, some of the items generated for the focus group domain of “color-blind attitudes” emerged in the EFA as negatively worded items contributing to Factor 5, Awareness of Contemporary Racism and Privilege. Consistent with this research, Factor 5 was most strongly (negatively) correlated with Factor 2. Ideally, successful multicultural programming in higher education engages students and helps them develop more positive attitudes toward multiculturalism. Factor 2 of the EMC/RSEE could be used to assess baseline negative attitudes toward cultural diver-

sity. Resentment and cultural dominance, as assessed by Factor 2, may reflect an expected part of a developmental/educational process that occurs in multicultural education. For example, McIntosch (1998) suggested that the dominant U.S. culture actively “teaches” us not to examine or challenge systems of privilege and oppression, thus reproducing these systems. By engaging students in difficult dialogues on social identities, hierarchical social structures, privilege, and oppression, critical multicultural programming requires individuals to face the discomfort associated with examining their roles in systems of oppression, as well as cherished beliefs and ideologies (Freire, 2008). Thus, Factor 2 may provide important information about an aspect of students’ developmental process. Factor 1, Cultural Openness and Desire to Learn, accounted for more variance than the other factors, and assesses broad goals that are among the most frequent targets of multicultural and diversity programming, such as increasing interest in other cultures and stimulating an interest in breaking out of one’s cultural encapsulation. Most of the 10 items tap the attitudes aspect of multicultural competencies together with strong themes of campus engagement, behavioral intentions, and positive affect (e.g., “I am interested in participating in various cultural activities on campus,” “I admire the beauty in other cultures”). In tapping elements of both selfawareness attained from, and positive attitudes toward, multicultural learning, Factor 1 is conceptually similar to the ODCS (Pascarella et al., 1996), as reflected in their strong correlation. In contrast to Factor 1, Factor 5, Awareness of Contemporary Racism and Privilege, taps a knowledge component of multicultural competencies, and is much more narrowly focused, namely, on awareness of contemporary racism and privilege. Note also that Table 4 shows Factors 1 and 5 were only modestly correlated (r ⫽ .18 at Time 1 and .34 at Time 2). Table 4 also shows that means for Factor 1 were more than 0.66 SD units higher than Factor 5 at both Time 1 and 2. Perhaps Factor 1 taps basic “entry-level” everyday multicultural competencies (EMCs), whereas the awareness of racism and privilege tapped by Factor 5 are more advanced EMCs. This is consistent with the inference that Factor 1 is conceptually similar to the ODCS (Pascarella et al., 1996), which was developed to assess openness to diversity and challenge in first-year college students. Factor 5, on the other hand, appears to reflect a more sophisticated component of awareness that eschews attitudes tapped by Color-Blind Racial Attitudes Scale (CoBRAS; Neville, Lilly, Duran, Lee, & Browne, 2000), which assesses Racial Privilege, Institutional Discrimination, and Blatant Racial Issues. Future research should examine the relationship between Factor 5 of the EMC/RSEE and the CoBRAS to determine the similarities and differences of these measures. Factors 4 and 6, Empathic Perspective-Taking, and Empathic Feeling and Acting as an Ally, respectively, both measure aspects of empathy that were well represented in subscales of the SEE. However, unlike some of the SEE subscales, neither of these factors was strongly correlated with IM in this study (all rs ⬍ .17). The two subscales were also not highly correlated with each other (r ⫽ .26 at Time 1, r ⫽ .23 at Time 2), indicating that empathic feelings are relatively independent of actions motivated by empathy. The difference in the two constructs is that Factor 4 fits a more conventional definition (e.g., Ridley & Lingle, 1996) of empathy as a cognitive perspective-taking ability (e.g., “ . . . put myself in the shoes . . . ,” “ . . . understand what it would feel like . . . ” “ . . .

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

EVERYDAY MULTICULTURAL COMPETENCIES

relate to the stories . . . ”), whereas Factor 6 reflects empathy more as the impetus for concrete behavioral and affective reactions (e.g., confronting racist jokes, speaking up for friends, getting disturbed, and sharing the anger or pride of oppressed minorities). Finally, we turn to Factor 3, Anxiety and Lack of Multicultural Self-Efficacy, which together with Factor 2 are the only two subscales of the EMC/RSEE keyed such that higher scores represent undesirable programming outcomes. Much more than any of the other factors, Factor 3 taps the skills aspect of EMCs (e.g., “I really don’t know how to go about making friends with someone from a different culture”), although several items tap a negative affective component (e.g., “I feel uncomfortable when interacting with people from different cultures”). Self-efficacy theory (Bandura & Locke, 2003) predicts that students in our sample might experience anxiety or discomfort, and engage in avoidance of social situations they feel ill-equipped to handle. For this reason, we anticipate that scores on EMC/RSEE Factor 3 may be especially responsive to programming that emphasizes experiential learning of skills through face-to-face interactions, for example, intergroup dialogue. Intergroup dialogue is explicitly structured in the early stages to establish a safe group climate in which students can learn and develop new ways of communicating with one another about difficult issues (e.g., Zúñiga, Nagda, Chesler, & Cytron-Walker, 2007). Research on intergroup dialogue suggests that this intervention is associated with increased confidence and self-efficacy in intergroup interaction, and with participation in social action (DeTurk, 2006). Factor 3 of the EMC/RSEE provides a useful, standardized way to examine the development of confidence and self-efficacy as the result of participation in multicultural programming like intergroup dialogue. Although the six subscales of the EMC/RSEE provide coverage for a broad range of racial/ethnic diversity and multicultural programming targets, it is important to acknowledge limitations of the measure. First, it is intended to assess only aspects of White undergraduate students’ EMCs related to racial or ethnic diversity, and not the many other important dimensions of diversity and culture (e.g., ability, gender, sexual orientation). The item generation process did not take into account EMCs, for example, that students of color or international students need for success on a predominantly White campus. Therefore, the validity of the EMC/ RSEE for students of color is unknown. In fact, because research suggests that bicultural competency skills are associated with coping in this type of college environment for ethnic minority students (Wei et al., 2010), the EMC/RSEE is certainly lacking in some critical dimensions for these students. Another omitted set of coping skills are those used by sexual minority students to cope with heterosexist campus environments (e.g., Schmidt, Miles, & Welsh, 2011). A further limitation is that the measure was designed only for use with college students. Its validity for assessing multicultural programming outcomes in other populations remains unknown. The title implies that the EMC/RSEE assesses undergraduate multicultural competencies, but the counselor training literature suggests that self-reports of skill competencies correlate very poorly with observer ratings (Worthington, Mobley, Franks, & Tan, 2000). In contrast, for many domains of self-efficacy, undergraduates’ self-report ratings accord well with external measures and observed behavior (Betz, 2013). Nevertheless, the EMC/RSEE is prone to the limitations of self-report measures. Although we tried

143

to create a measure as broadly applicable as possible, providing accurate context-sensitive assessment may require that the EMC/ RSEE be supplemented with selected specifically targeted measures. The practice of pairing the EMC/RSEE with other measures will help establish construct validity, while providing a common metric for comparison across interventions. Several of the EMC/ RSEE subscales appear to be conceptually similar to existing measures (e.g., Factor 1 and the ODCS; Factor 5 and the CoBRAS; Neville et al., 2000). Further research is needed to examine the incremental predictive validity of the EMC/RSEE factors relative to existing measures. Several additional avenues for further research are suggested by these findings. Propensity for the development of multicultural competencies and sociocultural empathy may also be related to individual differences, such as multicultural personality (e.g., Ponterotto, 2010), not assessed in the current study. Multicultural personality encompasses relatively stable traits that allow for successful navigation of intercultural situations (e.g., cultural empathy, open-mindedness, social initiative, emotional stability, and flexibility; van der Zee, van Oudenhoven, Ponterotto, & Fietzer, 2013). Although our goal was to develop a standardized measure of multicultural competencies gained through participation in a wide-range of racial and ethnic diversity-related programming, perhaps the EMC/RSEE taps relatively stable individual differences. In addition, change in scores on the EMC/RSEE (reflecting the development of multicultural competence and sociocultural empathy) may be strongly influenced by more stable personality traits. Clearly, several of the factors found in the current study (e.g., Cultural Openness and Desire to Learn, Empathic Perspective-Taking, and Empathic Feeling and Acting as an Ally) bear similarities to several of the multicultural personality traits suggested by van der Zee et al. (2013). Additional research using the EMC/RSEE is needed to illuminate the relationships between individual differences such as these, and the development of multicultural competencies and sociocultural empathy, thereby allowing educators and administrators to refine multicultural programming to fit the needs of individual students. Further field testing of the EMC/RSEE is needed to demonstrate its utility. As more data are gathered, the subscales can be standardized to provide a common metric for comparison that we hope will advance theory and research regarding how students acquire “everyday” multicultural competencies necessary for to successfully function in a diverse society.

References Adams, M., Bell, L. A., & Griffin, P. (Eds.). (2007). Teaching for diversity and social justice (2nd ed.). New York, NY: Routledge. Alhija, F. N.-A., & Wisenbaker, J. (2006). A Monte Carlo study investigating the impact of item parceling strategies on parameter estimates and their standard errors in CFA. Structural Equation Modeling: A Multidisciplinary Journal, 13, 204 –228. doi:90/10.1207/s15328007sem1302_3 Bandura, A., & Locke, E. A. (2003). Negative self-efficacy and goals effects revisited. Journal of Applied Psychology, 88, 87–99. doi: 10.1037/0021-9010.88.1.87 Betz, N. E. (2013). Assessment of self-efficacy. In N. E. Betz (Ed.), APA handbook of testing and assessment in psychology: Testing and assessment in clinical and counseling psychology (Vol. 2, pp. 379 –391). Washington, DC: American Psychological Association. doi:10.1037/ 14048-022

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

144

MALLINCKRODT ET AL.

Bond, T. G., & Fox, C. M. (2007). Applying the Rasch model: Fundamental measurement in the human sciences (2nd ed.). Mahwah, NJ: Erlbaum. Browne, M. W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In K. A. Bollen & J. S. Long (Eds.), Testing structural equation models (pp. 136 –162). Newbury Park, CA: Sage. Buja, A., & Eyuboglu, N. (1992). Remarks on parallel analysis. Multivariate Behavioral Research, 27, 509 –540. doi:10.1207/s15327906mbr2704_2 de Alaya, R. J. (2009). The theory and practice of item response theory. New York, NY: Guilford Press. Dessel, A. B. (2010). Effects of intergroup dialogue: Public school teachers and sexual orientation prejudice. Small Group Research, 41, 556 –592. doi:10.1177/1046496410369560 DeTurk, S. (2006). The power of dialogue: Consequences of intergroup dialogue and their implications for agency and alliance building. Communication Quarterly, 54, 33–51. doi:10.1080/01463370500270355 Doucette, A., & Wolf, A. W. (2009). Questioning the measurement precision of psychotherapy research. Psychotherapy Research, 19, 374 –389. doi:10.1080/10503300902894422 Elleven, R. K., Allen, J., & Wircenski, M. (2001). Resident assistant training: A southwestern perspective. College Student Journal, 35, 609 – 615. Embretson, S. E., & Reise, S. (2000). Item response theory for psychologists. Mahwah, NJ: Erlbaum. Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychology research. Psychological Methods, 4, 272–299. doi:10.1037/1082-989X .4.3.272 Floyd, F. J., & Widaman, K. F. (1995). Factor analysis in the development and refinement of clinical assessment instruments. Psychological Assessment, 7, 286 –299. doi:90/10.1037/1040-3590.7.3.286 Fox, C. M., & Jones, J. A. (1998). Uses of Rasch modeling in counseling psychology research. Journal of Counseling Psychology, 45, 30 – 45. doi:10.1037/0022-0167.45.1.30 Freire, P. (2008). Pedagogy of the oppressed. New York, NY: Continuum. Fuertes, J. N., Miville, M. L., Mohr, J. J., Sedlacek, W. E., & Gretchen, D. (2000). Factor structure and short form of the Miville-Guzman Universality-Diversity Scale. Measurement and Evaluation in Counseling and Development, 33, 157–169. Gurin, P., Dey, E. L., Gurin, G., & Hurtado, S. (2004). Educational benefits of diversity. In P. Gurin, J. Lehman, & E. Lewis (Eds.), Defending diversity: Affirmative Action at the University of Michigan (pp. 96 –188). Ann Arbor, MI: University of Michigan Press. Gurin, P., Dey, E. L., Hurtado, S., & Gurin, G. (2002). Diversity and higher education: Theory and impact on educational outcomes. Harvard Educational Review, 72, 330 –366. Gurin, P., Nagda, R., & Lopez, G. (2004). The benefits of diversity in education for democratic citizenship. Journal of Social Issues, 6, 17–34. doi:http://dx.doi.org/10.1111/j.0022-4537.2004.00097.x Holden, R. R., & Fekken, G. C. (1989). Three common social desirability scales: Friends, acquaintances, or strangers? Journal of Research in Personality, 23, 180 –191. doi:10.1016/0092-6566(89)90022-6 Howard-Hamilton, M. F., Cuyjet, M. J., & Cooper, D. L. (2011). Understanding multiculturalism and multicultural competence among college students. In M. J. Cuyjet, M. F. Howard-Hamilton, & D. L. Cooper (Eds.), Multiculturalism on campus: Theory, models, and practices for understanding diversity and creating inclusion (pp. 11–18). Sterling, VA: Stylus. Hoyt, W. T., & Mallinckrodt, B. (2012). Improving the quality of research in counseling psychology: Conceptual and methodological issues. In N. A. Fouad, J. A. Carter, & L. M. Subich (Eds.), APA handbook of counseling psychology: Theories research and methods (Vol. 1, pp. 31–57). Washington, DC: American Psychological Association. doi: 10.1037/13754-003

Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indices in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1–55. doi:90/10.1080/10705519909540118 Kline, R. B. (2011). Principles and practice of structural equation modeling (3rd ed.). New York, NY: Guilford Press. Linacre, J. M. (2009). A user’s guide to WINSTEPS/MINISTEP Raschmodel computer programs: Program manual 3.68.0. Beaverton, OR: Winsteps.com. Retrieved from http://www.winsteps.com/ Mallinckrodt, B., Miles, J. R., & Recabarren, D. A. (2013). Advantages of using Rasch model item response theory in instrument development: Empirical demonstration in a measure of multicultural competencies. Manuscript submitted for publication. McIntosch, P. (1998). White privilege and male privilege: A personal account of coming to see correspondences through work in women’s studies. In M. L. Andersen & P. H. Collins (Eds.), Race, class, and gender: An anthology (3rd ed., pp. 94 –105). Belmont, CA: Wadsworth. Miles, J. R., & Kivlighan, D. M. (2012). Perceptions of group climate by social identity group status in intergroup dialogue. Group Dynamics: Theory, Research, and Practice, 16, 189 –205. doi:10.1037/a0028634 Miville, M. L., Gelso, C. J., Pannu, R., Liu, W., Tauradji, P., Holloway, P., & Fuertes, J. (1999). Appreciating similarities and valuing differences: The Miville-Guzman Universality-Diversity Scale. Journal of Counseling Psychology, 46, 291–307. doi:10.1037/0022-0167.46.3.291 Muthén, L. K., & Muthén, B. O. (2010). Mplus user’s guide (6th ed.). Los Angeles, CA: Muthén & Muthén . Nagda, B. A., Gurin, P., Sorensen, N., & Zúñiga, X. (2009). Evaluating intergroup dialogue: Engaging diversity for personal and social responsibility. Diversity & Democracy, 12, 4 – 6. Neville, H. A., Lilly, R. L., Duran, G., Lee, R. M., & Browne, L. (2000). Construction and initial validation of the Color-Blind Racial Attitudes Scale (CoBRAS). Journal of Counseling Psychology, 47, 59 –70. doi: 10.1037/0022-0167.47.1.59 Neville, H. A., Yeung, J. G., Todd, N. R., Spanierman, L. B., & Reed, T. D. (2011). Color-blind racial ideology and beliefs about a racialized university mascot. Journal of Diversity in Higher Education, 4, 236 –249. doi:10.1037/a0024334 O’Connor, B. P. (2000). SPSS and SAS programs for determining the number of components using parallel analysis and Velicer’s MAP test. Behavior Research Methods, Instruments, & Computers, 32, 396 – 402. doi:90/10.3758/BF03200807 Oh, E., Choi, C., Neville, H. A., Anderson, C. J., & Landrum-Brown, J. (2010). Beliefs about affirmative action: A test of the group self-interest and racism beliefs models. Journal of Diversity in Higher Education, 3, 163–176. doi:10.1037/a0019799 Osterlind, S., & Everson, H. (2010). Sage quantitative series in the social sciences: Differential item functioning. Thousand Oaks, CA: Sage. Pascarella, E. T., Edison, M., Nora, A., Hagedon, L. S., & Terenzini, P. T. (1996). Influences on students’ openness to diversity and challenge in the first year of college. Journal of Higher Education, 67, 174 –195. doi:10.2307/2943979 Paulhus, D. L. (1984). Two-component models of socially desirable responding. Journal of Personality and Social Psychology, 46, 598 – 609. doi:10.1037/0022-3514.46.3.598 Phillips, D. A. (2012). Exploration of the value of electronic discourse for developing cultural competency in college freshmen. Dissertation Abstracts international Section A: Humanities and Social Sciences, 72, 2340. Ponterotto, J. G. (2010). Multicultural personality: An evolving theory of optimal functioning in culturally heterogeneous societies. The Counseling Psychologist, 38, 714 –758. doi:10.1177/0011000009359203 Rasoal, C., Jungert, T., Hau, S., Stiwne, E. E., & Andersson, G. (2009). Ethnocultural empathy among students in health care education. Evaluation & the Health Professions, 32, 299 –311. doi:10.1177/ 0163278709338569

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

EVERYDAY MULTICULTURAL COMPETENCIES Ridley, C. R., & Lingle, D. W. (1996). Cultural empathy in multicultural counseling: A multidimensional process model. In C. R. Ridley, & D. W. Lingle, Counseling across cultures (4th ed., pp. 21– 46). Thousand Oaks, CA: Sage. Schmidt, C. K., Miles, J. R., & Welsh, A. C. (2011). Perceived discrimination and social support: The influences on career development and college adjustment of LGB college students. Journal of Career Development, 38, 293–309. doi:10.1177/0894845310372615 Spanierman, L. B., & Heppner, M. J. (2004). The Psychosocial Costs of Racism to Whites Scale (PCRW): Construction and initial validation. Journal of Counseling Psychology, 51, 249 –262. doi:10.1037/00220167.51.2.249 Sue, D. W., Arredondo, P., & McDavis, R. J. (1992). Multicultural counseling competencies and standards: A call to the profession. Journal of Multicultural Counseling and Development, 20, 64 – 88. doi:10.1002/j .2161-1912.1992.tb00563.x Sue, D. W., Bernier, J., Durran, M., Feinberg, L., Pedersen, P., Smith, E., & Vasquez-Nuttall, E. (1982). Position paper: Cross-cultural counseling competencies. The Counseling Psychologist, 10, 45–52. doi:10.1177/ 0011000082102008 Teel, F. S. (2003). The relationship between impression management and multicultural awareness, knowledge, and skills. Dissertation Abstracts International: Section B: The Sciences and Engineering, 64, 2587. Uziel, L. (2010a). Look at me, I’m happy and creative: On the effect of impression management on behavior in social presence. Personality and Social Psychology Bulletin, 36, 1591–1602. doi:10.1177/ 0146167210386239 Uziel, L. (2010b). Rethinking social desirability scales: From impression management to interpersonally oriented self-control. Perspectives on Psychological Science, 5, 243–262. doi:10.1177/1745691610369465 van der Zee, K., van Oudenhoven, J. P., Ponterotto, J. G., & Fietzer, A. W. (2013). Multicultural Personality Questionnaire: Development of a short form. Journal of Personality Assessment, 95, 118 –124. doi:10.1080/ 00223891.2012.718302

145

Wang, Y.-W., Davidson, M. M., Yakushko, O. F., Savoy Bielstein, H., Tan, J. A., & Bleier, J. K. (2003). The Scale of Ethnocultural Empathy: Development, validation, and reliability. Journal of Counseling Psychology, 50, 221–234. doi:10.1037/0022-0167.50.2.221 Wei, M., Liao, K. Y.-H., Chao, R. C.-L., Mallinckrodt, B., Tsai, P.-C., & Botello-Zamarron, R. (2010). Minority stress, perceived bicultural competence, and depressive symptoms among ethnic minority college students. Journal of Counseling Psychology, 57, 411– 422. doi:10.1037/ a0020790 Wirth, R. J., & Edwards, M. C. (2007). Item factor analysis: Current approaches and future directions. Psychological Methods, 12, 58 –79. doi:10.1037/1082-989X.12.1.58 Worthington, R. L., & Dillon, F. R. (2011). Deconstructing multicultural counseling competencies research: Comment on Owen, Leach, Wampold, and Rodolfa (2011). Journal of Counseling Psychology, 58, 10 –15. doi:10.1037/a0022177 Worthington, R. L., Mobley, M., Franks, R. P., & Tan, J. A. (2000). Multicultural counseling competencies: Verbal content, counselor attributions, and social desirability. Journal of Counseling Psychology, 47, 460 – 468. doi:1O.1O37//0O22-0167.47.4.460 Worthington, R. L., Navarro, R. L., Loewy, M., & Hart, J. (2008). Colorblind racial attitudes, social dominance orientation, racial-ethnic group membership and college students’ perceptions of campus climate. Journal of Diversity in Higher Education, 1, 8 –19. doi:10.1037/1938-8926 .1.1.8 Worthington, R. L., & Whittaker, T. A. (2006). Scale development research: A content analysis and recommendations for best practices. The Counseling Psychologist, 34, 806 – 838. doi:10.1177/0011000006288127 Zúñiga, X., Nagda, B. A., Chesler, M., & Cytron-Walker, A. (2007). Intergroup dialogue in higher education: Meaningful learning about social justice. ASHE Higher Education Report, 32(4), 1–128.

Received July 23, 2012 Revision received October 13, 2013 Accepted October 15, 2013 䡲