Asem paper final-2012

A COMPARATIVE ANALYSIS OF TWO POPULAR QUALITY MEASUREMENT SYSTEMS IN THE AUTOMOBILE INDUSTRY Ozge Senoz, MS., Missouri University of Science and Technology William Daughton, Ph.D., University of Colorado - Colorado Springs Abhijit Gosavi, Ph.D., Missouri University of Science and Technology ___________________________________________________________________________________________ Abstract Consumer Reports and J.D. Power and Associates are two widely-known agencies that gather data on automobile quality and prepare reports on the data gathered. The reports they prepare usually contain a wealth of data related to various aspects of automobiles for different models for each manufacturer. The manufacturers surveyed include US-based companies, such as Ford Motors, GM, and Chrysler, and Asianbased companies, such as Toyota, Honda and Hyundai. Many of the metrics needed for quality and reliability measurement are common to reports from both agencies, but the scales on which they are measured are not always consistent. In this paper, we first present an approach to develop a realistic comparison of data related to a given metric for any given automobile type from the two reports. The rationale for this is to determine if the ratings (rankings) provided by the two agencies agree/match. We also conduct a preliminary analysis of the data from two categories of vehicles, Small and Mid-size SUVs, which currently tend to be the most popular automobiles in the US. Our goal here is to determine if there are any clear-cut differences between US and Asian vehicles in terms of the ratings and trends. Further, we also want to test how closely reports from the two agencies agree with each other for the vehicle classes selected. Regression analysis was used to determine any potential trends in the quality of the vehicle over time. Introduction In an earlier paper (Senoz et al., 2012), we carried out a qualitative comparison of two quality measurement systems, Consumer Reports (CR) and J.D. Powers and Associates (JDP). These agencies provide voluminous amounts of data on consumer surveys of a large number of automobiles of different types sold within the US. Although there are similarities and differences between these two sources, in terms of how they collect their data, the information they present, and how they evaluate and score vehicles, both sources provide information that sheds light on the current state of quality, reliability, and safety of automobiles currently on the market in the U.S. The majority of the data comes firsthand from owners who used the vehicles concerned.

In this paper, we first develop an approach to perform a realistic comparison of data related to a given metric for any given automobile type from the two reports. The reason for this, of course, is to determine if the ratings (rankings) provided by the two agencies agree/match. We look specifically into the ratings provided by the two systems for a limited set of models, in particular, Small and Midsize SUV models. These models were selected because they have large volumes of data on popular cars, and this provides us with an opportunity to make meaningful comparisons. The overarching goals of this study are (I) to determine if there are any clear differences between American and Japanese vehicles in terms of score range, and trends, and (ii) to determine if there is agreement between the two systems in terms of the results for the models we selected for comparison. The rest of this article is organized as follows. We first present some background on CR and JIP for the reader unfamiliar with these consumer surveys. Then, we discuss the methodology we used. Thereafter, we present some numerical results from our comparisons. Finally, we draw some conclusions from our empirical findings. Background on CR and JIP Consumer surveys have historically produced a significant impact on design and manufacturing of automobiles (Hauser and Clausing, 1988). Hence, it is no surprise that CR and JDP have acquired importance; they provide voluminous amounts of consumer survey data that manufacturers can use to improve quality. Brand names often convey signals of product quality to the consumer (Rao and Rukert, 1994), and consumer surveys can perform reality checks for potential buyers, with many brand names performing poorly. As a result, consumer surveys have become very important instruments in marketing. Unfortunately, product quality is often unobservable (Kirmani and Rao, 2000), and therefore consumer surveys are valuable tools for potential buyers. In what follows, we provide some background on the nature of data provided by CR and JDP. CR places

Copyright, American Society for Engineering Management, 2012

2

International Annual Conference of the American Society for Engineering Management

heavy emphasis on reliability issues and ratings related to that. It provides individual ratings on seventeen well-known trouble spots that vehicles may face. JDP, on the other hand, clusters these areas (trouble spots) into fewer groups of ratings. JDP provides ratings on two major categories: mechanical issues and design issues. It is important to note that CR has no survey ratings in terms of quality; however, it provides results and opinions based on their own road tests and evaluations. In spite of these differences, both CR and JDP provide useful information on the current status of reliability, quality, and safety of vehicles in the U.S. Also, both are well recognized in the media with frequent reports on news outlets such as CNN citing both sources. Overall, the large amount of information contained in these two sources can be quite illuminating, but given the lack of any scientific approach to compare or validate the two sources, it is fair to be concerned about how much value should be attached to the annual reports generated by them. In particular, one aspect that needs to be tested is whether both reports provide consistent results on a large number of vehicle categories. If consistency is found, that alone can increase the public trust in their findings. Our goals in this research, as noted above, are twofold. The first is to determine if there is consistency in at least some of vehicle categories. The second is to determine if there are any clear differences in the Asian and US auto-makers. Such comparisons are not new, and Hauser and Clausing (1988) is one of the earliest of such comparisons, where US and Japanese cars from the late 1980s were compared. However, what is interesting to test is if significant differences still exist in the perceived quality of US and Asian car-makers and if these differences have changed over time. We now provide a discussion on the methodology we used. Methodology The two systems were studied in Senoz et al. (2011) in terms of their data collection methodolgy, scoring/ratings, and categorization of attributes. The resulting similarities and differences that we discovered from our analysis are summarized in Exhibit 1. For the purposes of the study in this paper, from midsize and small SUV categories of vehicles, a representative model was selected for comparison. The first critical step in our methodology was to bring the results from the two systems to a common scale. The two sources used different scales; hence we had to conduct a conversion to a consistent numerical scale

for each vehicle for every year from each source – in order to obtain readings that could be compared. Exhibit 1. A qualitative comparison of CR and JDP.

In order to make the most useful comparisons, four representative attributes have been selected for our overall study. These are: Powertrain dependability, Body and Interior dependability, Feature and Accessory dependability, and Overall dependability. Models that had the most amounts of comparable data in terms of year were included in the analysis. Our intent was to determine whether the CR and JDP were providing similar numerical values for all the models, or only for some specific models, and whether the results varied from year to year. Empirical Results Assuming the data was normal, we performed linear regression analysis, seeking to predict JDP based on CR. This would potentially show how they were related. The General Linear Model ran the analysis based on vehicle classes (i.e., Family Sedans, Large Sedans, Sports, Small SUV, Midsize SUV) for every year between 2001 and 2010. As a result of this analysis, it was concluded that the correlation of vehicle ratings between CR and JDP is positive; however, it is very weak. Nevertheless, this correlation did not change from year to year, and it was constant across the 10 year span. R2 values were different from each other for every class, the highest being lower than 50%, which indicates that these two sources cannot be treated as giving the same results. CR and JDP are positively related, but not strong enough to conclude that we can substitute one for the other. For this paper, we present data from one out of the four attributes of reliability selected (named above): Powertrain reliability. In future work, we will analyze the other three attributes. Vehicle models with the most amount of year data from both CR and JDP were chosen. Data were plotted on charts according to

Author instructions and template to prepare ASEM manuscripts for IAC (Paper title)

vehicle class (i.e., Small SUVs and Midsize SUV), and reliability category for each source separately. The raw data plots were then converted to linear trendlines, and R2 values were noted. Polynomial trendlines and their corresponding R2 values were also plotted for the same data. Comparing the two, the trendlines with the best fit (i.e., highest R2) were chosen as the representative lines for that vehicle model. The Korean brands were left out for this analysis, because there is not enough data to make meaningful comparisons with Japanese and American models. Our results are presented via Exhibits 2 through 7. Exhibits 2 and 3 show the results of regression analysis performed for Small SUVs and Mid-size SUVs respectively. Exhibits 4 through 7 show the yearly trends.

Exhibit 2. Small SUVs

Exhibit 3. Mid-size SUVs

3

4

International Annual Conference of the American Society for Engineering Management

Exhibit 4. CR: Yearly Trends of Powertrain Reliability of Small SUVS.

Exhibit 5. JDP: Yearly Trends of Powertrain Reliability of Small SUVs.

Conclusions For this paper, we focused on the most meaningful results and trends that show obvious and clear similarities and differences between the American automakers and their Japanese counterparts. Further, we examined whether the two systems agreed or disagreed in their trendlines and scores. We presented specific examples of where and how they agreed, and how American and Japanese brands were doing against

Exhibit 6. CR: Yearly Trends For Powertrain Reliability of Midsize SUVs.

Exhibit 7. JDP: Yearly Trends For Powertrain Reliability of Midsize SUVs.

each other. In future work, we intend to perform a more comprehensive analysis of our findings with additional data sets. Our results are somewhat mixed and do not tell a clear story at this point. A significant amount of additional work, which we are in the process of carrying out, needs to be completed before a clearer picture emerges. It is evident, however, that the perceived differences of

Author instructions and template to prepare ASEM manuscripts for IAC (Paper title)

quality between US and Asian auto-makers are diminishing. A significant study has been carried out of the Korean automakers that links buyers’ supplier ratings to quality management practices (Park et al., 2001). Hence, it appears that a similar study for US automakers may reveal a similar relationship, making this an important topic for future research.

References

Hauser, John, and Clausing, Don, “The House of Quality”, Harvard Business Review; MayJune 1988; pp 63—73. Kirmani, Anma, and Rao, Akshay. R., “No Pain, No Gain: A critical Review of the Literature on Signaling Unobservable Product Quality”, Journal of Marketing, Vol 64, 2000; pp 66-79. Park, Suengwook, Jartley, Janet, and Wilson, Darryl, “Quality Management Practices and their Relationship to Buyer’s Supplier Rating: A Study in the Korean Automotive Industry”, Journal of Operations Management, Vol 19; 2001; pp 695-712. Rao, Ashkay R., and Ruekert, Robert W., “Brand Alliances as Signals of Product Quality”, Sloan Management Review, Fall 1994, pp 8797 Senoz, Ozge, Daughton, William, Gosavi, Abhijit, and Cudney, Elizabeth “An evaluation of professional quality measurement systems for the automotive industry”, International Journal of Engineering, Science and Technology, vol 3(7), 2011; pp 101-108. About the Authors Ozge Senoz obtained her BS and MS in Engineering Management both from the Missouri University of Science and Technology. She is currently pursuing a Ph.D. in Engineering Management at Missouri University of Science and Technology. Her research interests include quality management. William Daughton is Professor Emeritus of Engineering Management at Missouri University of Science and Technology. He is currently the Director of Extended Learning in the College of Engineering

5

and Applied Science at the University of ColoradoColorado Springs. He is also the Executive Director of ASEM. Abhijit Gosavi holds a BS and an MS in Mechanical Engineering from India and a Ph.D. in Industrial Engineering from the University of South Florida. He joined Missouri University of Science and Technology in 2008 as an Assistant Professor. His research interests include simulation-based optimization, Markov decision processes, and quality control.