Ewan Camel, The American University
New views of mature ideas on sohare quality and productivity.
The software-package segment of the soft7are market most close4 resembles the manufarmring-for-consumption model on which most quality techniquesare based. Ewan Camel reports on his obsmations of the development processes in 1Y package companies, specijkally ho?v these developers v i m quality-assurance practices and what they have in place. Camel is an assistant professor at the Kogod College of Business Administration at the American University in Washington, DC. -Dave Card
AN EVER GREATER SLICE O F T H E US software pie is going to software packages rather than to the more traditional custom software tailored for a single customer or user. Moreover, US companies command an estimated 80 percent of the global software-package market. Given such dominance, more attention should be paid to how these software companies develop software -to the process itself. In some ways the development approaches of these companies have been successful and should be emulated. In other ways - most notably the qualityassurance process -the jury is still out. Software packages are generally reliable, but everyone has encountered bugs in them. For example, shortly after I wrote this column, Microsoft released DOS 6.0, a package that reportedly received extensivetesting with thousands of users and still there were complaints of bugs.
to this. When I asked questions related to quality assurance, many of those interviewed were slightly embarrassed and tended to answer in rationatizations. In one case, at the end of the quality-assurance part of the interview, a manager whistled in relief. T h e attitude toward quality assurance was closely related to the absence or incomplete use of an overall development methodology. Most companies were allocating relatively little effort to formalizing specifications,design, process documentation, or testing procedures. Written documents, if they existed at all, tended to be brief. Overall, there was little in the way of softwarequality-assurance practices besides testing. O n l y two c o m p a n i e s had regular c o d e walkthroughs; none seemed to have validation and verification in place; only one company used software metrics, although several companies collected problem reports. Testing itself was not comprehensive in either its application or its scope, and there was little investment in testing tools. Furthermore, developers were violating several software-testing principles: + N o t only were some smaller companies testing their own programs and writing their own test suites (as you might expect), but so were some larger companies. + Developers were modifying and adding features during testing. + Developerswere testingincrementally instead of comprehensively. Finally, as a further distraction from quality assurance, some companies were still struggling without configuration-management methods or tools.
IN MOST COMPANIES, QUALITY ASSURANCE TOOK A BACK SEAT TO OTHER CONSIDERAmS, BUT SOME MANAGERS SEEMED TO FEEL GUlLTYABOlCT IT.
Editor: David Cord ComputerSciences Corp. Centml PokTwo#700 4061 Powder Mill Rd. Coberton, MO 20705 lntetnet
[email protected] IEEE SOFTWARE
1
~-
QUALITY GUILT. One measure of quality is the absence of defects,or bugs. In an ongoing study of development processes in softwarepackage companies, I chose 15 companies representing diverse applications and examined their development practices in-depth. Most had fewer than 100 employees, and all but one developed mostly for the desktop market The results of the study were not unanticipated: In most companies, the quality-assurance process took a back seat to other considerations and practices, but there seemed to be plenty of p d t attached
EXPLANATIONS AND ARGUMEHIS. This picture may seem discouraging, but many developers vehemently argue that they should not be practicing quality assurance in the classic way. They offer numerous arguments which I present below, along with a few that I propose on the basis of
1
my observations. The first set of arguments fall into what I call market, o r external, arguments. They explain why customers buy buggy software products, for example. + Expertatiom gap. Developers know that the marketplace will tolerate some level of defects. Furthermore, they know that power users and early adapters are more forgiving than later adapters. “[Our] customers would like 100 percent error-free software, but they expect 96-97-98 percent.” + Marginal cost. With quality assurance being a major component of development cost, it would drive up the product cost, pricing software at unreasonable levels and reducing revenues -“I can’t sell my software at a million dollars a copy. So the Japanese write a 12-line program that’s six sigma. Big deal.” + Market niche. This is my observation. I contend that the more vertical the product’s market, the less rigorous the testing. T h e customers in vertical markets maintain a direct relationship with the vendor, and errors are fixed quickly. Furthermore, the narrower the niche, the less competitive it tends to be. The second set of arguments has to do with internal pressures (or lack thereof) to apply more formal quality assurance - the “we don’t need more quality assurance because...” arguments: + We’rebetter than the rest. One developer took this argument beyond the visceral, explaining that formalization of development processes is a way for the expertise embedded in the method to compensate for the weaknesses of individual developers. Many software-package companies felt that they had enough developer expertise to accomplish what formal quality-assurance procedures are designed to do. - “We’ve got the best [developers]on the planet.” + Pew pressure. Some companies use pride to reward and shame to pressure developers into being more thorough. Peer pressure then replaces formal quality assurance. + Reuse. Code reuse is high. Libraries of traditional or object-oriented code, which
I
has been thoroughly tested, make up an everincreasing part of s o h a r e packages. I see two additional internal factors that affect attitudes toward adopting formal quality-assurance procedures: + Hacker dominance. Many package companies are dominated or influenced by the hacker culture, and hackers are generally averse to many of quality assurance’s formalisms. + Ignorance. Most developers have little or no experience or training in quality assurance.
BEFORE WE JUDGE THE PRACTICES OF PACKAGE DEVELOPERS, WE SHOULD ’ ASK IF THEY R EALLY NEED FORW LL QA PROCE DURES*
86
RISK ASSESSMENT. All these arguments figure into the larger issue of risk assessment. Risk assessment is conducted at every juncture, either implicitly or explicitly, as companies weigh their quality-assurance decisions against what it takes to keep them competitive in the marketplace. They ask questions like + Do we add functionality during the late testing phase in order to compete with X? + Do we drop quality levels in order to make a deadline and release ahead of competition? + How many rounds of beta testing can we afford to do with this version? In smaller companies, risk assessment is done implicitly, almost subconsciously. Better managed companies make explicit decisions about quality and time-to-market trade-offs. Many companies appear to be assigning lower risk-consequence values to quality-related items than to market-oriented factors.
POINT of LIGHT. One conspicuous exception in my study was a company with fewer than 10 employees that develops packages for the mass market. T h e company had a textbook-like software process that followed a 120-page softwareengineering life-cycle manual written by the key developers. Testing procedures included binders of carefully documented test suites, release thresholds, and a defect database. This unusual quality-assurance rigor
came about because one team member brought quality-assurance experience with him from Hewlett-Packard. H e argued that, even for such a small company, quality assurance was a good investment. First, he claimed good results: only two minor fures and no major software errors in the 18 months since release of the company’s very first product. Second, the methodology introduced predictability, allowing his small company to accurately predict release dates more than a year in advance.
IS IT THE RIGHT FIT? Before judging the practices of software-package quality assurance as a repeat of the US auto industry in the 1950s, we should question if applying the current quality-assurance paradigm in its entirety fits the needs of package development. T h e philosophy of software quality assurance is derived from and directed to Fortune 500 management-information systems and military applications - “big software.” Although there are more instances these days of “big” software-package efforts in the microcomputer market (particularly systems software such as IBM OS/2, Microsoft Windows NT, and Apple System 7 ) , these are still the exception. Most package development is not big. Thus, it may not be appropriate to apply the process formalisms of the US Department of Defense, IEEE, or the International Standards Organization to package companies with 20 or SO employees. There are also cultural differences. Package companies are entrepreneurial in nature, and their developers may indeed have stronger financial and emotional motivations to excel than developers in custom-software environments, who tend to use more bureaucratic process models. Finally, preliminary comparisons from Howard Rubin and Associates of Pound Ridge, New York (H. Rubin, “Inside the Information Systems Black Hole,” Rubin Review, No. 4, 1992) between software packages and traditional software show that, on average, package software has half as many defects per line of code, and the best package software has defects that are several orders of magnitude less than the best traditional software. Perhaps there is more than one way to achieve an acceptable level of quality. SEPTEMBER
1
1993
