Session T4A
Practical Program Outcomes Assessment – A Case Study Venu G. Dasigi Department of Computer Science & Software Engineering, Southern Polytechnic State University, 1100 S. Marietta Parkway, Marietta, GA 30060-2855
[email protected] Abstract - In this paper, we characterize assessment in terms of outcomes that indicate that the stated objectives are met, and assessment methods that measure outcomes at desired levels of performance. We consider factors such as the need for some direct measures, issues of faculty buyin, the extra cost of formal assessment, etc. We identify two assessment methods, neither of which is new, but brought together, make it practical to assess many common outcomes. The Faculty Course Assessment Report (FCAR), developed by John Estell, provides for directly assessing course outcomes, which are mapped to program outcomes. The second direct assessment method involves using an Industry Advisory Board (IAB) in evaluating capstone courses. Here, many program outcomes, including students’ employability, are assessed. We discuss why these methods are practical and desirable. We describe some implementation considerations, as well as supplementary methods. Finally, we consider the need for an integrated approach to assessment. Index Terms – Course and Program Outcomes, Faculty Course Assessment Reports, Capstone Evaluations, Industry Advisory Boards. INTRODUCTION Assessment has lately become a major endeavor in many academic programs [1]. Many universities have also been looking into putting infrastructures in place for across-theboard assessment of all programs, because of regional and professional accreditation requirements. The importance of and the need for regular assessment of academic programs are undeniable; sometimes budget allocations are tied to assessment For most academic programs and academic institutions, it is not a question of if they should do assessment, but just a question of how, how soon, and how effectively assessment can be implemented. But what exactly is assessment? For most purposes, it is a system of practices to assess and improve an endeavor. In this paper, we will restrict our attention to assessment in the context of academic accreditation agencies. For computer science programs in the United States, the Computing Accreditation Commission (CAC) of ABET, Inc. lists several major accreditation criteria, which includes Objectives and Assessment as the first one. Regional accreditation agencies (e.g., Southern Association of Schools and Colleges) are also moving in that direction. The major elements of assessment can be summarized succinctly as, “Say what you do; Do what
you say; Prove it; Improve it.” This slogan is not new, and is well-known, especially in the quality assurance arena. Generally speaking, assessment starts with documenting what needs to be assessed or measured, e.g., a set of objectives, refined into or mapped onto a set of desirable levels of measurable outcomes at appropriate levels (the Say what you do step). Then, the actual process in place to meet the objectives is described in detail (the Do what you say step). This process should also include subprocesses to measure the outcomes, so it can be verified that the desired levels of outcomes are achieved to indicate that the objectives are being accomplished. (This is the Prove it step.) If some objectives are not met, the processes used to meet the objectives in question need to be improved based on any deficiencies noted; sometimes the objectives may be revised in the interest of continuous improvement. (This is the Improve it step.) The cyclic nature of this process and the feedback used in continuous improvement, with potential improvements identified, implemented, and assessed for effectiveness, are often referred to as “closing the loop”. An important reason why assessment has assumed an increasingly important role in the context of accreditation is that the long term of accreditation requires not only that the program meet the requirements initially, but that its future quality be assured, as well. A set of assessment and continuous improvement processes represents just such a selfsustaining system. It is perhaps possible to maintain a high quality academic program in the absence of any formal assessment, and indeed many programs have relied for a long time on informal and anecdotal means for improvement. However, a well-defined assessment process can be very helpful in collecting data and analyzing them in a routine manner, and in identifying very specific areas where efforts can be focused for best improvements. Such a process is also easier to sustain. ISSUES IN ASSESSMENT Assessment is a major task, involving a significant amount of work. It is very important for accountability and for continuous quality improvement, however. The major issues in performing assessment are usually in the Prove it step described above. There are two major categories of issues: The first involved in making sure the measurements put in place cover all outcomes and all objectives, and the second involved in making sure the volume of data collected during the measurement phase is manageable. We can refer to the first category of issues as defining the assessment process, and
San Juan, PR
July 23 – 28, 2006 9th International Conference on Engineering Education T4A-1
Session T4A the second category of issues as implementing the assessment process. Defining the Assessment Process In order to prove that the processes put into practice actually accomplish the stated objectives, the objectives must be stated in such a way that the data collected in the measurement process actually cover all the stated objectives. Often this is done in multiple steps: First the objectives are stated as accurately as possible to reflect the mission of the program, making sure that they are consistent with the mission of the larger entities the program is a part of. Then, specific measurable outcomes are identified that relate to the objectives. Often, accreditation criteria, e.g., those of the Computing Accreditation Commission of ABET, specifically stipulate that specific student outcomes must be identified by the program. At this point, specific measurement/assessment methods are identified for measuring each of the outcomes. Depending on the method, sometimes it may be important to define some performance indicators that indicate that an outcome is met at the desired level. There are different ways to make sure all student-oriented objectives are covered by the measurement methods put in place. The major issue here is to choose a system that keeps the number of combinations somewhat contained, so that the different measurement methods that are required is still a manageable number. It is also often recommended that multiple different assessment methods be used, so as to provide a degree of redundancy (sometimes referred to as triangulation), making this issue even more challenging. Implementing the Assessment Process Once the process is sufficiently defined so that all specific assessment methods are identified, the methods need to be applied, data collected and analyzed. A major issue here is the management of the assessment methods and the sheer volume of data that result from the assessment methods. Many programs, especially at smaller colleges and universities, find it very hard to take on the challenge of such an assessment process, and sustain it. In institutions of higher education, there are generally many demands on the faculty members besides teaching, such as scholarly activities, including research, publications, and professional development in the face of a rapidly developing field, as well as professional service. Most faculty members love to improve student outcomes through individual efforts in their courses and collective efforts in refining the curriculum, infrastructure, pedagogy, etc. However, they can be reluctant to take on additional responsibilities relating to assessment, such as data collection and analysis. The value of such new practices would generally not be evident for some time in the future, which does not help provide the necessary motivation. Further, the fact that many departments have always striven for improvement even before the advent of formal assessment, often somewhat successfully, makes the cost of formal assessment appear unreasonably high.
Another important consideration is that assessment methods that are common and relatively easy to implement do not generally provide information that is of sufficient value for the purpose of outcomes-based assessment. Almost all programs administer student evaluations of faculty/courses, and many programs administer a variety of surveys. The problem with using these tools for assessment is that they often provide only indirect measurements (e.g., opinions of students, alumni, and so on) rather than direct measurements of program outcomes. This consideration is easy to overlook, while mountains of survey data are collected. Therefore, the major challenge in implementing assessment is in identifying assessment methods that are practical and direct relative to program outcomes. The additional work involved in putting them in practice should be minimal. An important purpose of assessment in the first place is to ensure that a program is not just of high quality, but that the quality will be sustained by the processes in place. It goes without saying that assessment methods that add a significant amount of additional work to the faculty work load are not sustainable, defeating the very purpose of assessment. TOWARD PRACTICAL ASSESSMENT In this section, we address the issues raised in the previous section and identify practical techniques we have put in place for our assessment. Along the way, we describe some of the considerations that have gone into the choices we have made. The rest of this section will be organized parallel to the previous section. Defining the Assessment Process The first approach we considered to define our assessment process is a hierarchical one. In this approach, we started with several objectives, and for each objective, attempted to identify a small number of student outcomes, which, when verified at a desired level, would indicate that the associated objective has been accomplished. For each such student outcome, some performance indicators were identified that would indicate that the outcome is met at a desired level of competence. Finally, each performance indicator could be measured in one or more ways using different measurements. This approach may be visualized in a tree-like structure, where the nodes at the first level correspond to objectives, and the children nodes of each objective node correspond to student outcomes, the descendent nodes at the next level are the performance indicators, and finally, we would have the actual measurement methods. The number of nodes in such a tree grows exponentially by the level, and can become extremely large. We also noted that the structure is actually not a strict hierarchy in the sense of single reporting relationships generally assumed in a hierarchical structure (or a tree, for computer scientists, because parent nodes often share the same children nodes). In fact, we have noticed that it is much more appropriate, and is indeed more common, to use a matrix structure to map objectives to outcomes, etc. An example is in Table I, which shows two mappings in one table. The portion on the right
San Juan, PR
July 23 – 28, 2006 9th International Conference on Engineering Education T4A-2
Session T4A starting from and including the bolded column shows mapping from the objectives to student outcomes. The portion on the left starting with and including the bolded column shows the mapping between outcomes and assessment methods (see next subsection for more on these assessment methods). An X in a column indicates that the objective is associated with the outcome or that the assessment method can measure the student outcome. It is important to note that no row and no column are empty. TABLE I OBJECTIVE-OUTCOME/OUTCOME-METHOD MAPPINGS Ob1=Grounding in key principles and practices Ob2=Understanding of ethical aspects, … Out1=Understanding and problem solving through artifacts Out2=Use and employ data structures and algorithms … Out6=Understanding of social, professional, and ethical issues Out7=Skills and knowledge to be employable M1=Faculty Course Assessment Reports (FCARs) M2=Capstone evaluations by Industry Advisory Board (IAB) M3=Senior Surveys M1
M2
M3
X
X
Out1
X
X
X
X
Out2
X
X
Out3
X
X
X
X
Ob1
X
X
X
Out4
X
X
X
Out5
X
Out6
X
Out7
X X
Ob2
X
Ob3
Ob4
X
X
X
X
X
X
X
X
X
X
X
We discuss the assessment methods in the next subsection. However, it should be mentioned that we had two additional objectives, which we consider to be curriculumoriented, as opposed to the afore-mentioned objectives, which were all student-oriented. The curriculum-oriented objectives (e.g., the curriculum meets national standards) were viewed as directly measurable without being mapped onto student outcomes, and it was made sure that they were also covered by the stated assessment methods. It may be noted that the first two methods (M1=FCARs and M2=IAB capstone evaluations) were, in and of themselves, adequate to cover all the outcomes. They were also adequate to cover the curriculum objectives, as well, and therefore, were designated to be the primary assessment methods. Triangulation was built into the assessment process by including several other supplementary assessment methods, one of which was shown as M3 (Senior surveys) in Table I. These supplementary methods provide a useful degree of redundancy, but are not discussed further. Implementing the Assessment Process
The major implementation issue is the practicality of the assessment methods. The focus should be on minimizing the extra work involved in administering the assessment methods, collecting a large volume of data, and analyzing it. We favored a distributed approach, where the work can be distributed as evenly as possible among all faculty members. Other important issues relate to the directness of the methods in measuring the objectives/outcomes and to the cost of implementing formal assessment. With these goals in mind, the two primary assessment methods identified in the previous subsection have been chosen not just because they cover all the outcomes and objectives, but because they are practical, as well. Data are collected, analyzed, and indicated improvements identified and implemented periodically, based on these (and other) methods. We discuss these two methods below. •
Faculty Course Assessment Reports (FCARs):
A Faculty Course Assessment Report is a multi-purpose document adopted from the work of Estell [3]. Although it is represented as a single column in Table I, it truly should be represented as many different columns, each column corresponding roughly to one course offering each semester. (For practical purposes, a single FCAR is usually used for any course-instructor combination.) Thus each FCAR can cover different student outcomes, and an X in a cell in the M1 column of Table I simply says that some FCAR covers the outcome corresponding to the cell (in a disjunctive sense). We have also mapped the courses in our curriculum to each of the seven student outcomes mentioned in Table I. This table is not shown here in the interest of space. When a certain course is mapped on to certain student outcomes, each FCAR for that course measures the extent to which the outcomes are met every time the course is offered. (Each instructor teaching any number of sections of the same course in a semester produces one FCAR.) Once again, we make sure that each student outcome is covered by some course each student will have to take. The details of how the measurements are made may be outlined as follows: An FCAR has several useful sections in it, including the specific course outcomes and program outcomes the course contributes to. In addition, there are sections on assessment documentation (which is most important for our present purpose and detailed a little further below), proposed actions for further improvement based on the FCAR, and any actions taken as a result of past FCARs. The assessment portion of the FCAR deals with course outcomes and program outcomes. Course outcomes are specific to the course in question, while program outcomes are longitudinal across the curriculum. John Estell, the original proponent of FCARs, suggests slightly different “attitudes” in measuring the course outcomes and the program outcomes. The course outcomes are measured for each student taking the course on a 3-point scale (excellent, effective, and marginal, with a score of 0 for unsatisfactory). The program outcomes are also measured on a 3-point scale (exemplary, proficient,
San Juan, PR
July 23 – 28, 2006 9th International Conference on Engineering Education T4A-3
Session T4A and apprentice, with a score of 0 for novice). The terms used to describe the latter scale are more appropriate for longitudinal analysis. For instance, if the same program outcome (e.g., use and employ data structures and algorithms) were measured in an early course (e.g., a course corresponding to CS1) and later in an advanced course (e.g., CS Capstone), the evolution of a particular student and, more importantly, that of any group of students should be evident. The exact measurement of each outcome is controlled by the performance indicators the instructor identifies in each particular course offering, e.g., a certain question on a certain test and/or a component of a certain project, etc. A score on the afore-mentioned scales is computed for each student in a course from on a weighted formula the instructor defines, based on the importance of each performance indicator for each outcome. Ultimately, a single score is computed for each outcome by averaging the scores of all students. (Withdrawn students are excluded, but all others are included in this average.) The scores on the course outcomes are of more immediate relevance to the instructor on where (s)he needs to focus her pedagogical approach. The scores on the program outcomes are periodically analyzed by the respective curriculum groups to assess their impact on the curriculum as a whole. We provide a concrete example. One of the outcomes, Out6, is that students demonstrate an understanding of social, professional, and ethical issues. This outcome is measured using FCARs (method M1) in at least two courses, SWE 2642 – Professional Practices and Ethics, and SWE 4624 – Software Engineering. In his FCAR, the instructor of SWE 2642 in the Summer 2005 semester claims to have assessed this desired outcome in all assignments and examinations, whereas in her FCAR, the instructor of SWE 4624 in the Fall 2004 semester claims to have assessed this outcome in specific questions on the midterm and the final examinations. Each student’s performance on the specific criteria is rated on a 4point scale somewhat differently in the two FCARs, and this outcome is assessed by averaging the students’ performance. The attractiveness of FCARs as a very important assessment method may be attributed to several factors: First, the effort in implementing FCARs is truly distributed among all faculty teaching the courses, and thus adds only a small incremental degree of additional work. It has been estimated that completing the FCARs may be limited to one day during the final exam grading period, although the planning for them, especially the effort involved in identifying the performance criteria specific to each course, and the effort involved in collecting and documenting all relevant data is distributed as a small additional factor throughout the semester. Second, they represent a true grass-roots level approach and, as indicated above, have utility both to the instructor (course outcomes) and to the program as a whole (program outcomes). In other words, they can provide both formative and summative measures of the outcomes. Third, the involvement of the entire faculty makes the process more robust, adds a sense of community, and also helps evolve the instrument for the better. Faculty members learn from each other and as they see
the benefits of FCARs to their own teaching, the “buy in” becomes easier. Finally, from the viewpoint of the quality of assessment, there is yet another very significant benefit. We have mentioned that FCARs contain sections on proposed actions for further improvement based on the FCAR, and any actions taken as a result of past FCARs. Taken together, they make the “closing of the loop” remarkably well-documented – an important step in establishing that the results of assessment process are used for continuous improvement. •
IAB Capstone Evaluations:
The one course that goes through a substantial additional scrutiny beyond the FCAR is the Capstone course in our curriculum. The course is evaluated each Fall and Spring semester by the industry advisory board (IAB) of the department. The IAB evaluations cover all of our program outcomes, except one (namely, describe and explain concepts in operating systems, programming languages, architecture, distributed computation, etc). In particular, they cover student outcomes related to oral and written communication, team work, employability, understanding of social and ethical aspects of computing, foundations of problem solving, using and employing data structures and algorithms. As a minimum, the IAB members are asked to express interest in specific student projects that they want to evaluate, and are sent reports and other data as the projects evolve. The Capstone instructor makes sure that each project is covered by more than one IAB member. On the day set aside for the IAB meeting, at the end of the semester, the IAB members review presentations and demos by each capstone project team and have an opportunity to ask questions. They are given “grading rubrics” and surveys to cover all the necessary data sought after. Their evaluations are summarized and reviewed by the instructor. They are periodically analyzed by the curriculum group. IAB evaluations of capstone projects, as well as their feedback on the program as a whole, are significant because of their summative nature. In this sense, FCARs and IAB evaluations complement and supplement each other. We give a concrete example of assessing an outcome through the IAB method (method M2). One of the outcomes, Out7, is that students demonstrate skills and knowledge to be employable. This outcome is specifically framed as a question on the IAB Capstone project evaluation form. Each IAB member is asked to rate the achievement of this outcome on a 10-point scale, and is asked to give reasons supporting his/her assessment. The factors IAB members use in this assessment include a student’s knowledge and depth of understanding, communication skills, process maturity, avoidance of “reinventing the wheel”, as well as less tangible factors such as the need for good mentors. IAB members have occasionally recruited a capstone student right after the presentations, and we believe their rating provides a valuable assessment of the outcome. When we first began asking the IAB members to evaluate capstone projects, we were skeptical both of the degree of their interest in helping us and of the value of the evaluations.
San Juan, PR
July 23 – 28, 2006 9th International Conference on Engineering Education T4A-4
Session T4A It was an extremely pleasant surprise to us that the members were deeply and genuinely interested in getting involved. The faculty members were impressed to see the “academic side” of the members of the industry! In fact, the involvement of the IAB in the affairs of the department, including and going beyond the capstone evaluations, was much more meaningful to both sides compared to the previous generation of IAB meetings, which used to serve mostly for information exchange, some feedback/suggestions, and public relations. It is easy to limit the cost of organizing IAB meetings to be within the means of most academic departments. ONTO AN INTEGRATED APPROACH Our experience suggests that the particular choice of elements we have put together in our assessment process is an excellent starting point for assessment in general. We feel it has been very effective in our program assessment. It has also proved to be practical from the viewpoint of faculty workload. Regional accreditation agencies, such as Southern Association of Schools and Colleges, have been moving in a broader direction of assessment. Certainly, a program, such as ours, that has put together an assessment process in place is well-prepared for the next step. However, regional accreditation and assessment cuts across all programs, including the core curriculum of an institution, and therefore, institutional support and commitment are of particular significance. There are efficiencies to be gained from an institution-wide implementation, not just limited to a new support infrastructure. Automatic or semi-automatic approaches / tools for data collection and documentation can be very helpful, as are identification of classes of data common to all programs, e.g., exit surveys, employer surveys, employment data, etc. CONCLUSIONS As pointed out before, one reason the FCAR-based assessment is practical is because no single individual needs to do the lion’s share of the work. The work is distributed in a meaningful way among all teaching faculty members. On the flip side, the need for an initial training phase should be highlighted. Without such training and standardization,
faculty members could produce FCARs that are inconsistent with each other and fail in their intended purpose. An important element that we have left for the instructor’s discretion in the initial implementation is the identification of performance indicators for each outcome measured in a course. Most instructors are guided either directly or indirectly by the well-known Bloom’s taxonomy in this process. However, we feel that in the future, criteria could be developed to incorporate Bloom’s taxonomy explicitly into the performance criteria. Interestingly, another recent work refers to this aspect [4]. The authors also mention the use of course portfolios for assessment and acknowledge that they were the weak link in their process that they have since attempted to rectify. It appears that in their course portfolios, they were missing the kind of detail and specific information FCARs would provide, in spite of the large amount of other data. We have experimented a little in automating some aspects of our assessment process. A course database has been developed that can be modified by course coordinators, which lists course outcomes, etc. and may be used to generate FCAR templates. A series of capstone projects have been defined to automate support for generating FCARs, but are not ready for testing by the faculty. Right now, the computation of scores in FCARs is accomplished through spreadsheet support. ACKNOWLEDGMENT The author thanks the faculty of the School of Computing and Software Engineering, with a special acknowledgement to Professor Briana Morrison, for their help with various phases of implementing, and in some cases, defining the assessment system at his university. REFERENCES [1]
Cooper, S., Cassel, L., Cunningham, S., and Moskal, B., “OutcomesBased Computer Science Education”, SIGCSE 2005, St. Louis, MO (February 23-27, 2005), pp. 260-261.
[2]
Estell, J. K., “The Faculty Course Assessment Report”, 33rd ASEE/IEEE Frontiers in Education Conference, Boulder, CO (November 5-8, 2003), T4B-8. (Also in Best Assessment Processes VI, A working Symposium, Rose-Hulman Institute of Technology, (March 1-2, 2004).)
[3]
Abunawass, A., Lloyd, W., and Rudolph, E., “COMPASS – A CS Program Assessment Project”, ITiCSE 2004, Leeds, UK (June 28-30, 2004), pp. 127-131.
San Juan, PR
July 23 – 28, 2006 9th International Conference on Engineering Education T4A-5
