Assessment Handbook: Methods of Assessment

Note: This section includes guidelines on using the most common assessment methods.


Skidmore Departments Using Direct Assessment Methods

Skidmore Departments Using Indirect Methods

• Capstone Courses
• Capstone Course for Assessment of Learning in the Sociology Major,
  Article by Kate Berheide
• Portfolio
• Portfolio Assessment
  "Portfolio Assessment: Benefits, Issues of Implementation..."
  "Portfolio Structure and Student Profiles"
• The Idea Notebook, Michael Marx and Kate Berheide
• Varieties of Embedded Assessments
• Pre- and Post-Assessment, or Value Added Assessment
• Student Performances
• Standardized Tests
• Indirect Assessment Methods
• Survey Research
• Inventories
• Sample Inventory from Sociology Program
• Model of course inventory spreadsheet
• Scoring Rubrics

Methods of Assessment

Capstone Courses Definition: A capstone course is a course designed to be offered in the final semester of a student’s major, a course that ties together the key learning objectives that faculty expect the student to have learned during the major, interdisciplinary program, or interdepartmental major. The faculty member who teaches the course gives the grade for the students; the program faculty or a sub-group of the faculty review and evaluate the work for assessment purposes. Advantages: Capstone courses enable: Faculty to assess the cumulative abilities of students within the context of one course; Faculty to develop the assessment materials to be evaluated within the context of a course; Students to produce work to be assessed as they would produce work for any course; Students to demonstrate how they can integrate the knowledge, abilities, and values that faculty have been teaching or demonstrating. Disadvantages: The capstone course: May not allow enough time for students to devote enough time and effort to truly comprehensive projects; May not produce the data faculty need if the exercises or projects are not directly linked to the program learning outcomes and if the faculty teaching the course can do not require what the program faculty have agreed upon; May only provide time for students to address the major program outcomes and therefore not allow faculty to assess more detailed learning outcomes or sub-outcomes. Varieties of Capstone Courses: The major project course: The major project course requires students to work on one project primarily, such as a research paper or an experiment or a creative project. The course can be designed so that students work on the project in stages, allowing faculty to determine students’ abilities to revise and/or reconceptualize their work. Student presentations of the project may be both written and oral, allowing faculty to assess both of those student abilities in addition to knowledge and/or skills. The multiple experiences or exercises course: Faculty design the course so that students must provide evidence through a variety of means, such as examinations, research papers, oral presentations, group work, and multimedia presentations. For assessment purposes, the faculty need to determine who and how they will assess each of the types of assignments. If the faculty member teaching the course is the only faculty member to evaluate all of the work, then the faculty must rely upon that person’s judgments regarding the implications for the entire academic program. The portfolio in the capstone course: The major project for a capstone course may be a requirement that students produce a portfolio of work that then provides one item that the program faculty can assess. This portfolio can be designed so that students include a variety of evidence regarding their abilities. The major limitation is that students may not have access to work that they produced in earlier courses, and so the portfolio may be limited as a document for assessing the entire program. The field experience or internship as a capstone course: A number of academic programs require students to fulfill a field experience or internship experience as the culminating activity in the program. In this case, students can demonstrate their knowledge, skills, and values in a wide variety of ways. Field experiences and internships may be evaluated by both a faculty member and a field supervisor under whom the student is working. Evaluations may consist of check sheets and evaluation forms that the supervisor and faculty complete both during and at the end of the experience, notes from advisory meetings with the student during the experience, and materials that the student produces during the experience, perhaps gathered into a portfolio. Faculty overseeing the field experiences will need to share their observations of student strengths and weaknesses with the other faculty for them to discuss and assess. Creating and Designing a Capstone Course: Determine the specific broad learning objectives for the academic program; If you have not already done so, determine how those are translated into the individual courses; Determine the kinds of student work that should be expected during the capstone course (the content and performance standards); Design the capstone course to enable students to produce that work; Determine how and when the faculty will assess the work that students produce; Inform students in the syllabus or related handouts how the objectives of the course are designed to reflect a culmination of their abilities, knowledge, and/or values. Provide information in department assessment plans and other documents to be reviewed by various constituents regarding how the course relates to the standards for the program and how they are evaluated. Evaluating the Work from a Capstone Course: How you evaluate or assess the work student produce in a capstone course depends upon the nature of the work and the learning outcomes being assessed. For example, you may want to use a rubric to evaluate major projects or portfolios. If more than one faculty member observes the student work, you may want to develop checklists or key questions which can be used to describe what the faculty observe. If examinations are part of the course, there may be a few key questions embedded in the exams that will be used for the assessment purposes. Some departments have professionals in the field observe and evaluate student work, such as business persons evaluating case study presentations, directors and actors evaluating student stage productions, artists and musicians evaluating student art work and musical performances, or teachers evaluating student teachers. The key step is that the faculty as a whole must have a chance to take part in assessing student work or review the assessment results if a designated sub-group of the faculty or professionals in the field assess the work. This review, discussion, and determination if anything needs to be revised in the curriculum needs to be scheduled as part of the regular schedule for faculty work, including time for any recommendations for change to be submitted to appropriate curriculum committees or administrators. Often, some real surprises result from a faculty’s assessment effort: these may, in turn, lead to modifications of a future assessment, such as focusing upon a specific question that the faculty are concerned about.

Using the Capstone Course for Assessment of Learning in the Sociology Major Catherine White Berheide Skidmore College* As assessment initiatives become more widespread, higher education is increasingly moving beyond externally developed standardized tests to assess student learning in majors. Senior capstone projects are often proposed as qualitative instruments for assessing student learning because their greater flexibility adapts better to the unique circumstances of each department and because they reveal actual student performance. Capstone projects, therefore, can be a vehicle for assessing how well students have met the goals of the sociology major. For example, is the student able to formulate empirical research questions? To describe important theories in a substantive area? To summarize current research in that area? To generalize appropriately? This paper looks at how capstone courses are being used to assess student learning in the sociology major. To be useful in assessing the major curriculum, capstone projects must be amenable to comprehensive group level assessment. Disciplines, such as sociology, need models for how to use senior research projects to evaluate the department’s success in fulfilling program goals. The fundamental question is what are the criteria upon which these senior projects should be evaluated? How are the goals of a sociology major translated into a set of criteria that can be used to evaluate student projects to assess the program’s success in meeting its learning goals for the major? This paper addresses the question of how to assess the effectiveness of academic majors, particularly the sociology major. Assessment in Higher Education Currently within higher education, the mandate to engage in assessment, particularly for accreditation, refers to assessing student learning outcomes. Assessment, however, is not limited to the individual student level; it can also occur at the classroom and the program level. Faculty resist the mandate to engage in program assessment for a variety of reasons. Faculty may be suspicious of the way administrators and other authorities might use assessment. Pratto (1996:122) concludes that “assessment for purposes of improvement that is summative and carried out with the direct involvement of the individual being assessed is the least susceptible to abuse.” Faculty may also be reluctant to add an additional time-consuming obligation to their workload, especially one for which they are not rewarded. Perhaps more importantly, though, faculty may resist program assessment initiatives because they are skeptical about the assessment techniques being used (Hartmann 1992). In response to the demand for accountability, particularly from state legislatures, higher education has tended to rely on quantitative assessment tools that allow national norming, such as standardized tests. Cross and Steadman (1996) as well as Palomba and Banta (1999) reflect the positions of many faculty when they criticize these tests for failing to measure the full range of learning that goes on in college. The second of the American Association of Higher Education’s (2001) “9 Principles of Good Practice for Assessing Student Education” states that, “assessment is most effective when it reflects an understanding of learning as multidimensional, integrated, and revealed in performance over time.” To gain a more comprehensive picture of the outcomes of higher education, the AAHE as well as Palomba and Banta call for using a diversity of methods, so that schools do not simply assess those aspects of a college education that are easy to measure quantitatively. This critique of quantitative techniques does not mean, however, that they have no place in a department’s assessment plan. Quantitative research methods have significant advantages. They can be high on reliability and validity. They can allow for national norming and benchmark comparisons. They can allow schools, programs, or students to be compared. In addition to standardized tests either nationally or locally designed, commonly used quantitative techniques include surveys of seniors and alumni as well as analysis of measures of student success, such as grade point averages, graduation rates, percent going on to graduate school, etc. Institutions often adopt standardized tests to measure student learning outcomes to take advantage of these strengths of quantitative methods as well as the fact that they do not require as much faculty time as other assessment tools, especially if they use a pre-existing national test, such as an ETS Major Fields Test. Such tests allow students to demonstrate what they know in their major field in comparison to a national standard. The problem is that the major at a particular school may not be designed to cover the same areas as the Major Fields Test, for example, covers. Thus the test may not accurately measure what students have learned. Even worse, the national test may lead some schools to “teach to the test.” Others may decide to develop their own test to measure their departmental learning goals for majors more accurately. This option, though, reduces the likelihood of national comparability and increases the time faculty have to spend on assessment initiatives. To gain a more comprehensive picture of student learning, some programs have turned to qualitative techniques that can provide a richer set of data than quantitative ones. As is often true of qualitative research methods, they may have less reliability than quantitative methods, such as standardized tests, but they often have greater validity. Qualitative techniques, like quantitative ones, require that the academic program define its goals and objectives and then identify appropriate methods for measuring whether those goals and objectives have been achieved. Qualitative techniques commonly used in higher education for program assessment include exit interviews, focus groups, reflective essays, synthesizing projects, comprehensive exams, and student portfolios. As long ago as 1992, Hartmann proposed that sociology departments use the bachelor’s paper, that is a synthesizing project, to assess the program as a whole. The bachelor’s paper is simply the student’s attempt to use the theoretical and methodological tools of his or her discipline to address a substantively important topic. The resultant paper should demonstrate the student’s ability to meet the department’s standards for a competent piece of sociological research. The body of papers produced should allow departments to identify curricular areas in need of reform. (Hartmann 1992:125) In short, he argues that the performance of sociology programs ought to be measured by their student’s ability to do sociology and that that is best done through a capstone paper. The Capstone Course in Sociology The Association of American Colleges report, The Challenge of Connected Learning, and those in the companion volume, Reports from the Fields, note that a key element in study in depth is some type of integrative capstone experience. The second volume (1991b), Reports from the Fields, presents the reports of the thirteen disciplinary task forces on study in depth in undergraduate majors. The fourth recommendation in the expanded version of the sociology report, Liberal Learning and the Sociology Major (Eberts, Howery, Berheide, Crittenden, Davis, Gamson, and Wagenaar 1991) is that sociology majors should have at least four levels in a sequence of courses rather than a “ferris wheel” where an introductory sociology course is the most students need for access to the rest of the sociology curriculum. The four levels the sociology taskforce (Eberts et al. 1991:15-16) recommended are Introductory courses Required courses in methods, statistics, and theory as well as those substantive courses that do not require a background in methods and theory Advanced substantive courses that require prior exposure to theory and methods Capstone courses that ask students to synthesize their previous work in the major. The sociology task force’s “analysis of 86 catalogues showed remarkably high consensus on an introductory course, one or more methods and statistics courses, and one or more theory courses” as the requirements for a sociology major nationally (Eberts et al. 1991:8). In contrast, Kain (1999) found that of f the 36 colleges and universities with sociology majors he studied, only one-fifth required a capstone course; only half of those courses involve research training, thereby building on earlier coursework in methods and statistics. To be a capstone, a course must require students to integrate their substantive work in sociology with their required courses, particularly research methods and sociological theory. While the nature of capstone courses varies nationally, three approaches are particularly common. In a research seminar, students are exposed to advanced methods and theory while pulling all their previous coursework in sociology -- statistics, methods, theory, and substantive fields -- together into a culminating piece of scholarship, such as a bachelor’s paper or a thesis. In an internship seminar, students discuss how their sociological education, including methods and theory, applies to their internship and reveals social patterns across their placements. In the overview seminar, students engage in a systematic review of the discipline, consisting of various reading and writing assignments to integrate, critique, and apply sociology, with an emphasis on methods and theory. This type of course could help students prepare for a comprehensive examination whether oral or written, whether a nationally-normed standardized test or a locally designed one. (See Eberts et al. 1991 and Wagenaar 1997 for further discussion of capstone courses in sociology.) While the capstone can take a variety of forms, graduating seniors benefit from building a learning community as they discuss the common issues arising from independent research projects or internships. Therefore the capstone should involve a senior seminar in addition to whatever independent work might be required of students. Such a capstone is a critical ending point for a major carefully constructed not only to expose students to the discrete aspects of sociology as a discipline, but also to give them the opportunity to demonstrate their in-depth knowledge of the field. Some Advantages of Using a Capstone Paper for Program Assessment The synthesizing paper that students write in a research-based capstone course requires them to define a research problem, create and implement a research design, and analyze and interpret data. The advantage of such papers for assessment is that they take students through the process of doing research. The student gains additional knowledge and skill from the assessment experience while the department gains information it needs to improve its major. Hartmann (1992:126) argues that the “additional learning experience for the student” is one of the eight advantages of using the bachelor’s paper as an assessment method. I adapted the following additional advantages for using a capstone project for assessing the sociology program from Hartmann’s list. The paper is a direct assessment of whether the student can do sociology akin to the art show or music recital as assessments of whether students can do art or music. The paper requires students to demonstrate a more sophisticated understanding of theory and its relationship to research than can typically be demonstrated on a test. The paper allows for original thought and for individually tailoring the project to the student’s interests and the department’s strengths. The paper requires all students to follow a minimum standard format (e.g., problem statement, literature review, theoretical framework, conclusion), allowing comparisons across students and conclusions about the overall program on these dimensions at least. The paper gives students an additional learning experience in writing and perhaps in oral presentations as well, both of which are important job skills. When the paper is written in the context of a capstone seminar, other activities, such as career planning, can be delivered to senior majors. “Because the papers do not produce a neat nationally comparable score they are less likely to be misused than other assessment devices.” (Hartmann 1992:127) The latter, of course, is also one of the main weaknesses of using a capstone project for assessment. Without such scores they are unlikely to satisfy the external stakeholders who want quantitative nationally normed data with which to hold higher education accountable. Using content and performance standards, however, faculty can rate capstone papers using quantitative scales. According to Heywood (1989), using more than a single evaluator and a single scale can increase the reliability of senior projects as assessment tools. (See Heywood for examples of scales that can be used to assess papers and projects in a variety of disciplines.) Hartmann (1992:127) argues that, “the key is that efforts to improve the reliability of assessment should be based in what is first of all a valid assessment project, such as the bachelor’s project.” He goes on to warn that, “the urge to quantify assessment beyond what can reasonably be required for reliability should be resisted.” Six Steps in Using a Capstone Project for Assessing the Major The first step in using a capstone project for program assessment is to identify the program’s goals and objectives (the third principle of good practice for assessing student learning, according to AAHE). Assessment involves measuring performance against goals. The literature on higher education contains lengthy discussions of goals (e.g., Bowen 1997, Ehrlich 2000, and Young 1997). Wagenaar (1991:93), for example, specifically develops ten goals for the undergraduate sociology major that he presents as a starting point for departments to use in developing their own. He rejects the position that “sociology is too fragmented a discipline, and there are too many sociologies to list a comprehensive set of goals.” In the same volume of Teaching Sociology, Hazzard (1991) offers a set of learning goals for introductory, intermediate, and advanced courses. Departments need learning goals for students majoring in their discipline. Rather than stated as generalities, each goal a department adopts needs to specify the "such that" statements that would indicate specifically how achievement of each objective can be assessed. For example, if a department adopts a goal on sociological research methods, exactly what must the student demonstrate to show compliance with the objective? Once a department has agreed upon a set of learning goals, it must then use them to inform the content of the curriculum and the major. Does the curriculum specifically reflect these objectives in how it is organized and taught? How should the department organize the major(s) and teach each course so that students achieve its learning goals? Are students made aware of the basic goals and how each course connects to them? Once the department has answered these questions collectively, faculty should include learning objectives for their courses on their syllabi, showing students how these courses contribute to the department's overall goals. Several departments have created matrices of department goals by courses that indicated which courses work on achieving which goals. (See Appendix A for an example of a Goals by Courses Matrix.) Next, these goals need to be translated into content standards and performance standards: that is the skills and knowledge the student should achieve and the level at which the student should demonstrate them (Glatthorn et al. 1998). For example, one of my department’s content standards is that, “The sociology major should be able to describe major theories in selected substantive areas of sociology.” California State University at Sacramento (see Dean Dorn’s piece, An Electronic Assessment Portfolio at California State University – Sacramento, in this volume) uses a very simple set of performance categories for assessing papers in the portfolios of sociology majors: What do sociology majors do that is outstanding? What do sociology majors do that is satisfactory? What do sociology majors do that needs improvement? These content and performance standards set the benchmarks for the program as well as the individual student. These standards should be developed into rubrics for measuring how well students demonstrate that they have met the goals of the major. (See the webpage for the College of DuPage Outcomes Assessment Committee for a rubric developed by the sociology faculty at the College of DuPage.) Once standards are developed, the department selects individuals to review the capstone projects. These multiple evaluators need to look for patterns of strengths and weaknesses across all the papers (or a sample of papers). They need to report the curricular implications of these patterns to the sociology program faculty. As Hartmann (1992:126) notes, “if a cohort of students is not meeting expectations with regard to theory, research skills, writing, policy relevance, or whatever else the department has agreed is important, curricular reform is in order.” The final step, then, is to use the data collected about the performance of the program to provide feedback to improve the major. For example, after my own department surveyed seniors to ask them to assess their sociology major as part of the AAC’s study in depth project (1991b), the students’ comments floored the faculty. The seniors said that the department ought to require statistics for the sociology major. As a result, we instituted a statistics requirement in addition to the methods requirement that already existed. Assessment, therefore, is not an end in and of itself, but rather a means to an end. The end is the improvement of learning at the individual, program, and institutional levels. How Programs have Used Capstone Papers to Assess Sociology Majors This approach to program assessment is labor-intensive for faculty as well as students. Some schools are committed to the capstone project to assess student performance but do not take the next step to use it for program assessment. The papers themselves, like the student work in a senior art show, can be displayed to external audiences with a flourish so as to say, “Ta da! Look at what our students can do.” These displays can be very impressive indeed, and while they may be “an unwieldy basis for external assessment, they provide the most direct and most unfiltered picture of students’ capabilities” and, by extension, of program effectiveness (Hartmann 1992:128). Some schools, including my own, go a step further and submit some of the papers for presentation at regional sociology meetings and for undergraduate paper contests. Success in getting senior projects accepted for presentation at professional meetings or in winning undergraduate prizes provides external validation of the quality of student performance and by extension of program effectiveness. Others use external evaluators to “grade” the papers and the programs. This approach is the “thumbs up, thumbs down” approach to evaluating the capstone projects akin to a juried art show. Best practice, though, involves going a step further, if only a baby step, to analyze the papers systematically for the evidence they provide about program quality and then to use that evidence to make curricular improvements. One step, albeit an informal and somewhat nonsystematic one, involves the “water-cooler conversation,” the casual comments that the faculty teaching the capstone course make to their colleagues about what the students seem well prepared or ill prepared to do. For example, I have complained for years about the students’ inability to define a sociological research question, let alone a testable hypothesis. That feedback has led the instructors in lower-level courses, including introductory sociology, to add assignments that give students multiple experiences in defining a sociological question and identifying hypotheses. Other departments, such as Valdosta State University, are taking more formal and systematic steps. The sociology department adopts a fairly holistic approach to assessing a sample of the capstone papers every other year. According to the chair of the department, a subset of faculty reading the papers consider the conceptual depth and clarity, use of resources, and the structure of the paper. James Madison University goes a step further. In the capstone course at James Madison, “the students integrate previous experience in the major by ‘doing sociology’ in the writing and presentation of a substantial paper displaying the student’s sociological skills – theoretical, methodological and practical.” Multiple readers use indicators derived from the department’s goals to look for patterns and trends in student performance on a sample of papers from various years and then use that information to revise the sociology curriculum. The readers rate each paper from on a five-point scale from fair to excellent on each indicator that is relevant for the paper. For example, one indicator is “interprets and uses social science data” (James Madison University Sociology Department). My own department has identified the program goals that performance on the capstone project can measure, concluding that six of our ten goals for sociology majors can be assessed through their senior seminar research projects (see Appendix A). Often departments choose to tackle assessing only one or two of these goals at most per year. The most elaborate approach, exemplified by California State University at Sacramento (again, see Dean Dorn’s piece, An Electronic Assessment Portfolio at California State University – Sacramento, in this volume), involves taking the rubric the department has developed for assessing how well the department’s goals for majors have been achieved and applying it to student papers. The department creates a collective department portfolio of student work by having faculty select a sample of student papers written for required courses as “examples of poor, average, and good to excellent work.” The department’s assessment committee then evaluates these papers to measure the performance of the sociology major as a whole. The department began by choosing to assess the success of the student papers in meeting only two of the program’s eleven goals. Based on the evaluation of student papers, the department developed a list of ways to improve students’ writing and their ability to apply sociological concepts (the two goals being evaluated). The department plans to analyze other goals in future years as well as to track trends in the strengths and weaknesses of student papers. Conclusion More and more, sociology departments are developing capstone courses. Some of them are beginning to use these capstones for program assessment. As of yet, though, only a handful have begun to evaluate systematically the students’ capstone products to see what they can tell the department about the quality of its program and to help identify what needs to be changed. Using capstone projects offers significant advantages as well as some significant disadvantages compared to using standardized tests to assess the quality of sociology programs. Sociology departments should see program assessment as a long-term process to be approached one step at a time. Each step taken, no matter how small, should lead to improvement. For example, adopting a set of goals for the sociology program requires faculty conversations that should improve the coherence of the major. In addition, developing a capstone course for the major if the department does not already have one will strengthen the major immensely. Sociology departments need to engage in systematic assessment of their majors to improve the quality of our students’ performance. The major papers students write in a capstone course provide a data-rich vehicle for doing program assessment References American Association for Higher Education. 2001. “9 Principles of Good Practice for Assessing Student Learning.” http://www.aahe.org/principl.htm, June 19, 2001. Association of American Colleges. 1991a. The Challenge of Connecting Learning: Liberal Learning and the Arts and Sciences Major, Volume One. Washington, DC: Association of American Colleges. _______ . 1991b. Reports from the Fields: Liberal Learning and the Arts and Sciences Major, Volume Two. Washington, DC: Association of American Colleges. Bowen, Howard R. 1997. Investment in Learning: The Individual and Social Value of American Higher Education. Baltimore, MD: The Johns Hopkins University Press. Cross, K. Patricia. and Mimi. H. Steadman. 1996. Classroom Research: Implementing the Scholarship of Teaching. San Francisco, CA: Jossey-Bass. Eberts, Paul, Carla Howery, Catherine White Berheide, Kathleen Crittenden, Robert Davis, Zelda Gamson, and , Theodore Wagenaar. 1991. Liberal Learning and the Sociology Major. Washington, DC: American Sociological Association. Ehrlich, Thomas, ed. 2000. Civic Responsibility and Higher Education. Phoenix, AZ: Oryx Press. Glatthorn, Allan A. with Don Bragaw, Karen Dawkins, and John Parker. 1998. Performance Assessment and Standards-Based Curricula. Larchmont, NY: Eye on Education. Hartmann, David J. 1992. “Program Assessment in Sociology: The Case for the Bachelor’s Paper.” Teaching Sociology 20:125-128. Hazzard, John. 1991. "Student Competencies and the Goals of the Undergraduate Curriculum: A Response to Theodore Wagenaar." Teaching Sociology 19:532-532. Heywood, John. 1989. Assessment in Higher Education. 2nd ed. New York: Wiley. James Madison University Sociology Department. nd. Sociology Program Goals/Indicators and Assessment Rate Form. Kain, Edward L. 1999. “Building the Sociological Imagination Through a Cumulative Curriculum: Professional Socialization in Sociology.” Teaching Sociology 27:1-16. Palomba, Catherine A. and Trudy W. Banta. 1999. Assessment Essentials: Planning, Implementing and Improving Assessment in Higher Education. San Francisco, CA: Jossey Bass. Pratto, David J. 1996. “Assessment Abuse by Design.” Teaching Sociology 24:119-122. Wagenaar, Theodore. 1991. "Goals for the Discipline?" Teaching Sociology 19:92-92. Wagenaar, Theodore, ed. 1997. Capstone Course in Sociology. Washington, DC: American Sociological Association. Young, Robert B. 1997. No Neutral Ground: Standing by the Values We Prize in Higher Education. San Francisco, CA: Jossey Bass. Sociology Department Web Sites California State University, Sacramento. http://www.csus.edu/psa/soc_v1/introduction1.htm http://www.csus.edu/acaf/socasmt.htm College of DuPage. http://www.cod.edu/outcomes/ProgDis/SOCInstr.htm#RUBRIC James Madison University. http://www.jmu.edu/catalog/97/sociology_and_anthropology.html Skidmore College. http://hudson2.skidmore.edu/academics/sociology/policies/courses-goals.html Valdosta State University. http://www.valdosta.edu/soc/

Portfolios Definition: A portfolio is a collection of student work, usually representing student work over time, such as from the first course in the major until the last semester of the senior year. Advantages: Portfolios enable faculty: To assess complex sets of tasks and objectives, with examples of many different types of student work, including interdisciplinary learning and capabilities; To assess more rigorous and higher order thinking, such as application, synthesis, and evaluation; To track student work over time; To examine not only final student projects, but also, if the faculty think it worthwhile, to look at drafts and earlier phases of student projects; To place the responsibility for demonstrating competence or mastery upon the student; To help students reflect upon their learning and, in the process of compiling the portfolio, to understand more about what they have and have not yet learned; To provide students with documentation for job applications or applications to graduate school. Disadvantages: Portfolios: Require more time for faculty to evaluate than tests or single-sample assessments; Require students to compile their own work, usually outside of class; Do not easily demonstrate lower-level thinking, such as recall of knowledge; May threaten students who limit their learning to cramming for tests or doing work at the last minute; Require a system of storage that may take time or space to set up (see electronic portfolios later. Varieties of Portfolios: Electronic Portfolios: The use of computers now allows institutions and faculty to require students to establish electronic portfolios. Some institutions, such as Dartmouth, Connecticut College, and Wesleyan, have created electronic portfolio systems that not only could be used for assessment, but for other purposes such as registration, advising, and resumé-building. In lieu of an electronic portfolio program, faculty can require students to submit all their work (outside of in-class tests) on disk or to upload copies of their work to a central file system. Then, when it is time to assess the student portfolios, faculty can access those files and assess them in whatever way they think best. (Electronic portfolios also facilitate the submission of video clips, audio clips, or other media.) Showcase Portfolios: A showcase portfolio is one in which students are asked to select and submit a selected sample of their work, such as (a) their best work; (b) their work that they wish they had a chance to improve; or (c) work they are most proud of. A reflective essay or an introductory memo to the faculty can then explain the work, reflect upon how it demonstrates the accomplishments of the student, and/or draw attention to why the student selected those particular examples. Comprehensive Portfolios: A comprehensive portfolio can include virtually everything that the student has ever produced, if the faculty design it that way. The guidelines for the portfolio can be directly related to the learning outcomes that the student is expected to achieve and can serve as a checklist for the student to follow. Faculty might ask the student to submit examples of a specific number of tests, experiments, problems solved, applications of knowledge outside of class, of case study analyses, or of any other types of work typically expected by the discipline. Open-ended Portfolios: An open-ended portfolio leaves it up to the student to decide what to submit in order to demonstrate mastery of the specific learning outcomes that the student has been expected to achieve. Students might submit evidence that is not drawn from the classes themselves, such as work on student clubs, travel, museum visits, summer work, internships, and other experiences. Because faculty cannot predict what students may submit, scoring the portfolio may be a more complex task than scoring the other types of portfolios, but may actually lead to very pleasant surprises about how students have benefited from what they have been taught. Creating and designing a portfolio assessment system: Determine the specific broad learning objectives for the academic program; List the kinds of student work that students might include to demonstrate mastery of the learning outcomes; Determine which kind of portfolio you want students to create; Develop a rubric to score the portfolio (see below); Include the rubric with your instructions to students so that they understand how the portfolio will be evaluated; Write instructions for the students on how to create the portfolio and how it will be used; Inform students that they are responsible for creating the portfolio; Instruct students to label each part of the portfolio according to the learning objective being demonstrated; Instruct students what they are to write as an introduction to the portfolio or as a reflective essay; Instruct students that they are to discuss each sample of student work included in the portfolio, either as an introduction to the sample or within the introductory or reflective essay; Determine how and when students will first be introduced to the portfolio requirement, such as an introductory course in the major or when the student meets with his or her advisor; Score the portfolio using the rubric that you devised. The Scoring Rubric: Since rubrics vary, try to design one that will allow the faculty to assess the portfolio both efficiently and effectively. List each learning objective in which the student must demonstrate mastery. Develop a scale for scoring the rubric, such as one of the options below: a. Grades for each learning objective; b. A score (e.g., 1 – 5) for each learning objective; c. A scale of three categories, e.g., below standard, meets the standard, and exemplary. Review actual portfolios to establish common agreement about the scoring. The faculty should agree upon what constitutes a specific grade, score, or category, perhaps reading a sample of portfolios together and discussing them before scoring all portfolios. Allow some space for faculty to make some notes on their observations, so that the team assessing the portfolios can summarize the strengths and weaknesses that faculty can address later.   Learning Objective Below Standard Meets the Standard Exemplary The student is able to present research findings accurately and clearly. The student does not communicate the research process or findings accurately and clearly. The student communicates the research process accurately and clearly. The student's presentation is creative and insightful, demonstrating great potential for future success. The student thinks critically The student's work simply meets the minimum standards of course work. The student shows evidence of a developing capacity for critical thinking. The student's thinking is creative, thoughtful, insightful, and challenges the reader to rethink the issues being considered. Learning Objective 1 2 3 4 5 The student is able to apply principles of fluid dynamics to describe phenomena in nature.           For other examples of scoring rubrics, see the section on Performance Assessment. Sample of Skidmore departments using portfolios: Education, Mathematics

Portfolio Assessment "Portfolio Assessment: Benefits, Issues of Implementation, and Reflections On Its Use" by Lorie Cook-Benjamin Assessment Update July-August 2001 Volume 13, Number 4 "Portfolio Structure And Student Profiles: An Analysis Of Education Student Portfolio Reflectivity Scores" by Phillip E. Messner, Wright State University; Donna J. Cole, Wright State University; Howard Swonigan, Central State University; Beverly Tillman, University of Dayton; National Forum of Applied Educational Research Journal Volume 5, Number 2, 1992-93. Contact Raymond Rodrigues for copies

Embedded Assessments Definition: Embedded assessments are assessments that make use of the actual work students produce in their courses. The assessments may simply select from work that students do in various courses or may be designed overtly for assessment purposes and then incorporated into the courses. Embedded assessments are also referred to as “classroom-based” or “continuous” assessments. The faculty teaching the courses give grades to the students, but the work selected for assessment is evaluated with program goals in mind and not used for grading. The results of the assessments should not be used to evaluate the faculty teaching the courses. Advantages: The students are simply fulfilling the normal requirements of the course(s) and so do not know that their work is being used for assessment purposes, thereby eliminating issues related to motivation; Embedded assessments can be used to evaluate developmental stages of student learning, rather than simply being summative or assessments at the end of the students’ programs; The assessment process is integrated into the work of both faculty and students; Designing an assessment process enables faculty to consider which skills or knowledge might best be introduced at which levels or in which sequence; There is a clear link between what is taught and what is assessed; Embedded assessment assignments that do not provide reliable information can be redesigned; Results can be compiled quickly by instructors reporting the results to the faculty; Results can be shared with students as a group, allowing them to understand better the criteria that faculty expect them to meet and helping them to evaluate their own strengths and weaknesses. Disadvantages: More complex assignments, such as research papers and projects, will have to be evaluated by a group of faculty using rubrics, thereby requiring more time; Test scores in and of themselves will not provide satisfactory data; Faculty teaching courses must include the embedded assessments that the program faculty decide upon; Assigning appropriate weight to the individual assignments may be difficult. Varieties of Embedded Assessments: Examinations: Specific questions can be inserted into specific examinations for the purpose of assessment. Entire examinations need not be used for assessment unless the faculty believe it best to do so. The faculty conducting the assessment of student responses will need to decide upon the criteria for rating them. For example, are you looking for specific concepts or skills in the student responses? Note: some departments have categorized the types of questions used on examinations to determine whether they are reasonably distributed according to the program goals or may be skewed too much or too little for some goals. Research Papers and Projects: These major projects can be evaluated by using a rubric (see, for example, the discussion of rubrics for portfolio assessments). Faculty should decide upon the criteria to be used for the assessments before the actual assignments are given to the students. Field Experiences or Internships: Student work produced as a result of the field work or internships can be used to assess their learning, work such as logs, field notes, and observations. Creating and Designing Embedded Assessments: Determine the specific broad learning objectives for the academic program; If you have not already done so, determine how those are translated into the individual courses; Conduct an inventory of the types of assignments given in the various courses; Decide which assignments would serve assessment purposes as they are and which might have to be modified to accommodate the assessment; Integrate the embedded assessments within the courses; Devise a way to gather the results of the assessments and translate those results for the entire faculty; Determine strengths and weaknesses of the students as a result of the assessments; Make appropriate changes to the curriculum if that is indicated or to the assessments when they do not provide the information desired.

Pre- and Post-Assessment, or Value-Added Assessment Definition: Value-added assessment attempts to measure student growth over time, from the time that a student enters a program until the student graduates. The most common method is pre- and post- testing, although other types of evidence could conceivably be developed. Advantages: Assessing the students when they first enter a program can establish a firm benchmark against which to measure growth or value-added. Pre-testing is especially helpful for measuring student knowledge, or cognitive learning, and skills, though somewhat less so for measuring values. Pre- and post- testing may work best with traditional four-year undergraduates rather than the more common situation now where students enter, stop-out, transfer, return, and take six years or more to graduate. Pre- and post- testing can be easily scored. Pre- and post- testing can be relatively easily analyzed using statistical procedures. Disadvantages: Pre- / post- testing offers little useful information if the students know little or nothing about the subject of the program when they first enter it. Deciding how to develop meaningfully comparable pre- and post-assessments is difficult, since the pre-test may have to be so basic that any additional learning could be seen as “growth” or value-added. If the assessment is not based upon a highly structured curriculum where the objectives are taught toward and adhered to across all courses in a systematic, it may be difficult to demonstrate the causes of the value-added or to correlate the results of the post-test with the specific courses within the curriculum. Varieties of Value-added Assessments: Note: Virtually all other assessment methods can be used for value-added assessment. Pre- and post- testing happens to be the most common form. Pre- and post-tests: These provide concrete data that could be easily scored analyzed using statistical procedures. Portfolios: Portfolios are almost impossible to construct for the pre- assessment. Essays or research papers: If the assignments and criteria are carefully constructed, these can be scored using a common rubric. Embedded assessments: The type of student work used as an embedded pre- and post-assessment will probably be one of the above. But you could also embed a common assessment, such as a test item or a research task, in a set of courses across all years of the student’s program. Standardized tests: Commercial testing agencies and companies have produced a variety of standardized tests that could be used for this purpose. See the discussion of standardized tests for the advantages and disadvantages. Creating and designing a value-added assessment system: Determine the specific broad learning objectives for the academic program; List the kinds of student work that students might include to demonstrate mastery of the learning outcomes; List the specific knowledge, skills, and/or values that you might want to measure through a value-added process; Decide upon the type of pre- and post-assessment that you will use; Determine which faculty will create the pre- and post-assessment or review examples of commercially available tests for this purpose; Decide when and where the pre- and post-assessments will occur; Decide how the assessments will be evaluated and analyzed; If the pre-assessment is given when students first enter the program, inform those in-coming students that they will be given a pre-assessment, especially if it is to be given outside of a particular class.

Student Performances Definition: Student performances, as defined here, include such student work as internships, field experiences, acting, dancing, musical performances, art shows, oral presentations, PowerPoint and other media presentations, as well as other creative work performed or demonstrated in public. The assessment of these is typically and perhaps best conducted by the use of a rubric, although external critics often write reviews that can be analyzed for assessment purposes. Advantages: Many majors are designed to lead toward students being able to perform in some fashion, so using those for assessment purposes is extremely valuable; Student performances may occur at many different stages of a student’s career, so the possibility of demonstrating value-added is strong; The assessment rubric can be used to teach students the standards that they are expected to achieve; A rubric designed around the goals of the performance and the academic program makes assessment by a variety of evaluators -- whether faculty, professionals in the field, other students, or other external constituents – relatively easy; A critical review can provide an external perspective. Disadvantages: Unless the evaluators agree upon the rubric and/or how it is to be used, the results may not provide consistent data; A performance may not measure all that a student is expected to learn within a program, and therefore other assessment measures may need to supplement the results of the performance assessment; A critical review by an external reviewer my not emphasize the same standards and criteria that the faculty value. Examples of Performance Assessment Rubrics: Example of a scoring rubric designed to evaluate college writing samples. Moskal, Barbara M. (2000). Scoring rubrics: what, when and how?. Practical Assessment, Research & Evaluation, 7(3). Available online: http://ericae.net/pare/getvn.asp?v=7&n=3. Meets Expectations for a first Draft of a Professional Report The document can be easily followed. A combination of the following are apparent in the document: 1. Effective transitions are used throughout, 2. A professional format is used, 3. The graphics are descriptive and clearly support the document’s purpose. The document is clear and concise and appropriate grammar is used throughout. Adequate The document can be easily followed. A combination of the following are apparent in the document: 1.Basic transitions are used, 2.A structured format is used, 3.Some supporting graphics are provided, but are not clearly explained. The document contains minimal distractions that appear in a combination of the following forms: 1.Flow in thought 2.Graphical presentations 3.Grammar/mechanics Needs Improvement Organization of document is difficult to follow due to a combination of following: 1.Inadequate transitions 2.Rambling format 3.Insufficient or irrelevant information 4.Ambiguous graphics The document contains numerous distractions that appear in the a combination of the following forms: 1.Flow in thought 2.Graphical presentations 3.Grammar/mechanics Inadequate There appears to be no organization of the document’s contents. Sentences are difficult to read and understand. Example based on the IUPUI STUDENT TEACHING PORTFOLIOS Observations/Evidence Underdeveloped Initial Practicitioner Exemplary Student Teacher's Portfolio: Describe how the prospective teacher's rationale and organization of the portfolio demonstrates an understanding of and adherence to the Skidmore and State standards for teacher preparation. Places to look for evidence: Introduction and rationale Short- and long-term professional goals Reflection on preparation Educational philosophy statement Coherence and clarity of the portfolio overall Portfolio contents provide evidence that the new teacher does not fully understand or operate on the standards. Prospective teacher does not articulate a clear rationale or lacks the professional discourse skills to communicate his or her philosophy. Portfolio contents reflect a general understanding of and intent to practice the standards. Prospective teacher is using the professional discourse with enough clarity to communicate his or her personal rationale. Portfolio contents exemplify a deep conceptual understanding of the standards. Prospective teacher has Appropriated the professional discourse and communicates with unmistakable clarity. PowerPoint Presentation: Describe how well the student designs PowerPoint screens. The screens include too many words, too many different types of actions, or too may colors, shapes, and/or fonts. The screen limits the number of words effectively; and does not mix too many colors, shapes, and actions. The screen is extremely Attractive in design, communicates in a memorable way, and commands the attention of the viewer. Creating and Designing Rubrics to Assess Performances: See the discussion under portfolios.

Standardized tests Definition: Standardized tests may be norm-referenced or criterion-referenced. Advantages: The tests have been developed and tested by others, so the faculty do not have to spend time developing tests or other measures; The tests may be scored by the commercial company or testing agency, thus saving faculty time; Most tests, especially norm-referenced tests, can provide comparisons with groups of other students, thus enabling the faculty to determine whether they are satisfied with their program or not; Professional programs that are expected to meet national or state standards may find the standardized tests useful, even if they are not required by accrediting associations. Disadvantages: A standardized test may not be constructed or weighted in ways that correlate with your particular academic program, thus decreasing the value of comparisons or components of the test; Student motivation to do well on the test may be very low unless it is a high stakes test—that is, unless it affects whether students proceed in the program or not or unless it counts toward an actual grade; Some students experience high anxiety in taking high stakes tests and may not be able to demonstrate their real abilities and knowledge; The results of standardized tests may not easily be disaggregated in ways that faculty can use in determining how effective components of their program are; Tests designed to make admissions decisions (e.g., the GRE, LSAT, MCAT, and GMAT) do not necessarily measure student knowledge, ability, and values, but, rather, are designed to predict the potential for success in the professional program; even then, the degree of predictive power tends to be limited only to the first semester in such programs; Standardized tests cost money: either the student or the institution must pay for them, and, given the disadvantages summarized above, the expense may not warrant the return. Varieties of Standardized Tests: Norm-referenced: Norm-referenced tests rank-order students to demonstrate achievement differences and are useful for placing students in appropriate courses or for pointing students toward special instructional programs, such as tutoring. Criterion-referenced: Criterion-referenced tests are designed to compare groups of students to groups of other students. They can establish performance levels on specific goals. Commercial tests available: ETS’s Major Field Achievement Tests can be used for specific disciplines. General education outcomes can be measured through such tests as the ACT CAAP or ETS’s Academic Profile. Whether the comparison groups used in reporting the results are appropriate for Skidmore is an important criterion. Analyses of the content of various general education tests are available in the literature. Selecting and using standardized tests: Determine the specific broad learning objectives for the academic program; List the specific knowledge, skills, and/or values that you might want to measure through a value-added process; Review available standardized tests to determine which correlates the most with your particular program; Determine whether the results can be disaggregated in ways that correlate with your program goals; Determine how the tests will be paid for; Arrange for the purchase and administration of the test; Determine when and how the Department faculty will analyze the results of the test in relation to your curriculum.

Indirect Assessment Methods Definition: Indirect assessment methods require that faculty infer actual student abilities, knowledge, and values rather than observe direct evidence. Among indirect methods are surveys, exit interviews, focus groups, and the use of external reviewers. Surveys: Surveys usually are given to large numbers of possible respondents, usually in writing, and often at a distance. Exit interviews and focus groups: Exit interviews and focus groups allow faculty to ask specific questions face-to-face with students. External reviewers: External reviewers are usually representatives of the discipline and usually are guided by discipline-based standards. Advantages: Indirect methods are easy to administer; Indirect methods may be designed to facilitate statistical analyses; Indirect methods may provide clues about what could be assessed directly; Indirect methods can flesh out areas that direct assessments cannot capture; Indirect methods are particularly useful for ascertaining values and beliefs; Surveys can be given to many respondents at a time; Surveys are useful for gathering information from alumni, employers, and graduate program representatives; Exit interviews and focus groups allow faculty to question students face to face; External reviewers can bring a degree of objectivity to the assessment; External reviewers can be guided either by questions that the Department wants answered or by discipline-based national standards. Disadvantages: Indirect methods provide only impressions and opinions, not hard evidence; Impressions and opinions may change over time and with additional experience; Respondents may tell you what they think you want to hear; The number of surveys returned are usually low, with 33 percent considered a good number; You cannot assume those who do not respond would have responded in the same way as those who did respond; Exit interviews take time to carry out; Focus groups usually involve a limited number of respondents; Unless the faculty agree upon the questions that are asked in exit interviews and focus groups, there may not be consistency in the responses. Variations: Electronic surveys: Surveys can be sent out as attachments to email messages. Another method involves having a survey appear on a student’s screen when the student first logs on. Some programs have made these surveys short, asking only one or two questions at a time so that the student is more likely to respond seriously. “Literary or Art Criticism” model: An external reviewer might actually write a review of the materials that he or she reviews, applying his or her own standards or those developed by external groups. This method is likely to be more subjective and may not answer the questions that the faculty want answered—unless they ask the reviewer to address them. On the other hand, the informed subjective opinions of a national expert in the field may provide valuable insights and advice regarding the academic program. Institutional research data: o Percentage of students who go to graduate school; o Statistics on job placement; o Retention; o Courses selected by students; o Faculty/student ratios; o Percentage of students who enroll in study abroad; o Enrollment trends; o Diversity of students in the program. NOTE: These types of data provide various forms of evidence about your program, but do not provide actual data about student learning. They may, however, give you various data on other aspects of program success. Guidelines: Surveys: To encourage responses, keep surveys short; Ask only for information that you want to use; Ask for more than Likert scale and attitudinal responses: o Simulations: “What if . . . ?” “Imagine that . . . .” o Open-ended: “Describe the hardest problem that you had to address in our program.” ”If you had time to re-do one of your research papers, which would it be, and what would you do differently?” “If you could design a new course for our program, what would it be and how would it work?” Do not use a lot of surveys with the same students; If you want to correlate responses with certain characteristics of the students, code surveys so that you can disaggregate specific groups even while you keep the individual’s responses confidential; Gather responses in a timely manner. Focus groups: Be alert to the power of the interviewer – a Department faculty member might intimidate the students; If possible, use an interviewer from outside of the Department; Have only a few key questions – develop follow-up questions as the interview proceeds; Be alert to the student who dominates the conversation – ask others for their opinions; Target your focus group population, e.g., seniors, students who have recently finished the introductory course, students who chose the thesis option; Consider, when possible and appropriate, focus groups with other populations, e.g., employers, parents, undecided freshmen; Ask open-ended questions; Ask questions that require specific examples rather than just attitudes; Keep the focus group small – 5 to 10 individuals; Let the group know how long the focus group will last before they attend it; Record conversations for later transcription or use a note-taker in addition to the focus group leader. A sample questioning pattern for a focus group session: (If you don’t know each other, let’s start by introducing ourselves. I’m . . . .) What was your overall impression of (this program)? What was the most difficult assignment or learning exercise in this program? What assignment or learning exercise was most useful in helping you to learn what the program required? Has you been able to learn anything that you’ve learned in this program outside of the program itself? How? If you could give any advice to the faculty teaching in this program, what would it be? Is there anything I should have discussed with you that I omitted?

Inventories: Inventories of various documents and activities may provide you with some insights into what is actually happening within your program. You can inventory, for example: Course objectives in relation to program objectives; Assignments across courses; Assignments within courses with multiple sections; Institutional data. One of the most valuable tools for guiding your curriculum development and assessment processes is the course objective inventory. To conduct such an inventory, Determine, if you have not already done so, what the learning goals for your program are; Develop a spreadsheet with those program learning goals in the left hand column; List all your courses across the top row; Note the learning objectives for each course – these will probably be more specific than the program learning goals (See “From Program Goals and Objectives to Course Objectives or Learning Outcomes” in the Policies and Guidelines section); Place an X or a check in any box where the program goals and course learning objectives coincide; Review the resulting spread of objectives to determine whether there might be unnecessary overlap or gaps in what is covered that you did not know about; Determine whether any course needs to be revised to address the overlaps or gaps that you have detected through the inventory. As one example, see the Sociology inventory of core courses in the pages following this.

COURSES AND GOALS OF THE SKIDMORE COLLEGE SOCIOLOGY PROGRAM This table links courses to specific goals. These linkages are to a considerable extent arbitrary since pursuance of most goals runs through all sociology courses. This table identifies only major or primary linkages. 1 0 1 O S 6 2 2 O S 7 2 2 O S 5 2 3 o r 4 2 3 O S 5 7 3 O S S E V I T C E L E O S The Skidmore College Sociology Program seeks to develop each student's knowledge of and abilities in the following areas:             1. Crit