Home >> Vice President Academic Affairs >> The Weekly Blab >> Outcomes Assessment

Outcomes Assessment

October 4, 2006

 

Outcomes Assessment

 

Last year, we began a program of Outcomes Assessment where each department was called upon to:

(a)    Identify a series of measurable outcomes goals for the department

(b)    Show how these goals match up with the SPSU’s outcomes goals (A list of University Outcome Goals is given below)

(c)    Show where (in our curriculum or practices) these outcomes are being met

(d)    Show how success in reaching the outcomes goals is defined and measured

and finally,

(e)    Show whether we are succeeding in meeting them. 

 

This document (which is a streamlined version of the one distributed last year) should be helpful in carrying out this process in a step-by-step fashion.  The method used here is very similar to that used in ABET assessments, and based on a paper entitled “Uses of the Assessment Record Book Portfolio and Other Mechanisms to Encourage Comprehensive Assessment of Student Learning” by Paul Cunningham (Rivier College) and Christopher Beach (Unity College).

 

Outcomes Assessment is part of an effective Strategic Planning process, as well as a means to identify budgetary priorities.  Our accrediting body (SACS) also requires all universities to engage in Outcomes Assessment, and will be looking for evidence that we are implementing and applying the process.

 

 

 


Table 1

University Outcome Goals

 

  1. Our students will benefit from a campus-wide environment, focused on student success.
  2. Our students will gain both practical, applied skills and theoretical knowledge through SPSU’s academic programs.
  3. Our students will be part of a learning community encompassing faculty, staff and students.
  4. Students from all demographic groups (full- or part-time; traditional or non-traditional; male or female; students from diverse backgrounds) will be well served by the University.
  5. Our students will interact with faculty and staff who treat them with respect and who are dedicated to helping them address any issues that are an impediment to their quality of life while at SPSU.
  6. Our students will benefit from a safe, comfortable and positive campus environment that exemplifies cooperation between faculty and staff for the benefit of the entire community.
  7. Our students will benefit from campus operations that are effective and efficient, where use of existing resources is maximized, and where carefully managed enrollment growth ties resources to increased student numbers.
  8. Our students will be supported by a campus-wide advising program focused on their retention, progression and graduation.
  9. Our students will benefit from a general education program that develops critical and creative thinking; cross-disciplinary understanding; and effective communication.
  10. Our students will benefit from a campus that exhibits clear and consistent communication.
  11. Our students will be engaged in a campus-wide culture that integrates academics and co-curricular programming.
  12. Our students will develop a background that fosters problem-solving, ethical awareness, an international outlook, and a desire for lifelong learning.
  13. Our students will graduate with the capacity to assume leadership roles.
  14. Our students will be given opportunities to remain engaged in the SPSU community after graduation.
  15. Our students will have multiple opportunities to showcase their success.

 

 

 

Step 1 of the Process:  Determine Your Department’s Outcome Goals

 

Each department should determine several specific, measurable academic outcome goals.  Using a hypothetical chemistry major as an example, a good framework for an academic outcome goal is to fill in the blank in the statement: 

 

“Each student who completes the degree requirements in the Chemistry program should be able to:                  ”.

 

 

Departmental goals should be broader than those for a particular course.  For example, the goal

 

“Each student who completes the degree requirements in the Chemistry program should be able to balance a chemical reaction”.

 

would not be an appropriate departmental goal—the goal is too narrow.  It would, however, be a perfectly reasonable outcome goal for a course like General Chemistry. 

 

Each department should determine a reasonable number (8-12) of academic outcome goals that relate to the University’s Outcome Goals.  Try to ensure that the goals are neither too narrow nor too vague.  Very important:  each goal should be measurable (see Step 4).

 

Here are some examples of academic goals:

 

                  “Each student who completes the degree requirements should be able to:

  • Define/use current terminology in the field.
  • Describe major historic developments and their impact on the field.
  • Apply the principles, concepts and laws of the field to new situations.
  • Critically read, abstract and interpret research in the field.
  • Analyze situations and make decisions based on data in the field.
  • Communicate orally and in writing in a clear and effective manner.
  • Apply mathematics to the field.
  • Write literature searches and research papers in the style of the field.
  • Discuss ethical principles as they relate to the field.
  • Carry out a research project in the field.
  • Etc.—write your own.

 

Similarly, each department should develop several administrative goals.  For our hypothetical example, a good framework for an administrative outcome goal is to fill in the blank in the statement: 

                  “The Chemistry department enhances student success by:                  ”.

 

Here are some examples of administrative goals:

 

“The Chemistry Department enhances student success by:

  • Scheduling courses to meet the needs of all students.
  • Reviewing curricular requirements to ensure they are current.
  • Having effective communications with our students.
  • Ensuring that each member of the department treats students with understanding and respect.
  • Spending its budget to maximize positive impact on students.
  • Providing research and travel opportunities for its faculty in order to keep them at the “state of the art”.
  • Providing strong advising to students.
  • Etc.—write your own.

 

 

Step 2:  Integrate the University’s Goals with your Department Goals

Each program should now compare its academic and administrative  goals with the University’s goals, to see how they interrelate.  For example, the academic goal:

 

“Communicate orally and in writing, in a clear and effective manner”

 

is clearly related to the University Goal

 

“Our students will benefit from a general education program that develops critical and creative thinking; cross-disciplinary understanding; and effective communication.”

 

The easiest way to show the interrelationships is to prepare a correlation chart.  An example is given in Figure 1.  Whenever the University Goal and Program Goal correlate, an “X” should be placed in the grid.  This figure is just an example—there are many additional or alternative goals that might be appropriate for a Chemistry Department.

 

Charts like this are helpful in clarifying thinking and seeing if the program goals are too heavily (or lightly) weighted in any particular direction.  Note that in the example, many program goals tied in with University Goal 2 (as one would expect for an academic department), but none tied in with University Goals 6 (positive campus environment) or 14 (relating to keeping alumni involved).  This may provide some food for thought in terms of gaps in the department’s planning.

 

Step 3:  Integrate the Department Academic Goals with Courses

Each department should now compare its courses (both required and elective) with its academic outcomes goals.  This is shown in Figure 2. 

Whenever the Departmental Goal and course correlate, an “E” should be entered if the emphasis is at the elementary level, an “I” for intermediate emphasis, and an “A” for advanced emphasis. 

 

Looking at Figure 2, we can see that no course in the Chemistry curriculum satisfies departmental goal 7 (“Discuss ethical principles as they relate to the field”) at an advanced level.  This sort of omission may or may not be a problem, and at least should be discussed.

 

Step 4:  Assessment Methods

The previous steps tell us what we want to do.  How do we know that we’re actually doing it?  Yes…it’s the dreaded “A” word—assessment.  Remember—each outcomes goal was supposed to be measurable. 

 

Some common ways of assessing outcomes as to whether you’re actually getting the results that you want are listed below.  Some are more appropriate for academic outcomes, and some for administrative outcomes.  No doubt you can think of others:

 

  1. Capstone project within course or program
  2. Portfolio (collection of student work samples)
  3. Writing samples
  4. Homework assignments
  5. Standardized test (i.e., from a professional society)
  6. Non-standardized test (i.e., developed at SPSU)
  7. Licensing exam
  8. Reviewed exhibition or performance
  9. Performance measures (project, thesis, recital)
  10. Internship (external and internally reviewed)
  11. Participation (club, research, civic engagement)
  12. Honors (awards)
  13. Graduate School Acceptances
  14. Job Placement Statistics
  15. Graduating Student Survey or Exit Interview
  16. Alumni Survey
  17. Employer Survey
  18. Student satisfaction surveys
  19. Attrition Exit Interview
  20. Retention/Transfer rate
  21. Graduation Rate
  22. Anecdotal information

 


 

You now have to decide what assessment measures are appropriate for each course or function, and within the department overall.  In order to have valid results, it is useful to use a variety of assessment methods.  Since students learn in many different ways, it is necessary to assess in many different ways as well.  This information should be included in Figure 3, with an “X” indicating that the particular form of assessment will be used for a particular course or function, or in the department overall.

 

Once you have decided upon various assessment measures for the department, it is time to set benchmarks.  What would be an indication of success in meeting the outcome goal?  Suppose that one of the ways of measuring the goal of “Communicate knowledge in the field orally and in writing, in a clear and effective manner” is to gather a portfolio.  The benchmark might therefore be “The portfolio will be evaluated by a jury of the faculty in the program and related disciplines.  90% of the portfolios will be judged to represent ‘acceptable’ or better work.”  A table of benchmarks for each of the department’s assessment measures should be prepared.

 

Step 5:  Follow-Up

Once you have established your outcomes goals, assessment methods and benchmarks, it’s time to do some measuring.  On at least an annual basis (and perhaps you might want to do this every semester), you should measure whether your outcomes goals are being met.  You need to determine if each benchmark for the major has been exceeded, met or not met.  You should also discuss if any desired outcomes are missing from your goals, or if any original outcomes are no longer relevant.

 

In all cases, the benchmarks should be reviewed.  Suppose that a benchmark was exceeded.  There are several possibilities:

(a)    Is that an indication that the benchmark was set it too low?  Perhaps the expectations need to be raised. 

(b)    Perhaps too much attention is paid to that area in the curriculum.  Perhaps the requirements devoted to this area should be cut back, and reallocated to other areas.

(c)    Perhaps it was set right, but something unusual caused it to be exceeded (i.e., it was an unusually strong class of students).

 

Suppose that a benchmark wasn’t met.  There are several possibilities here. 

(a)    Perhaps it was set too high—you overestimated what was reasonable. 

(b)    Perhaps it was set right, but there was something unusual that kept it from being met (i.e., a major piece of equipment blew up). 

(c)    Perhaps it was set right, but the curriculum has a flaw in it—there is insufficient preparation in a particular area and a course needs to be added.

 

This analysis of outcomes should be captured in writing in an outcomes assessment report.  The report should include details of any changes you are making in order to address your benchmarks.

 

Now, the process begins again.  After another semester (or year), measure if the new benchmarks have been met.  Remember—this is a process of continuous review, assessment and improvement. 

Step 6:  Bringing it to the Course Level

After carrying out this analysis at the program level, it is time to take it to the course level.  Every course should have a set of outcomes goals.  These goals should be much more specific than the program goals, and (for a hypothetical course in our hypothetical chemistry program) answer the question:

 

“Each student who successfully completes CHEM 1211 should be able

to:                      ”.

 

A grid similar to Figure 1 should be developed, correlating the course outcomes with the program outcomes.

 

The outcome goals for each course should be explicitly stated in the syllabus, and strongly communicated to the students.

 

Similarly, each course should have benchmarks for its evaluation methods, which should be reviewed and assessed, with course or benchmark modifications made as necessary.


Figure 1

University Goals and Chemistry Department Goals

 

Students completing the Chemistry program should be able to:

  1. Define/use current terminology in the field.
  2. Describe major historic developments and their impact on the field.
  3. Apply the principles, concepts and laws of the field to new situations.
  4. Critically read, abstract and interpret research in the field.
  5. Analyze situations and make decisions based on data in the field.
  6. Communicate orally and in writing in a clear and effective manner.
  7. Discuss ethical principles as they relate to the field.
  8. Carry out a research project in the field.
  9. Routinely providing help-sessions and/or peer tutoring for all required courses.
  10. Utilizing multiple teaching strategies to ensure student success.
  11. Advising all chemistry majors regarding course sequencing, networking, career opportunities and

 

The Chemistry Department enhances student success by:

attainment of personal goals.

  1. Ensuring that laboratories are up-to-date with regards to methods and equipment .

 

                                                                                                                                Department Outcome Goals

University Outcome Goals:

1

2

3

4

5

6

7

8

9

10

11

12

Goal 1: Our students will benefit from a campus-wide environment that is focused on student success

 

 

 

 

 

 

 

 

X

X

X

X

Goal 2: Our students will gain both practical, applied skills and theoretical knowledge through SPSU’s academic programs.

X

X

X

X

X

X

X

X

 

 

 

X

Goal 3: Our students will be part of a learning community that encompasses faculty, staff and students.

 

 

 

 

 

 

 

X

X

 

X

 

Goal 4: Students from all demographic groups will be well served by the University.

 

 

 

 

 

 

 

 

X

X

 

 

Goal 5: Our students will interact with faculty and staff who treat them with respect/dedicated to helping them address any issues that are an impediment to their quality of life

 

 

 

 

 

 

 

 

X

X

X

 

Goal 6: Our students will benefit from a safe, comfortable and positive campus environment that exemplifies cooperation between faculty and staff for the benefit of the entire community.

 

 

 

 

 

 

 

 

 

 

 

 

Goal 7: Our students will benefit from campus operations that are effective and efficient, use of existing resources is maximized, and where carefully managed enrollment growth ties increased resources to increased student numbers.

 

 

 

 

 

 

 

 

 

 

 

X

Goal 8: Our students will be supported by an advising program focused on their retention, progression and graduation

 

 

 

 

 

 

 

X

X

 

X

 

Goal 9: Our students will benefit from a general education program that develops critical and creative thinking; cross-disciplinary understanding; and effective communication.

 

X

 

 

 

 

X

 

 

 

 

 

Goal 10: Our students will benefit from a campus that exhibits clear and consistent communication

 

 

 

 

 

 

 

 

 

 

 

 

Goal 11: Our students will be engaged in a campus-wide culture that integrates academics and co-curricular programming

 

 

 

 

 

 

 

 

 

 

X

 

Goal 12: Our students will develop a background that fosters problem-solving, ethical awareness, an international outlook, and a desire for lifelong learning.

 

 

X

X

X

 

X

X

 

 

 

 

Goal 13: Our students will graduate with the capacity to assume leadership roles.

 

 

X

X

X

X

 

X

 

 

 

 

Goal 14: Our students will be given opportunities to remain engaged in the SPSU community after graduation.

 

 

 

 

 

 

 

 

 

 

 

 

Goal 15: Our students will have multiple opportunities to showcase their success.

 

 

 

 

 

 

 

X

 

 

 

 

 

Figure 2

Department Academic Outcome Goals and Chemistry Courses

 

Students completing the Chemistry program should be able to:

1:         Define current terminology in the field.

2:         Describe major historic developments and their impact on the field.

3:         Apply the principles, concepts and laws of the field to new situations.

4:         Critically read, abstract and interpret research in the field.

5:         Recognize problems, analyze situations and make decisions based on data in the field

6:         Communicate knowledge in the field orally and in writing, in a clear and effective manner.

7:         Discuss ethical principles as they relate to the field.

8:         Carry out a research project in a semi-independent manner       

 

 

                                                                      Department Academic Outcome Goals

Chemistry Courses

Req’d?

1

2

3

4

5

6

7

8

CHEM 1211—Principles of Chemistry I

Yes

E

E

E

 

 

E

E

 

CHEM 1212—Principles of Chemistry II

Yes

E

E

E

 

 

E

 

E

CHEM 2211—Environmental Chemistry

No

I

E

 

E

E

I

I

I

CHEM 2511—Organic Chemistry I

Yes

I

E

I

 

 

E

E

 

CHEM 2512—Organic Chemistry II

Yes

I

E

I

 

 

I

 

I

CHEM 3100—Analytical Chemistry

Yes

I

E

A

E

I

I

E

 

CHEM 3111—Biochemistry I

Yes

A

E

A

I

 

I

 

 

CHEM 3300—Instrumental Analysis

Yes

A

E

A

I

 

I

 

 

CHEM 4111—Physical Chemistry I

Yes

I

E

A

E

I

I

E

 

CHEM 4412—Physical Chemistry II

Yes

I

E

A

E

I

I

E

 

CHEM 4411—Inorganic Chemistry

Yes

I

I

I

E

 

I

 

I

CHEM 4801—Internship

No

 

 

A

A

A

A

E

A

CHEM 4901—Research

Yes

 

 

A

A

A

A

 

A

MATH 1113—Pre-Calculus

Yes

 

 

E

 

E

 

 

 

MATH 2253—Calculus I

Yes

 

 

E

 

E

 

 

 

MATH 2254—Calculus II

Yes

 

 

E

 

E

 

 

 

ENGL 1101—Writing I

Yes

 

 

 

 

 

E

 

 

ENGL 1102—Writing II

Yes

 

 

 

 

 

E

 

 

SPCH 2400—Public Speaking

Yes

 

 

 

 

 

E

 

 

STS 2400—Science and Society

Yes

 

 

 

 

 

 

 I

 

Area C1—Literature

Yes

 

 

 

 

 

E

 

 

Area C2—Fine Arts

Yes

 

 

 

 

 

E

 

 

Area E1—American Context

Yes

 

E

 

 

 

 

 

 

Area E2—World History

Yes

 

E

 

 

 

 

 

 

Area E3—Social Science

Yes

 

 

 

 

 

 

E

 

Area E4—Global Context

Yes

 

 

 

 

 

 

E

 

 


 

Figure 3

Assessment Types and Chemistry Courses

 

Types of Assessment Used:

  1. Capstone project within course or major
  2. Lab reports
  3. Non-standardized exams
  4. Reviewed research presentation
  5. Internship (external and internally reviewed)
  6. Homework
  7. Graduate School Acceptances
  8. Graduating Student Survey or Exit Interview
  9. Alumni Survey
  10. Graduation Rate
  11. Anecdotal information
  12. Annual Report
  13. Student Evaluations

                                                                                                                                Assessment Methods

Assessment of Students:  Chemistry Courses

1

2

3

4

5

6

7

8

9

10

11

12

13

CHEM 1211—Principles of Chemistry I

 

X

X

 

 

X

 

 

 

 

 

 

 

CHEM 1212—Principles of Chemistry II

 

X

X

 

 

X

 

 

 

 

 

 

 

CHEM 2211—Environmental Chemistry

 

X

X

 

 

X

 

 

 

 

 

 

 

CHEM 2511—Organic Chemistry I

 

X

X

 

 

X

 

 

 

 

 

 

 

CHEM 2512—Organic Chemistry II

 

X

X

 

 

X

 

 

 

 

 

 

 

CHEM 3100—Analytical Chemistry

X

X

X

 

 

X

 

 

 

 

 

 

 

CHEM 3111—Biochemistry I

 

X

X

X

 

X

 

 

 

 

 

 

 

CHEM 3300—Instrumental Analysis

X

X

X

X

 

X

 

 

 

 

 

 

 

CHEM 4111—Physical Chemistry I

 

 

 

X

X

 

 

 

 

 

 

 

 

CHEM 4412—Physical Chemistry II

X

 

 

X

 

 

 

 

 

 

 

 

 

CHEM 4411—Inorganic Chemistry

 

 

X

 

 

X

 

 

 

 

 

 

 

CHEM 4801—Internship

 

 

X

 

 

X

 

 

 

 

 

 

 

CHEM 4901—Research

 

 

X

 

 

X

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Assessment of Program:  Chemistry Major

 

 

 

 

 

 

 

 

 

 

 

 

 

Offering quality instruction to students in the field of chemistry

 

 

 

 

 

 

X

X

X

X

X

X

X

Advising chemistry majors

 

 

 

 

 

 

 

X

X

 

X

 

X

Maintaining records of academic progress of chemistry majors

 

 

 

 

 

 

 

 

 

X

 

 

 

Provide mentoring for untenured faculty members in chemistry

 

 

 

 

 

 

 

 

 

 

 

X

 

Administer chemistry budget

 

 

 

 

 

 

 

 

 

 

 

X

 

Review chemistry holdings in library

 

 

 

 

 

 

 

 

 

 

 

X

 

Overall Program

 

 

 

 

 

 

X

X

X

X

X

X

 

 

 


 

Chemistry Department Benchmarks

  1.                  Capstone project within program

By the end of the program, 100% of the students will have participated in a senior research capstone project.  They will perform a literature search on their research topic, write a research proposal, perform the research, and write up their results in a senior thesis. The literature search, research proposal and thesis referred to in item 1 will be evaluated as to spelling, punctuation, vocabulary, grammar and clarity of presentation by the research advisor and one member of the writing department.  At least 90% of the literature searches and theses will be evaluated as being “acceptable” or better in these areas.

2.                  Lab Reports

The lab reports will be evaluated as to spelling, punctuation, vocabulary, grammar and clarity of presentation by the course instructor.  Average initial grades of 70 are expected.  Students will have opportunity to rewrite lab report to address errors.  Average grade after rewrite of 80 expected.

3.                  Non-standardized exams

Comprehensive final examinations will be given in General Chemistry, Organic Chemistry and Physical Chemistry.  Students expected to achieve average of 75% on all exams.

4.                  Reviewed research presentation

The research results will be presented to all faculty and senior science majors in an oral presentation.  The presentation will be critiqued by all present, and the results will be discussed by the research director and student.  The thesis will be reviewed by the research director as to scientific content.  At least 90% of the theses will be evaluated as “acceptable” or better.

5.                  Internship (external and internally reviewed)

70% of interns in Chemistry will be rated as “good” or “excellent” by their internship supervisors.

6.                  Homework

All indicated chemistry courses will assign homework on at least a weekly basis.  90% of students will complete homework assignments and hand them in.

7.                  Graduate School Acceptances

33% of graduates in Chemistry will be accepted into a masters or Ph.D. program in a scientific field within 5 years of graduation.

8.                  Graduating Student Survey or Exit Interview

Each senior will fill in a graduating student survey.  At least 66% will rate the Chemistry program as being “good” or better, and 33% as “excellent”.

9.                  Alumni Survey

An alumni survey will be sent 5, 10, 15…years after students graduate.  At least 75% will agree with the statement “my education compared favorably with those in similar positions in my workplace”.


10.              Graduation Rate

At least 75% of students declaring chemistry as a major and successfully completing General Chemistry I with a grade of C or better will graduate with a degree in a science area .

11.              Anecdotal information

As part of the exit survey referred to in #8, all students will write an essay about their best and worst experiences within the major program and its cognates.

12.              Annual Report

An annual report measuring the chemistry Department’s accomplishments against these benchmarks will be issued each year in June.

13.              Student Evaluations

Standardized student evaluations will be administered in every chemistry course every semester (including summer and online).  All faculty will average at least “3.0/4.0” on all questions related to teaching effectiveness.  Written student evaluations will be solicited from all majors and 33% of all other students for all promotion and tenure cases.