Standardized Testing: A One-Size-Fits-All Assessment

 By Michelle Boucher

When we first started with formal department-wide assessment in our department, we identified the  skills we want our students to develop as chemists. These are the very skills that the American Chemical Society (ACS) expect students to demonstrate in a certified program. It made sense for us to assess these skills using ACS standardized exams as final examinations in those classes for which ACS offered a standardized exam.

This assessment strategy wasn’t perfect, but the exams offered the potential for a beautiful tool for assessment. They are created by a committee of faculty from diverse institutions, they are checked by the Examinations Institute Board of Trustees, and data on student performance is collected and collated by the ACS and distributed to every school administering the specific exams. Specific means and medians for each question and the exam overall are available from a reasonably large pool of students who took the exam nationwide in a given year. The exams are refreshed every 5-10 years through committee, and fresh student outcome data is collected.

While this all sounds fantastic in theory, the reality of the exams proved less utopian. The committee of faculty from “diverse institutions” who write the exam do not typically include equal representation from schools that serve first generation students or schools that are historically black schools or from community colleges. The faculty represented are typically a mix of faculty from Ph.D. granting institutions, highly selective small liberal arts colleges, and perhaps only 1-2 faculty (out of 17-20) from all the “other types of schools” (community colleges, small comprehensive colleges like ours, etc.). All the exams are political in one form or another. The ACS is clearly dictating, through this exam, what it feels to be of importance in the specific course. For example, one iteration of the organic exam had two questions (out of 70) on green chemistry, when there was a faculty representative on the committee who was a green chemist. The most recent organic exam has 4 carbonyl reactions that are specific subfields and also “named reactions” (after the chemist(s) who discovered or publicized the use of the reactions) in part as a direct answer to a recent push in chemical education to minimize the use of “named reactions” in the interests of promoting a more inclusive classroom experience.

Additionally, there are issues concerning equity and standardized exams. It has been shown, time and again in sociological and pedagogical literature, that there are inherent equity issues around standardized exams. There continues to be discussion and research nationwide around what root causes exist that lead to underserved students (students of color, students from lower economic brackets, first-generation students) and students who identify as women to perform at lower levels on standardized exams. Regardless of the reason, the faculty in our department believe disparities exist. We see it in our incoming students, who benefit greatly from the holistic application review that Utica University offers and are often high achieving students with poor standardized test scores, and we see it on our final standardized exams, where students who have performed exceptionally well all semester choke on a standardized exam.

This past year, we experienced another issue with these standardized exams:  we could not vouch for their reliability. The standardized exam results, in fact all of the assessment data, showed little to no impact on our student education and the 2020-2021 COVID-19 experience. While that makes us go “Yay!”, we know that our students right now have fewer skills than they would typically have at this point in their education. We know that our students are faring better than some cohorts at other schools; we all talk with multiple people at other institutions and are active in ChemEd circles nationally, and we can see where our students place. We know our students have a much smaller “COVID-lag” than cohorts at other institutions. We are proud of that. But we know there is a knowledge and experience gap, and our assessment methods do not show that.

We are making the move away from the ACS standardized exams, or we plan to use them the way we want to use them and bend them to our own wills. There is absolutely no good reason for our program to be dictated to, our learning goals determined and defined, by a committee of homogenous professors protecting a status quo that we have dedicated our professional lives to overthrowing.  In our department, the age of one-size-fits-all assessment and the lies it propagated is over.