Hidden Messages in Chemistry Assessments

Recently, Michael Farabaugh offered some very nice chemistry exam question examples. I appreciated how he emphasized that questions can be written in ways that make it possible to learn how students think rather than just whether they know the correct answer or not. I was intrigued by Michael’s comments about his progression from figuring out what to teach to figuring out how to teach content and know that students have learned it. There is also another facet that is about why to teach in particular ways and I wanted to share some of my thoughts on this.

There is some beautiful work on hidden messages that we send to students by our choices in how we teach that dates back to Douglas Roberts' work in the 1980s on curriculum emphasis. This work has been picked up on by a variety of researchers as well as designers of assessments, such as the Swedish national science tests, and the Programme for International Student Assessment (PISA) tests that are run by the Organisation for Economic Cooperation and Development (OECD). The idea of a “curriculum emphasis” is that how we teach, including the ways that text books are written and how we write assessments, sends hidden messages to students about what the purposes of science are and are not, and what are the roles of teachers and students in learning science, and who should or should not be included in science. There are not correct or incorrect curriculum emphases. Roberts’ point was that if we only expose students to a single curriculum emphasis, then we exclude some students from science. One way that we can be more inclusive in our teaching of chemistry is to consider what are the hidden messages in the assessment questions we choose or write for our students.

Assessment is a part of teaching, whether we write the tests ourselves or prepare students for tests written by others (such as the AP chemistry exam, PISA test, or Swedish national science tests). In this spirit, I thought it could be an interesting exercise to consider different curriculum emphases that could be used to adapt questions. Michael’s post was focused on using specific incorrect answer choices that could uncover particular misconceptions that may be constraining how students are thinking about that problem and probably other chemistry problems. This is really valuable because it helps us as teachers to better understand how students are thinking. What I’d like to focus on here instead is how assessment questions can also send hidden messages to students. This might not be the case, but if I were to assume that the questions in Michael’s post are typical of AP exam questions, then those questions would have a “correct explanations” emphasis.

question_1.png

Figure 1: Question on intermolecular forces using a correct explanations emphasis (derived from previous ChemEd X blog) 

First, let’s look at the “correct explanations” emphasis in the question above that I chose from Michael’s blog post (figure 1). The question seems to come from this emphasis because there is one correct answer, it requires objectivity and logic as the major efforts to answer the question (there is no room for opinions), and the focus of the question is on discerning whether students do or do not have the correct knowledge. Astrid Bulte, who has done a lot of work on curriculum emphases, once shared with me privately four aspects she clarified about each curriculum emphasis (I don’t think she ever published these in English, but she has published a lot of papers that relate to curriculum emphasis and her 2005 paper in the references listed below still has some open access tokens remaining – look for it in Google Scholar to find the link). About the "correct explanations" emphasis, Astrid clarifies that the view of science is that science is "the best meaning system ever developed for getting at the truth about natural objects and events". She identifies the view of the learner in this emphasis as "someone whose preconceptions need to be replaced and corrected". The view of the teacher, then, is as "one responsible for identifying and correcting the errors in students' thinking". This emphasis corresponds to a view of society as needing "believers in the meaning system most appropriate for natural objects and events".

In 2017, some researchers in Sweden, Marie Ståhl and Anita Hussénius, published an analysis of the Swedish national tests for chemistry to look at the “discourses” inherent in the test questions, which they based on Roberts’ view of curriculum emphasis. They found that the test questions had two main types of discourses (ways that the written test items contain hidden messages of norms and values). Most of the test questions belonged to what they called “disciplinary discourse”. This discourse focuses on scientific concepts and phenomena and how to speak and act scientifically. It’s pretty similar to what Roberts called the “correct explanations” curriculum emphasis. The disciplinary discourse questions on the Swedish national chemistry test were academic and skill-based, narrow, usually had one correct answer, and drawings (seldom used) functioned as backdrops to reduce the barrier to understanding the question but they were unnecessary for answering the questions. A smaller number of test questions belonged to what these researchers called “transgressing discourse”. This discourse focuses on interdisciplinary issues, putting chemistry in the context of the issues, and addressing what is science and what is not science. These questions often had more than one correct answer, and drawings (frequently used) usually functioned to make the question comprehensible so they were inseparable from the questions.

So, how might the question above (figure 1) be written with a different curriculum emphasis? Perhaps a multiple-choice version of a question with an "everyday coping" emphasis could look like this:

molecules.png

Figure 2: Revision of the question in figure 1 to use an "everyday coping" emphasis

Propane and ethanol are both soluble in water to different extents (the molecule structures are shown). If you had water that was contaminated with both propane and ethanol, and you wanted to make the contaminated water pure enough to drink, what could you do to get the water to be decontaminated by the propane and ethanol?

(A) Boil the contaminated water for at least 10 minutes so that the contaminants will not be poisonous anymore.

(B) Add chlorine to the contaminated water to deactivate what is toxic in the contaminants.

(C) Boil the contaminated water and collect the gas that comes off of it, and then condense it back to liquid and it will be decontaminated water.

(D) Watch the temperature while boiling the contaminated water, and when the temperature gets to 100 C the contaminants will be gone.

The question seems to come from the “everyday coping” emphasis because it starts from a problem that is about coping with a problem that could occur in the everyday life of a person. A student has to apply what is learned in chemistry to answer the question: specifically (1) propane, ethanol and water each have different boiling points, and (2) the intermolecular forces that stabilize water molecules in the liquid phase are stronger than those of propane and ethanol, so propane and ethanol release those binds more easily than water molecules do. The question still contains incorrect answer choices that reflect ways of thinking that students may have which can also constrain how students think about other problems in chemistry.

In Astrid Bulte’s perspective, the view of science inherent in the “everyday coping” emphasis of this question is that science is a "meaning system necessary for understanding and therefore controlling everyday objects and events". The view of the learner is as someone who "needs to master the best explanations available for comfortable and competent explanation of natural events, and control of mechanical objects and personal affairs". The teacher's view is as someone who sees value in regularly "explaining natural manmade objects and events by appropriate scientific principles". And society is viewed as being comprised of "autonomous knowledgeable individuals who can do mechanical things well, who are entrepreneurial, and who look after themselves and are valued members of the social order".

In her extensive chemistry curriculum reform work over several decades in the Netherlands, Astrid proposed that a chemistry curriculum that moves students through many different curriculum emphases during different parts of the curriculum can have major benefits for students, because it prepares them to deal with the world in different ways. Assessment is a part of teaching, whether we write the tests ourselves or prepare students for tests written by others (such as the AP chemistry exam). I suggest that questions for the AP chemistry exam, as well as tests that we all write for our students in other chemistry courses, could benefit from including more varied discourses. I think we can all be agents for change in making chemistry more inclusive, and more relevant and useful to all of our students throughout their lives, if we try to do more of this in our teaching of chemistry.

References

• Roberts, D. A. (1982). Developing the concept of “curriculum emphases” in science education. Science Education, 66(2), 243-260.
• Ståhl, M., & Hussénius, A. (2017). Chemistry inside an epistemological community box! Discursive exclusions and inclusions in Swedish National tests in Chemistry. Cultural Studies of Science Education, 12(2), 395-423.
• van Driel, J. H., Bulte, A. M., & Verloop, N. (2005). The conceptions of chemistry teachers about teaching and learning in the context of a curriculum innovation. International Journal of Science Education, 27(3), 303-322.
• van Driel, J. H., Bulte, A. M., & Verloop, N. (2008). Using the curriculum emphasis concept to investigate teachers’ curricular beliefs in the context of educational reform. Journal of Curriculum Studies, 40(1), 107-122.

word_cloud.png

The Word Cloud (Python program) used as the preview image was made from the main text of the Stahl & Hussenius article. Are there any words that surprise you? Whether you have read the article or not, "naked" may have caught your eye. The word comes up a lot in the article because it's one of the most popular types of discourses they found in the exams that they analyzed. It is called "naked discourse", which belongs to one of the two overarching groups of discourse (disciplinary discourse).