Week 2 - from goals, to assessment, to activities

Part 1 - Procedures for Educative Assessment

Procedures for Educative Assessment

Formulate one or two ideas for forward-looking assessment. Identify a situation in which students are likely to use what they have learned, and try to replicate that situation with a question, problem, or issue.

In Honors 101i  we frequently ask students to reflect on their research question, and articulate its significance in the context of one or more audiences. The associated learning outcomes on the syllabus for these activities are:

• create and refine a research topic that is meaningful to you and compelling to others
• communicate your finished and unfinished ideas to others in a variety of ways

But, at the activity level, we might be able to specify these learning outcomes a bit more:

• Articulate the significance of your research topic for at least two different audiences.
• Imagine at least one practical application of your research project, and articulate the relevance of that application with a particular audience in mind.

(Note, we're drawing heavily on chapter 4 of Booth's The Craft of Research for this activity.)

For both a formative and forward-looking assessment activity, students first write an early draft of their topic, question, significance, and practical application. We currently break this down in their research notebook like this:

They do this activity outside of class time. Then, in class students verbally articulate the topic, question, significance, and practical application to a classmate, who in this case is an emerging expert in a different subject area. These activities serve as authentic, formative, and forward-looking assessments of some of the main learning outcomes of the course, which are to communicate research ideas to a variety of audiences, and in a variety of ways. The summative assessment here comes at the end of the quarter, when students give a short presentation of their work that includes the topic, question, significance, and practical application. In "real life" many of our students present their work at UCLA's Undergraduate Research Week, giving them an opportunity to apply these skills in a real world context. 

More forward-looking assessment ideas

One could imagine a yet further forward-looking assessment in which we might ask students to imagine that a legislator has called for the elimination of the funding for their department and/or research program, and called into question its significance, saying it has not practical application. Students would imagine themselves testifying at the legislative hearing, making the case for the relevance of their work.

Another possible forward-looking assessment might be to ask students to imagine they are interviewing for a job they hope to have after graduation. Students would role play the interview, in which they are asked to connect the dots between their undergraduate research work and the job duties and responsibilities.

Criteria & Standards

Select one of your main learning goals, and identify at least two criteria that would distinguish exceptional achievement from poor performance. Then write two or three levels of standards for each of these criteria.

Learning outcome:

Articulate the potential significance of your research topic for two different audiences.

For the purposes of setting criteria, let's improve this by specifying an audience. Maybe something like this:

Learning Outcome	Needs Substantial Improvement	Good	Excellent
Explain the significance of your research topic and question to an emerging expert in a different subject area.	Listener is unable to accurately summarize the main research question and may have trouble articulating its significance.	Listener is able to accurately summarize one or more aspects of the main research question.	Listener is able to accurately summarize the main research question. Listener is able to articulate what makes the research question interesting/significant. Listener may be able to draw connections between the speaker's research topic/question and their own field.

Self-Assessment

What opportunities can you create for students to engage in self-assessment of their performance?

This is an interesting question - there is a bit of tacit, informal assessment going on in this activity. As students work on trying to articulate their research question and its significance, they get feedback from their peers about how well their topic is understood. Their peers ask follow up questions, and either get interested (or not) in various aspects of their topic. All of this provides quite a bit of authentic feedback, and could be used as the basis for a reflective self-assessment. It seems that if we evolved the activity a bit, so that students had an opportunity to hear a peer summarize their question/signficance, then we could follow with a short reflective paragraph in which they self-assess their communication of their topic. How well did their listener understand their research question and its significance? What might they change about their explanation next time if engaged with a similar audience?

“FIDeLity”

Feedback What procedures can you develop that will allow you to give students feedback that is:

• Frequent
• Immediate
• Discriminating, i.e., based on clear criteria and standards
• Lovingly delivered

Because students have opportunities throughout the quarter to re-articulate their research question and its significance, we are able to give them frequent opportunities for immediate feedback. I think this feedback is almost always lovingly delivered, as the class thus far has always become a very constructive learning community in which students support each other. What could be strengthened quite a bit here is the "discriminating/based on clear criteria and standards" part. It seems we could engage students with the collaborative definition of standards for this activity, and have them self- and peer- evaluate at multiple points in the quarter. That would help improve the degree to which we have clear criteria and standards.

Part 2 - Integration

1. Situational Factors

Assuming you have done a careful, thorough job of reviewing the situational factors, how well are these factors reflected in the decisions you made about learning goals, feedback and assessment, learning activities? 

What potential conflicts can you identify that may cause problems? 

Are there any disconnects between your beliefs and values, the student characteristics, the specific or general context, or the nature of the subject in relation to the way you propose to run the course?

Last week, I explored the situational factors for Honors 101i, including the teaching and learning context and the characteristics of the learners and teachers.

In general, I think there is a good match between the situational factors and the learning goals, feedback, assessment, and learning activities. One of the most important situational factors is that this is a credit/no-credit course, so learner motivation is crucial, and it is important that the total workload in the course meet student expectations.

In contemplating a possible self-directed and/or online version of this course, we would want to take into account a set of special considerations for online learning. In particular, we would need to address the community-building aspect of the course that is foundational to our face-to-face course. While we might be able to form a similarly effective online community, we might also consider asking students to work with a research and writing partner throughout the quarter to give them frequent opportunities for collaboration and interaction.

Learning Goals and Feedback & Assessment 

How well do your assessment procedures address the full range of learning goals? 

The main learning goals of the course, from the syllabus are:

1. create and refine a research topic that is meaningful to you and compelling to others
2. develop strategies for finding and incorporating a rich variety of sources into your research projects, including rare and unique materials, images, sound, film, etc. 
3. document sources through citation and annotation 
4. experiment with digital tools to develop a personal approach to organizing your research process
5. understand your rights as a creator, and choose how you would like your work to be shared 
6. communicate your finished and unfinished ideas to others in a variety of ways
7. build a supportive community within our class, in which each person’s abilities to grow and learn are respected and encouraged

Of these, 1-4, and 6-7 are assessed formatively in the research notebook and class activities, and summatively in the written research proposal and final presentation. #5 is assessed informally in class activities, but could likely be better assessed as an aspect of the final research proposal.

Is the feedback giving students information about all the learning goals? 

Yes - weekly feedback in the form of comments on the research notebook, as well as peer and instructor feedback during in class activities address all of the learning goals (though, as mentioned above #5 could likely be better assessed). In an online and/or self-directed version of the class, feedback with FIDeLity (see above) would be significantly more challenging to provide. Peer feedback is a core aspect of the current face-to-face course, and this would likely need to be integrated even more in an online and/or self-directed version of the class.

Do the learning goals include helping the students learn how to assess their own performance?

One of the primary means of self-assessment (and student assessment of the instructors) comes in the form of a the weekly Feedback, Participation, and Ideabox activity. In this activity, students share their answers to the following questions in a form on the course Moodle website:

• What were the highlights of today's class for you? (e.g., something you learned, something you enjoyed, ....)
• What would have made today's class better?
• Please rate your participation for today: (a. Excellent; b. Good; c. Meh.; d. Very little or no participation.)
• Optional: If there is anything you would like us to know about your participation today, tell us here:
• Add to the class Ideabox... 
• What would you like to learn and do in future classes?
• Any other feedback or ideas for us?

The research notebook also offers weekly opportunities for reflection. There are a few self-assessment components integrated into the research notebook, but this is certainly an aspect of the notebook that could be strengthened.

Learning Goals and Teaching/Learning Activities

Do the learning activities effectively support all your learning goals? 

Yes! (I hope so). Though in future versions of the course I hope to strengthen student investigation of disciplinary writing, with a particular focus on investigating the writing strategies and techniques authors in their field use when incorporating sources into their writing.

Are there extraneous activities that do not serve any major learning goal?

We previously had a bunch of very fun activities in which students explored a very diverse set of source types (art, music, film, ....). While this was fun, each particular source type was more or less relevant to each student depending on their topic. More recently we have used this time to help students connect with the particular subject-specialists and source types most relevant for their own project. It would be great if we had enough class time to do both of these things!! It is not that the previous activities were "extraneous" - but more that we needed more time!

Teaching/Learning Activities and Feedback & Assessment 

How well does the feedback loop work to prepare students for understanding the criteria and standards that will be used to assess their performance? 

How well do the practice learning activities and the associated feedback opportunities prepare students for the eventual assessment activities?

In some cases we do this very well. For example, in the first class, students create a collaborative set of criteria for excellent participation and then apply these criteria to their self-evaluations throughout the quarter. In other areas, I think we could seek to improve. Perhaps we could follow a similar model in the peer evaluation of research presentations at the end of the course, for example.

Regarding practice learning activities, the research notebook and in class activities all lead directly into the creation of the final research proposal and class presentation. .So these practice activities are well aligned with the summative assessment.