Week 4 - next steps for the digital research notebook

Throughout #IDE15ALA, I have been reflecting on a possible adaptation of Honors 101i - "Research Today," an upper-division undergraduate seminar on research methods that includes both extensive in-class interaction and a strong self-directed online component outside of class time. In this class, students create a digital research notebook in which they explore and develop their topics and reflect on the research process through guided weekly assignments. During class, students engage in a variety of collaborative activities, including sharing their research notebook progress, providing peer feedback and review, and generating and sharing ideas.

For my final project, I hope to lay the pedagogical groundwork for the development of  a new version of the digital research notebook that could serve as the basis for a self-directed "course" or perhaps more accurately, "sequence of activities" on developing research skills. If successful, there could be a variety of potential applications for this notebook:

• incorporation of the entire notebook or modular elements of the notebook into other courses seeking to strengthen the scaffolding of the research and writing process outside of class time;
• as a resource in support of a reference interview in which peer-to-peer research assistants could help students focus on their current research/writing challenge, and then identify subsequent steps as activities in the research notebook;
• as a recommended resource for students engaging in writing consultations with the Undergraduate Writing Center;
• as the foundation for the development of a series of self-directed, sequenced learning modules in research skills and strategies. 

That all sounds rather ambitious at this point, but - it is good to dream, right? ;-)

1. What technologies (and these can be old, new, or emerging) might be most appropriate for your final project? 

• Campus google docs: http://g.ucla.edu/
• CCLE (UCLA's Moodle instance)
• A growing series of YouTube videos on research strategies

And perhaps also Zotero/Endnote, online mindmapping software, and other digital research tools.

2. Does your final project align with any of the trends represented in the Horizon Report you reviewed? 

Related trends from the NMC Horizon Report > 2014 Higher Education Edition:

Scaling Teaching Innovations 

It is hard to say whether Honors 101i is innovative or not, but our assessments to date have all shown that it has been an effective course, both in terms of helping students develop a deeper and more diverse personal approach to the research process and in building a supportive learning community. For me the trend in this case is not so much "scaling teaching innovations" but rather: how do we scale effective instructional practice, particularly for a course that involves extensive engagement and interaction both online and face-to-face (f2f).

The question of scaling effective instructional practice is an exceptionally important one for us. Continually increasing enrollment increases the pressure on our existing service models for instruction and reference. For instructional services, our team is at or near our capacity. For research assistance services, while we are not currently close to our service capacity, our campus partners at the Undergraduate Writing Center are frequently fully booked. As we continue to learn the secrets of their success, we have the potential to greatly increase the number of students engaging with research assistance services in the future. Our challenge then is not one of advocacy or "inserting ourselves into the curriculum" -- in many ways we are already well integrated into the curriculum in strategic areas, and likely have more opportunities for embedded/consultative librarianship than our current staff size can effectively deliver. As just one example, there are multiple Undergraduate Education Initiatives with research components where we would welcome an opportunity to increase our instructional engagement given the staffing/capacity. This will only be possible if we learn to build on the success of our smaller-scale engagements, and make some of those learning outcomes accessible to large groups of the undergraduate population through at-scale solutions. This speaks for the creation of a service ecosystem built out of the interaction between our existing services, and the innovative development of more scalable solutions.

Integration of Online, Hybrid, and Collaborative Learning

One could call the current approach to 101i "integrated" in that it attempts to bring together collaborative f2f instruction with independent and reflective learning outside of class time. By some definitions it is "hybrid" or "blended" instruction, though it would be hard to say whether the instruction leans toward either the face-to-face or online..... I think of the course as instruction involving a variety of modes, and find it challenging to imagine a course called "Research Today" taught without some sort of mix of the digital and "traditional."

The possible adaption of the course I'm exploring here relates well to the "integration" trend mentioned in the Horizon Report in that it is an attempt to adapt an effective instructional practice currently delivered in one context into a framework or collection of "de-bundled" instructional elements potentially impactful across multiple contexts and/or modes of learning.

Virtual Assistants?

It is interesting to imagine the potential connections here. This trend is identified as later in the adoption timeline ("four to five years"), but nonetheless a few possibilities and ideas come to mind (all quite imaginary at this point). One could imagine the research notebook being combined with a next generation Research Project Calculator, enhanced with google calendar integration, text reminders, and "smart" applications of learning analytics and the quantified self to assist students with the planning, personal organization, time-on-task, and meta-learning required for success with a long format research/writing assignment. Of course, we would want to make sure that in all of this, we would not be following entirely in the footsteps of clippy.



3. How does your choice of technology enhance or improve instruction, or motivate learners?

The technology here is not absolutely essential, and I hope that in addition to developing the google docs version of the research notebook, we will at the same time be able to develop a similar series of paper handouts that accomplish very similar learning outcomes. The higher-tech version would benefit from the use of supported campus technologies (like http://g.ucla.edu and http://ccle.ucla.edu). There is also a potential benefit from the integrated use of multiple tools (e.g., google docs + zotero/endnote + ....).

The general idea though, is to further diversity our instructional toolbox, and create learning opportunities for students that include but extend well beyond the one-shot, are effective for diverse learning styles, and are discoverable at a variety of points of need (including the syllabus, course management system, and through direct engagement with library service structures).

Week 3 - Learning Theories and Motivation

Part A - 

Discuss which theory/ies might be most applicable to your instruction and outline a specific activity/assignment/exercise that would facilitate learning according to that theory.

The design of Honors 101i is largely based around constructivist, or perhaps more specifically a social constructivist approach to learning and teaching.

The typical structure of a class week (hopefully) illustrates how this approach works in practice:

• Digital Research Notebook - After week 1, each week begins with students contributing to their digital research notebook. In this notebook students take notes on their readings, record their progress and discoveries, reflect on their research and writing process, analyze and comment on their growing collection of sources, and engage in extensive writing activities in which they explore their research topic from a variety of perspectives.
• Questions -  Class sessions typically begin with something we call "Questions" - class time devoted first to students sharing their research notebooks with each other and asking each other questions about their weekly work (e.g., What are your favorite strategies for reading academic work? How has your research question changed as a result of your reading? What sorts of data and evidence have you collected so far?). We then move on to questions that frame the experiential learning that happens later in the "Lab" part of class (e.g., What kinds of information do experts in your field view as credible evidence?). We do frequent think-pair-share activities during "Questions" and students record the notes from these conversations in their research notebooks in a section called "Class notes."
• Lab - During the "Lab" phase, we do a series of experiential learning activities which both revolve around predetermined learning outcomes, and involve a certain amount of instructional improvisation as we incorporate student questions and areas of interest. During a lab session we might analyze peer-reviewed articles selected by each student, identifying key structural elements in the articles, finding similarities and differences in the structure of articles across disciplines, and identifying strategies authors use to signal to the reader what is new about their research while at the same time making connections between their work and previously published work by other authors. Lab time also involves a mix of individual, pair, group, and whole class interaction.
• Feedback, Participation, and Ideabox - Each class concludes with dedicated time for students to reflect on their own learning, self-assess their participation, offer feedback on the effectiveness of the instructors, and generate ideas for future class sessions. The instructors read this feedback each week, and often incorporate student ideas into subsequent classes. In one class students asked for more "secrets of the library" so we followed up and made this a recurring theme in future classes. In another, a student requested "more exclamation points" - so we complied with the request!!!! 

Part B

Next, consider what you learned from Small's article on motivation and address how you are going to motivate your learners/students

Selected quotes from the reading:

"... one widely accepted goal of education is to develop intrinsically motivated, life-long learners who not only want to learn but also enjoy the learning experience while it is occurring, and want to continue learning after the instruction has formally ended."

 Yes! This is always our hope in 101i! In addition to continuing their undergraduate research projects, our hope is that students will apply the skills, strategies, concepts and approaches they develop in 101i in their future coursework, continuing studies, and in their personal and professional lives.

"Attribution theory (e.g., Weiner, 1972; 1979), proposes that people will
ascribe one of four (two internal, two external) attributions to their success or failure at a task. These attributions are: ability, ... effort, ... task difficulty, ... and luck."

One might add "community" or "collaborative effort" to this list? But yes, we try to scaffold activities throughout the course to help students build on their successes as they engage in increasingly complex tasks.

Expetancy-Value Theory: effort requires "valuing the task" and "expecting to succeed at the task."

 Incorporating rationale for tasks (i.e., ensuring that students "buy-in" to the value of each task, and also see how the tasks build on each) is an important part of helping students "value the task." The scaffolding process mentioned above helps students develop a continuing expectation of success, even as tasks become more challenging as the quarter progresses.

Curiosity: "Zone of relaxation" | >> "Zone of Curiosity" << | "Zone of Anxiety" - Day (1982). "The 'zone of curiosity' is  characterized by excitement, interest, and exploration to resolve the conceptual conflict."

Flow Theory: flow experiences "... have three requirements: (1) appropriate level of challenge, (2) clearly defined goals, and (3) immediate and useful feedback on progress.

 As instructors, we're continually in the "Zone of Curiosity"  and we hope that is true for students, too! Modeling curiosity and enthusiasm for the research and creative process helps reduce anxiety, and hopefully improve the flow. I feel that student-student interaction can very positively impact both engagement and a sense of "flow" in a class, as can collaborative work around meaningful challenges.

The ARCS Model -
[A]ttention [R]elevance [C]onfidence [S]atisfaction - Keller

Sounds good to me! ;-)

Week 2 - from goals, to assessment, to activities

Part 1 - Procedures for Educative Assessment

Procedures for Educative Assessment

Formulate one or two ideas for forward-looking assessment. Identify a situation in which students are likely to use what they have learned, and try to replicate that situation with a question, problem, or issue.

In Honors 101i  we frequently ask students to reflect on their research question, and articulate its significance in the context of one or more audiences. The associated learning outcomes on the syllabus for these activities are:

• create and refine a research topic that is meaningful to you and compelling to others
• communicate your finished and unfinished ideas to others in a variety of ways

But, at the activity level, we might be able to specify these learning outcomes a bit more:

• Articulate the significance of your research topic for at least two different audiences.
• Imagine at least one practical application of your research project, and articulate the relevance of that application with a particular audience in mind.

(Note, we're drawing heavily on chapter 4 of Booth's The Craft of Research for this activity.)

For both a formative and forward-looking assessment activity, students first write an early draft of their topic, question, significance, and practical application. We currently break this down in their research notebook like this:

They do this activity outside of class time. Then, in class students verbally articulate the topic, question, significance, and practical application to a classmate, who in this case is an emerging expert in a different subject area. These activities serve as authentic, formative, and forward-looking assessments of some of the main learning outcomes of the course, which are to communicate research ideas to a variety of audiences, and in a variety of ways. The summative assessment here comes at the end of the quarter, when students give a short presentation of their work that includes the topic, question, significance, and practical application. In "real life" many of our students present their work at UCLA's Undergraduate Research Week, giving them an opportunity to apply these skills in a real world context. 

More forward-looking assessment ideas

One could imagine a yet further forward-looking assessment in which we might ask students to imagine that a legislator has called for the elimination of the funding for their department and/or research program, and called into question its significance, saying it has not practical application. Students would imagine themselves testifying at the legislative hearing, making the case for the relevance of their work.

Another possible forward-looking assessment might be to ask students to imagine they are interviewing for a job they hope to have after graduation. Students would role play the interview, in which they are asked to connect the dots between their undergraduate research work and the job duties and responsibilities.

Criteria & Standards

Select one of your main learning goals, and identify at least two criteria that would distinguish exceptional achievement from poor performance. Then write two or three levels of standards for each of these criteria.

Learning outcome:

Articulate the potential significance of your research topic for two different audiences.

For the purposes of setting criteria, let's improve this by specifying an audience. Maybe something like this:

Learning Outcome	Needs Substantial Improvement	Good	Excellent
Explain the significance of your research topic and question to an emerging expert in a different subject area.	Listener is unable to accurately summarize the main research question and may have trouble articulating its significance.	Listener is able to accurately summarize one or more aspects of the main research question.	Listener is able to accurately summarize the main research question. Listener is able to articulate what makes the research question interesting/significant. Listener may be able to draw connections between the speaker's research topic/question and their own field.

Self-Assessment

What opportunities can you create for students to engage in self-assessment of their performance?

This is an interesting question - there is a bit of tacit, informal assessment going on in this activity. As students work on trying to articulate their research question and its significance, they get feedback from their peers about how well their topic is understood. Their peers ask follow up questions, and either get interested (or not) in various aspects of their topic. All of this provides quite a bit of authentic feedback, and could be used as the basis for a reflective self-assessment. It seems that if we evolved the activity a bit, so that students had an opportunity to hear a peer summarize their question/signficance, then we could follow with a short reflective paragraph in which they self-assess their communication of their topic. How well did their listener understand their research question and its significance? What might they change about their explanation next time if engaged with a similar audience?

“FIDeLity”

Feedback What procedures can you develop that will allow you to give students feedback that is:

• Frequent
• Immediate
• Discriminating, i.e., based on clear criteria and standards
• Lovingly delivered

Because students have opportunities throughout the quarter to re-articulate their research question and its significance, we are able to give them frequent opportunities for immediate feedback. I think this feedback is almost always lovingly delivered, as the class thus far has always become a very constructive learning community in which students support each other. What could be strengthened quite a bit here is the "discriminating/based on clear criteria and standards" part. It seems we could engage students with the collaborative definition of standards for this activity, and have them self- and peer- evaluate at multiple points in the quarter. That would help improve the degree to which we have clear criteria and standards.

Part 2 - Integration

1. Situational Factors

Assuming you have done a careful, thorough job of reviewing the situational factors, how well are these factors reflected in the decisions you made about learning goals, feedback and assessment, learning activities? 

What potential conflicts can you identify that may cause problems? 

Are there any disconnects between your beliefs and values, the student characteristics, the specific or general context, or the nature of the subject in relation to the way you propose to run the course?

Last week, I explored the situational factors for Honors 101i, including the teaching and learning context and the characteristics of the learners and teachers.

In general, I think there is a good match between the situational factors and the learning goals, feedback, assessment, and learning activities. One of the most important situational factors is that this is a credit/no-credit course, so learner motivation is crucial, and it is important that the total workload in the course meet student expectations.

In contemplating a possible self-directed and/or online version of this course, we would want to take into account a set of special considerations for online learning. In particular, we would need to address the community-building aspect of the course that is foundational to our face-to-face course. While we might be able to form a similarly effective online community, we might also consider asking students to work with a research and writing partner throughout the quarter to give them frequent opportunities for collaboration and interaction.

Learning Goals and Feedback & Assessment 

How well do your assessment procedures address the full range of learning goals? 

The main learning goals of the course, from the syllabus are:

1. create and refine a research topic that is meaningful to you and compelling to others
2. develop strategies for finding and incorporating a rich variety of sources into your research projects, including rare and unique materials, images, sound, film, etc. 
3. document sources through citation and annotation 
4. experiment with digital tools to develop a personal approach to organizing your research process
5. understand your rights as a creator, and choose how you would like your work to be shared 
6. communicate your finished and unfinished ideas to others in a variety of ways
7. build a supportive community within our class, in which each person’s abilities to grow and learn are respected and encouraged

Of these, 1-4, and 6-7 are assessed formatively in the research notebook and class activities, and summatively in the written research proposal and final presentation. #5 is assessed informally in class activities, but could likely be better assessed as an aspect of the final research proposal.

Is the feedback giving students information about all the learning goals? 

Yes - weekly feedback in the form of comments on the research notebook, as well as peer and instructor feedback during in class activities address all of the learning goals (though, as mentioned above #5 could likely be better assessed). In an online and/or self-directed version of the class, feedback with FIDeLity (see above) would be significantly more challenging to provide. Peer feedback is a core aspect of the current face-to-face course, and this would likely need to be integrated even more in an online and/or self-directed version of the class.

Do the learning goals include helping the students learn how to assess their own performance?

One of the primary means of self-assessment (and student assessment of the instructors) comes in the form of a the weekly Feedback, Participation, and Ideabox activity. In this activity, students share their answers to the following questions in a form on the course Moodle website:

• What were the highlights of today's class for you? (e.g., something you learned, something you enjoyed, ....)
• What would have made today's class better?
• Please rate your participation for today: (a. Excellent; b. Good; c. Meh.; d. Very little or no participation.)
• Optional: If there is anything you would like us to know about your participation today, tell us here:
• Add to the class Ideabox... 
• What would you like to learn and do in future classes?
• Any other feedback or ideas for us?

The research notebook also offers weekly opportunities for reflection. There are a few self-assessment components integrated into the research notebook, but this is certainly an aspect of the notebook that could be strengthened.

Learning Goals and Teaching/Learning Activities

Do the learning activities effectively support all your learning goals? 

Yes! (I hope so). Though in future versions of the course I hope to strengthen student investigation of disciplinary writing, with a particular focus on investigating the writing strategies and techniques authors in their field use when incorporating sources into their writing.

Are there extraneous activities that do not serve any major learning goal?

We previously had a bunch of very fun activities in which students explored a very diverse set of source types (art, music, film, ....). While this was fun, each particular source type was more or less relevant to each student depending on their topic. More recently we have used this time to help students connect with the particular subject-specialists and source types most relevant for their own project. It would be great if we had enough class time to do both of these things!! It is not that the previous activities were "extraneous" - but more that we needed more time!

Teaching/Learning Activities and Feedback & Assessment 

How well does the feedback loop work to prepare students for understanding the criteria and standards that will be used to assess their performance? 

How well do the practice learning activities and the associated feedback opportunities prepare students for the eventual assessment activities?

In some cases we do this very well. For example, in the first class, students create a collaborative set of criteria for excellent participation and then apply these criteria to their self-evaluations throughout the quarter. In other areas, I think we could seek to improve. Perhaps we could follow a similar model in the peer evaluation of research presentations at the end of the course, for example.

Regarding practice learning activities, the research notebook and in class activities all lead directly into the creation of the final research proposal and class presentation. .So these practice activities are well aligned with the summative assessment.

Honors Collegium 101i - Research Today

Specific Context of the Teaching/Learning Situation 

How many students are in the class? Is the course lower division, upper division, or graduate level? How long and frequent are the class meetings? How will the course be delivered: live, online, or in a classroom or lab? What physical elements of the learning environment will affect the class? 

• Honors 101i (current syllabus)
• Seminar - Up to 20 students
• Undergraduate Course - Upper Division
• Currently class meets for 3 hours once a week for 10 weeks.
• Course delivery - Outside of class students develop a digital research notebook in which they explore and develop their topics and reflect on the research process through guided weekly assignments. Class sessions are highly interactive, and include a number of collaborative activities in which students share their research notebook progress, provide peer feedback and review, generate and share ideas.

Note - For this instructional design course, I am contemplating a number of possible curricular redesigns - including a self-directed course, a completely online version of the course, a more hybrid version of the current course, and a restructuring of the course from a single 3 hour session to two 1 hour sessions / week. I am also considering a new course text.

General Context of the Learning Situation 

What learning expectations are placed on this course or curriculum by: the university, college and/or department? the profession? society? 

The course is Credit/no-credit and is a requirement for participation and completion of the Mellon Mays Undergraduate Research Fellowship Program. It also provides 2 of the 12 upper division honors units required for graduating from the College of L&S with Honors. It seems particularly popular among transfer students seeking to graduate with honors.

Nature of the Subject 

Is this subject primarily theoretical, practical, or a combination? Is the subject primarily convergent or divergent? Are there important changes or controversies occurring within the field? 

The course has a very practical and applied approach in that the majority of the course is devoted to the research process, and the development of a research proposal. There are also several reflective elements built into the course.

Characteristics of the Learners 

What is the life situation of the learners (e.g., working, family, professional goals)? What prior knowledge, experiences, and initial feelings do students usually have about this subject? What are their learning goals, expectations, and preferred learning styles? 

Usually about half of the students are Mellon Mays Fellows, and the other half are students seeking honors credit with the goal of graduating with College Honors. Many of the honors students are transfer students. In my first two years teaching the course I have found each diverse group of students to be very motivated and excited to dive into the research process. Many have a mix of excitement and nervousness about embarking on such a big project, and we devote time in our early sessions to building a community of supportive engagement among our students.

Characteristics of the Teacher(s) 

What beliefs and values does the teacher have about teaching and learning? What is his/her attitude toward: the subject? students? What level of knowledge or familiarity does s/he have with this subject? What are his/her strengths in teaching? 

The class is currently co-taught by two librarians who each have a background in language instruction as well as the research and writing process. Because the students are researching topics across a broad range of subject areas, we often bring in subject experts to assist with aspects of the course and meet with students individually and in small groups. We also arrange a visit to Library Special Collections to foster an exploratory engagement with primary sources and rare and unique materials.

My personal instructional approach is guided by a focus on inquiry-based, active learning and a desire to help foster knowledge building communities. As a librarian, I believe our instructional work is most effective when we partner with students, staff, and faculty to facilitate diverse opportunities for learners to explore, create, and share knowledge in a variety of forms.  I actively embrace reflective instruction and assessment to better understand how, what, and why people learn. I am a firm believer that academic learning occurs within the context of the health and wellness of the whole individual, and view the library and our instructional practice as relevant and impactful across the full spectrum of the learning experience.

~~~~

Questions for Formulating Significant Learning Goals 

"A year (or more) after this course is over, I want and hope that students will .” 

Course learning goals from current course description:

Through active engagement in this course, learners:

• create and refine a research topic that is meaningful to you and compelling to others
• develop strategies for finding and incorporating a rich variety of sources into your research projects, including rare and unique materials, images, sound, film, etc. 
• document sources through citation and annotation 
• experiment with digital tools to develop a personal approach to organizing your research process
• understand your rights as a creator, and choose how you would like your work to be shared 
• communicate your finished and unfinished ideas to others in a variety of ways
• build a supportive community within our class, in which each person’s abilities to grow and learn are respected and encouraged

Foundational Knowledge

What key information (e.g., facts, terms, formulae, concepts, principles, relationships, etc.) is/are important for students to understand and remember in the future? 

What key ideas (or perspectives) are important for students to understand in this course? 

This is an interesting question to reflect on, and it makes me realize that there isn't a tremendous amount of "content" in the course. Instead, the course is largely built upon an exploration of process with a variety of activities designed to help students both diversify their personal toolbox for engaging in research and further explore, develop, and refine their personal research process. Key concepts might include an understanding of the fundamentally social and interactive nature of research, the importance of uncovering ongoing research conversations and seeking opportunities to contribute, and the development of a personal identity as a researcher and knowledge creator engaging in ever-evolving communities of knowledge and creative practice.

Application Goals

What kinds of thinking are important for students to learn?

• Critical thinking, in which students analyze and evaluate - Yes!!
• Creative thinking, in which students imagine and create - Yes!!
• Practical thinking, in which students solve problems and make decisions  - Yes!!
• What important skills do students need to gain?
• Iteratively revising and developing sets of research questions and areas of inquiry
• Identifying and engaging with guides and helpers in the research and creative process
• Critically evaluating creative work in the genre(s) in which they hope to create (usually written scholarship)
• Providing constructive peer feedback
• and more!!

Do students need to learn how to manage complex projects? 

Yes! Much of the course is devoted to the development of a digital research notebook, which is intended to be a preparotory / planning document for an undergraduate capstone research project.

Integration Goals

What connections (similarities and interactions) should students recognize and make...: 

Among ideas within this course?

Among the information, ideas, and perspectives in this course and those in other courses or areas?

Among material in this course and the students' own personal, social, and/or work life? 

Hopefully everything we do in this course is directly applicable to other courses in which students are doing research-based creative work (ok, that's a rather optimistic way of saying "writing papers").  We also hope that students will be able to take these skills and apply them later in life, whether in graduate school, on the job, or in personal life.

Human Dimensions Goals 

What could or should students learn about themselves?

What could or should students learn about understanding others and/or interacting with them? 

Through the course we strive for students to build on and and strengthen their identities as researchers and knowledge creators that actively engage and contribute to ever-evolving communities of knowledge and creative practice.

We also hope they continue to build on their skills as supportive contributors to the work of others, and constructive critics as they assist each other in the research process.

Caring Goals 

What changes/values do you hope students will adopt? 

Feelings? Interests? Ideas? 

For each of the past two quarters we have taught the course, a very empathetic and engaged learning community has emerged. Students truly support each other in their research endeavors, and become inquisitive about and invested in each others research ideas. We always hope to foster this community approach to knowledge creation, and the curious-minded interest in cross-disciplinary discussions. Not to mention that talking about your work with a group of helpful and encouraging peers can be fun!

"Learning-How-to-Learn" Goals 

What would you like for students to learn about:

how to be good students in a course like this?

how to learn about this particular subject?

how to become a self-directed learner of this subject, i.e., having a learning agenda of what they need/want to learn, and a plan for learning it? 

Yes - all of the above! This is really a central part of what the whole course is all about!