Design, Development and Evaluation of the Citizen Science Cancer Curriculum (CSCC): a Design and Development Case Study

When we asked the CSs about their motivation for working on cancer-related research projects, we found that their motivation was related to: a) a sense of social responsibility, b) the value they attribute to their work as CSs and c) personal stories and patient status. For example, in terms of social responsibility, one participant said that her motivations to work on cancer-related projects is because “cancer affects so many people”:

This is probably a general motivation that you'll hear from anybody, but cancer affects so many people. And so, you know, I have people on my personal life that have cancer, and have been fighting it and it just seems like such a horrible, horrible situation and our only hope is research, you know, and so it's kind of like whatever you can do to jump on board and help with the hope that we have is, is what you do. Yeah, that's, that's my motivation.

Another participant mentioned her own health condition, patient status and other personal reasons for her motivation (e.g., grandfather who died from leukemia, a cousin who died from breast cancer). She described her motivation as being “a personal thing”:

I've lost my grandfather to leukemia many years ago, I had a cousin who died from breast cancer a few years ago. And so [...] just being on the personal side a little more. […] This is why we're going to do that. So now [it] is a personal thing for me.

When we asked researchers to describe their motivation for working with CSs on cancer-related projects, they: a) mentioned how CSs can help bridge communication between scientists and the community, b) emphasized the need for accountability in face of existing disparities between communities and c) discussed the challenges of creating partnerships between researchers and communities to support robust cancer-related research. For instance, in relation to enhancing communication between the scientific community and community stakeholders, one of the researchers said:

Because I think a citizen scientist has the potential of being the connection with the community in the sense that we don't stay in those silos of the scientists over here or the physicians over here and then the community over there.

Another researcher described his views that, as a country, the U.S. innovates, but these innovations are often not accessible to the people who fund them or who are part of the community in which those discoveries are made.

We use this (cancer research) to better human conditions, but at the same time, there are some political things, some educational barriers in our society, which doesn't allow to apply this in our own population […] and our patients can simply not afford it. And I think the only way to change this on the long run is to really to have people, which are better than we as scientists and more centered in their communities. So to bring this up, this is nothing people talk about.

He mentioned the challenge of a scientist not being able to fully get the message in the hands of the community, which highlights why CS involvement is vital to mitigate communication barriers:

So that, that's why I think it's really [sic] to have that link and I think I want to help. I will […] if we find the right tool and I have confidence from people which work in the community that we're hitting the right tone, so that we make a difference […] if we can work together with a citizen scientist, then we can amplify these efforts because there's only so much you can do as a scientist.

As a culturally universal product (Landrum et al., 2019), storytelling has been used as a pedagogical approach across different disciplines and contexts, including science, social science, psychology and patient self-management (Landrum et al., 2019; Lugmayr et al., 2017). Storytelling supports instructional strategies because stories can be used to: a) create interest, b) provide a structure for remembering course material, c) share information in a familiar and accessible form and d) create a more personal student-teacher connection (Green, 2004; Landrum et al., 2019). Furthermore, stories provide relatable examples and memory links that support recall, learning and problem-solving (Hung et al., 2012; Jonassen & Hernandez-Serrano, 2002).

In addition to its pedagogical value, storytelling can also support motivation by helping learners: a) become involved in the learning situation (Hung et al., 2012), b) make connections with their personal experiences (Yang & Wu, 2012), c) develop/sustain positive attitudes toward science (Valle et al., 2021) and d) improve perceptions of social representation (Valle et al., 2020), task value and self-efficacy (Yang & Wu, 2012). In the context of the curriculum, the motivational role of storytelling was expected to support learners’ motivation to act (e.g., become involved in cancer-related research) based on how they perceive their agency over their learning (autonomy), ability to contribute to cancer-related research (competence) and how they perceived themselves as part of the multi-disciplinary research group (relatedness).

Each instructional video within a module includes clearly stated learning objectives and thought questions serving as pre-instructional strategies to heighten a learner’s awareness of the topics addressed within the lesson and to focus the learner’s attention on salient details and vocabulary (Hartley & Davies, 1976; Morrison et al., 2019). The writing of the learning objectives resulted from consultations with the subject-matter experts (SMEs) and followed best practices from instructional design literature. Each objective included actionable and measurable verbs connected to the subject matter under consideration. Following standard instructional design practices, the learning objectives were used to craft the embedded assessment items in an effort to clearly connect the objectives and assessment activities (Dick et al., 2015). In distinction, the thought questions were analogous to simultaneously provide an advanced organizer (Barnes & Clawson, 1975) for cuing the structure of the forthcoming materials and to provide relevance to the learners.

We intentionally designed two types of videos for the CSCC: 1) instructional videos and 2) supplemental videos. All videos were created following best practices and evidence and are intentionally short (no more than 10 min) on well-defined topics with controls for stopping, playing, rewinding and fast-forwarding (Fiorella & Mayer, 2018; Mayer et al., 2020). The instructional videos are integrated with the learning objectives, thought questions, interactive embedded assessments and reflection questions; and serve as the primary intended learning outcomes connected to the topics identified in the needs assessment. Each of these instructional videos are presented by a SME and are recorded on a green screen with PowerPoint slides designed with the inspiration from the multimedia principles from the Cognitive Theory of Multimedia Learning (CTML) (Moreno & Mayer, 1999). The SMEs were encouraged to use a conversational style in the instructional videos and avoid the use of extraneous details within their explanations (coherence principle). The video scripts were created with the SMEs and an instructional designer to achieve these objectives and the slides made use of relevant images (multimedia principle) inclusive of both representational and organizational pictures (Levin, 1981) with minimal verbal redundancy (redundancy principle). Appropriate slides design involved placing imagery with verbal explanations concurrently (spatial and temporal contiguity principles) and included cues (e.g., bold or italics) (signaling principle) to emphasize key points in the narrative (modality principle). While the inclusion of the speaker on-screen in an instructional video remains an area of ongoing research (Wang & Antonenko, 2017), our intentions were to showcase diversity (e.g., gender, ethnicity and age) among the speakers to reach a broad audience of CSs. In contrast, the supplemental videos provided additional enrichment on topics that were not essential to the training of a CS to effectively participate in cancer-related research projects based on our needs assessment. The supplemental videos were designed with the same guiding principles CTML, but did not include the interactive embedded assessments or always make use of PowerPoint slides. The supplemental videos enrich the curriculum with additional relevant examples and discussion points and intentionally involved the use of storytelling.

To support CS’s motivation and self-efficacy to contribute to cancer-related research, the instructional design team met with presenters of both supplemental and instructional videos to discuss how the content in the videos could highlight concrete examples (storytelling) and the role CSs can play as part of a multi-disciplinary approach in cancer-related research. For example, in the supplemental video “Mapping Cancer Health Outcomes and Disparities”, the presenter introduced the topic by saying:

For cancer population sciences, maps are very important for describing and understanding the disparity in cancer health outcomes of our communities. As a Citizen Scientist, your research team is likely to be using maps from the literature and generating your own maps from public health data.

While we used storytelling and the role of CSs as important stakeholders in those narratives to support CSs’ sense of belonging (relatedness) and self-efficacy (competence), we also intended to support CSs’ autonomy through the user control and self-paced nature of the curriculum. In attending to CSs’ sense of belonging, self-efficacy and autonomy, we enhanced the alignment between our instructional design strategy and the key psychological needs described by the SDT framework (Ryan & Deci, 2017).

Using the learning objectives as the blueprint, we created assessment items for each of the instructional videos within the CSCC. The creation and use of the assessment items were based on research and evidence on providing learners distributed practice experiences (Roediger & Karpicke, 2006) clearly aligned to learning objectives. For these assessments, we employed five multiple-choice questions on each practice assessment that combined recognition, analysis and evaluation of the content covered in each video. The combination of different types of questions was used to create a balanced learning experience where learners could contextualize the content and develop higher-order learning outcomes (Buchanan Hill, 2016). To illustrate how these types of questions were implemented, we provide a few examples from the assessment related to the instructional video “Causes of Cancer” in Table 2 with the correct response presented in bold.

To support CSs’ conceptual learning (Finn et al., 2018) and transfer to novel contexts, all assessment items had elaborated feedback for both correct and incorrect responses. Since feedback is a powerful learning aid, we chose to incorporate the feedback messages for all of the multiple-choice questions (Shute, 2008). All assessments were carefully reviewed by the team to ensure best practices in multiple-choice item-writing (Miller et al., 2013) and were also reviewed by the SMEs to ensure accuracy of the content.

We intentionally included videos of the CSs sharing their motivation, personal experiences and how those aspects influenced their interest in cancer-related research. The CS spotlight videos create authenticity and relevance for the learners and can instill in them a sense of belonging within the learning community (Balyasnikova & Gillard, 2018; Delmas, 2017). Similar to the approach used for the supplemental and instructional videos, the instructional design team met with each CS featured in the “Spotlight on Citizen Scientists” videos. The goal of those meetings was to provide background information related to the video production and prepare CSs for the general questions that would be asked to guide the conversational style intended for the video format. The questions asked included: a) what is your motivation for working on cancer-related research? b) can you please talk about your previous experiences with cancer and some lessons learned through those experiences? c) how do these experiences influence your work as a CS? In addition to these more generic questions, other questions were developed based on CSs’ initial narratives. For example, one CS described the importance of humanizing the patient. This prompted the design team to address this topic in the final video recording with the question “Based on your experience, why is it important to humanize the patient?” The “Spotlight on Citizen Scientists” videos were recorded using a video conference platform (Zoom) and ranged between 7:35 and 8:20 min in length. The edited final versions of the videos only show the CSs telling their stories and addressing related questions.

The patient case study video series was informed by the conceptual framework to a motivational and emotional account of the importance of cancer-related research and the critical role CSs play as advocates in this process. We intentionally captured the stories from a cancer survivor, his caregivers (i.e., parents) and the physician who provided the treatment to humanize the CS role in cancer-related research using the notion of storytelling. The development of the patient case study video series required two meetings for preparation: one with the patient’s physician and a second meeting with the physician, the patient and the patient’s parents. In the first meeting, the physician described the case from his perspective and how the patient was able to enroll in an experimental clinical trial. In the second meeting, the patient and his parents described the family’s journey from the initial onset of symptoms through the diagnosis and treatment of the patient’s cancer. These accounts helped the instructional design team identify key points that could be highlighted across six segments of the final version of the case study. Two segments addressed the patient’s perspectives and focused on his experience growing up with, receiving treatment for and becoming a survivor of cancer. Three segments focused on the physician’s narrative of the diagnosis, treatment plan, opportunity for an innovative cancer treatment and the collaboration between researchers and the patients’ health care providers and family. Finally, one segment focused on the patient’s caregivers and how they relied on extensive supportive systems (e.g., family, friends, doctors, nurses) to navigate the challenges imposed by a cancer diagnosis. The caregivers detailed how this support system was critical as they cared for their child and enrolled him in the clinical trial that ultimately contributed to long-term remission of the cancer and survival of the patient.

To ensure that the rights and well-being of the patient and his family were protected and that the project followed the most ethical approaches and guidelines, our team contacted the university’s cancer center office of communications at the planning stage of the project. As part of this process, the cancer center required the use of a consent form that the patient filled out with help from a trained communications person not involved in the development of the curriculum. The form was filled out prior to the shoot and before any identifying information about the participant was published in any format.

While the initial evaluation during the needs assessment stage served to support the rationale for the design and development of the CSCC curriculum, the formative evaluation based on qualitative and quantitative data was important to examine CSs’ affective and content-specific outcomes. This multi-method approach has been used in other design and development research studies (Richey & Klein, 2014). In terms of evaluation models, the formative evaluation reflects the reaction- and learning-level described in the Kirkpatrick Model, where learners’ reactions to the training (e.g., CS satisfaction with the instructional approach) and learning outcomes (e.g., CSs can identify the main causes of cancer) are assessed (Jones et al., 2018). The following objectives were considered based on the intent of the course and its design to support CSs’ motivation and learning outcomes: 1) the CSCC incorporates sound instructional design informed by the conceptual framework, 2) the CSCC supports the learning and application of cancer-related concepts among the CSs and 3) the CSs’ motivation to engage in cancer-related research projects is positively shaped by the design of the CSCC and its affordances. To assess the extent to which each of these objectives was achieved, we employed a range of formative evaluation techniques with the CSs to assess the resources within the CSCC using an iterative process of creation, evaluation and revision by the team and related stakeholders (e.g., SMEs). The last two levels of the Kirkpatrick Model (behavior- and results-levels), which assess how the training translated into behavioral changes (e.g., continuing engagement of CS with cancer research) and the outcome of the training (e.g., successful collaborations between CSs and cancer researchers), respectively, were not addressed in the present study.

The same seven CSs (2 Males, 5 Females) who participated in the needs assessment also participated in the design, development and formative evaluation (between five and seven participants in each session) of the CSCC. Their experience working as CSs ranged between one and five years. Following the research protocol approved by the university’s institutional review board (IRB), we collected data on CS’s learning performance and perceptions of the course and its learning materials via surveys in Qualtrics™. The formative evaluation of the instructional videos was conducted in nine separate instances, one per instructional video and included five-point Likert-type questions (1. Strongly Disagree to 5. Strongly Agree) to address the quality of assessment items (e.g., I think the assessment items were easy to understand) and instructional videos (e.g., The presenter did not use too much technical language or jargon). Standardized Cronbach’s alpha values for the quality of assessment items scores (α = 0.85) and for the quality of video scores (α = 0.83) indicate good internal consistency. Open-ended items for additional feedback were also used for additional insights.

To understand how the patient case study video series influenced CSs’ general interest/motivation, relatedness, competence/self-efficacy and autonomy as they related to cancer research, we asked four open-ended questions, which were presented in the survey along with the six videos associated with the patient case study:

As the CSCC project was implemented in the 2020–21 academic year, the COVID pandemic shaped our approach to creating and evaluating the resources as we had to ensure the safety (e.g., adherence to Centers for Disease Control and Prevention [CDC] guidelines) of all stakeholders involved. Consequently, the videos and assessments were recorded and created out of order and evaluated asynchronously by the CSs using the Qualtrics survey platform.

As the CSCC is an interactive resource with the option of both linear or non-linear navigation of the curriculum resources, we elected to seek the expertise of a user experience design professional to conduct think aloud analysis of the final CSCC prior to production release. The user experience design professional has more than 20-years of experience developing interactive learning resources. The user experience design expert carefully reviewed the full CSCC resource while using a think aloud approach as a video recording. The full think aloud video lasted approximately 30 min and attended to a range of instructional design, usability, navigation and accessibility thoughts on the CSCC. After receiving the video from the user experience design expert, the team carefully reviewed the points raised and made slight alterations (e.g., changed lettering, formatting and navigation) to the CSCC. This review served as the final check-point before the final production release of the CSCC.

Quantitative and qualitative results from the formative evaluation sessions indicate that the curriculum and its learning materials where helpful to support CSs’ learning outcomes and motivation. More specifically, all instructional videos (IV) and related assessments presented over 85% of correct responses as shown in Fig. 3.

Even when we consider specific items within each module, most items presented greater than 70% of correct responses. Only two instructional videos (Palliative Care and Clinical Trial Enrollment) had one item each with a lower percentage (see Fig. 4). These data suggest a clear alignment among the learning objectives, the instructional videos and the practice assessments.

CSs’ learning performance in the assessment items indicates that they comprehended the topics with a high level of mastery and were able to apply the concepts covered in the modules to the criterion-referenced assessment items. This result aligns with CSs’ overall positive feedback of the assessment items and instructional videos (Fig. 5), which highlights the importance of involving CSs in the instructional design process from the onset of the project. Note, the quality of the assessment items and the videos were rated on a five-point scale.

CSs’ critical reflections were also present in the data. Their reflections were particularly helpful to guide changes regarding content presentation and scaffolding. For example, for the Clinical Trial 101 module, a CS mentioned:

I also feel that more explanation should be given to Phase 0 as this is seldom mentioned, in studies, and how is it determined if the drug [works] or is useful to go one [sic] studying in phase 0???.

Similar critical feedback was provided for the assessment items, which was used to make useful changes to the language and presentation of these items. For example, for the module about the Importance of Social Determinants of Health (Instructional Video 6), a CS said:

The first assessment asking for regurgitate [sic] a statistic seemed a little unfair. I didn't think the point of the video was to remember if it was 17% or 71% but to recognize that there is a relationship.

The feedback provided during the formative evaluation were essential to guide revisions to assessment items and for the final recording of the instructional videos. The comments and suggestions provided by CSs were addressed in the slides and verbal remarks. For instance, in light of their comments about the phases of clinical trials, we added about 30 s of content, especially regarding phase 0, as it was essentially a passing mention earlier. Initially, we considered the information to be only a supplemental component, but since the CSs asked for more detail, we changed course and added more information in the presentation.

CSs’ feedback about the instructional videos indicate that their motivation was also supported by the interaction with the instructional materials. CSs were particularly appreciative of the attention presenters gave to their role as CSs, which fostered a sense of belonging and competence in their work as CSs, as described by one participant:

I felt valued as a Citizen Scientist and enthusiastic about my role in the research process.

However, when they did not feel the connection between the content being presented and their roles as CSs, they also voiced their concerns as illustrated by the feedback provided by another CS:

The information was clear, easy to follow and comprehensive. The actual ROLE of Citizen Scientists did not seem to be addressed.

Importantly, the CSs really appreciated the use of stories in the videos and assessment items to contextualize the concepts covered as seen in the following suggestions by a CS:

I think the scenario questions were particularly helpful.

Think there could be a few more scenarios to further ascertain understanding of the video material.

The feedback from CSs about the patient case study video series indicate an overall approval of the storytelling approach as an instructional strategy to support their motivation to work on cancer-related research. The feedback was mostly positive (83%) and outlined four emerging themes: a) importance of information and research, b) positive outcome, c) multiple stakeholder engagement and d) role of CSs (Table 3).

It is worth mentioning that there were a few negative (17%) and neutral (29%) types of comments as well. For example, one CS mentioned that despite the story’s happy ending, they did not perceive a connection with the role of CS in cancer research:

I do not see any empowerment related to my role as Citizen Scientist in this particular patient study!! I feel happy that the young man finally made a connection that worked to save his life.

The negative feedback suggests that the CSs felt entitled and comfortable to share their concerns when their expectations regarding the quality of the learning material and instructional approach were not met.