Product Designer & Prototyper | Kaboom: an inquiry-based Virtual Learning Environment | Harvard Graduate School of Education
Aug 2020-Dec 2020
Team: Kexin Kang, Lydia Y. Cao, and Jaweria Sethi
INTRODUCTION
The pandemic has changed the global education landscape. Luckily, today’s technology offers far more connectivity and flexibility than we have ever experienced. However, we have not yet updated our learning models to harness the power of the Internet. Research has shown that collaboration, choice, and relevance are critical to enhancing learning engagement. The reality is that much online learning focuses on transmission. Such passive learning models fail to engage learners.
We started our innovation journey by interrogating the underpinning pedagogical principles and features of the existing virtual learning environments (VLE), such as Moodle, Canvas, Blackboard and D2L. According to Martin Weller (2007), the main learning and teaching functions of VLEs are:
• Content delivery
• Asynchronous discussion
• Online assessment
• Student tracking
• Synchronous discussion
• Student tools
Depending on how these features are used, VLEs can support various pedagogies, such as instructivist learning, project-based learning, and peer learning. We noticed that most VLE designs are teacher-centered, used mostly for instructivist learning. Most VLEs function as ‘content management systems’ that focus on the allocation of resources and keeping track of ‘transactions of learning’ rather than giving agency for personal meaning-making, co-construction of understanding and community building.
We introduce Kaboom, a new model of VLE to equip children with 21st-century skills (i.e. cognitive skills, interpersonal skills and intrapersonal skills) (Bransford et al., 2000). Kaboom provides an inquiry-based virtual learning environment for children at age group 10-16 to solve real-world problems and create social impact. At Kaboom, learners start a project or join one that has been initiated by other learners. They can then choose an inquiry guide for the team, who is an expert in their area of inquiry, to guide and provide feedback for the learning process. Now, they are ready to embark on their inquiry journey and make real impacts in the world.
IMPACT PROMISE
Kaboom aims to create impact in three ways.
First, Kaboom helps to enhance student’s engagement by making learning more relevant, meaningful and exciting. Second, Kaboom equips children with 21st-century skills by empowering them to become active citizens to make the changes that they want to see in the world. Last but not least, Kaboom helps teachers to adopt inquiry-based learning pedagogy with ease.
HOW ARE WE GOING TO EVALUATE OUR IMPACT?
Impact Promise 01: Enhance Student Engagement
How it looks like:
Learners might talk about their projects with friends and parents at home, actively spend their free time working on the project, and actively ask their inquiry guides for assistance. We expect teachers to report more enthusiasm and active engagement in students and improved learning attainment. We expect schools to report increased attendance rates. As shown in Figure 2, we can see that the frequency of students' visits to VLE has increased significantly after shifting their VLE to Kaboom.
Figure 2 student visits to VLE before and after using Kaboom
Impact Promise 02: Support Learners to Develop 21st-Century Skills and Become Active Citizens
How it looks like:
We expect to hear testimonials from students as the following.
“Kaboom is helping me to become an active and engaged citizen of the world. We learn so many global issues in class, but it feels like we can’t do anything about it, which makes me sad and stressed. Kaboom connected me with like-minded peers from different parts of the world. Together we studied the AIDS pandemic in sub-Saharan Africa with our inquiry guide who is a Ph.D. student in epidemiology. We made videos with our grandma on social media as part of the grandmothers-to-grandmothers campaign, which allowed us to raise $8000 for the Stephen Lewis Foundation.”
“I like Kaboom because it makes your learning real. I am motivated to learn because I see myself making a difference. Our team learned about the complexity of the ecosystem, the interdependence of different species, and how our daily consumption habits are endangering animals and their habitats. Our inquiry guide, Jim, who is a volunteer in the Orangutan Foundation opened our eyes by sharing his experience working at the reserve. We were able to share our learning with our local communities to raise awareness and change our consumption habits.”
Impact Promise 03: Support Teachers to Adopt Inquiry-Based Learning
How it looks like:
We expect to see changes in classroom practice in teachers. A teacher’s testimonial might look like:
“It has always been a challenge to conduct genuine inquiry-based learning with my students since there is just so much to manage at the same time. Kaboom has made it so much easier by providing an overarching structure and tools for children to organize themselves. I also really appreciate the extra help I get from the inquiry guides who volunteer their time to share their expertise with children. It is like having parent volunteers in my classroom all the time!”
RESEARCH FOUNDATIONS AND THEORIES OF ACTION
Inquiry-Based Learning with Guidance
As research has found that inquiry-based activities can boost students’ learning in a wide range of school subjects, we had the idea of incorporating inquiry-based learning into our VLE.
There is evidence that inquiry-based learning can motivate students to learn and advance their problem-solving and critical thinking skills (Lazonder & Harmsen, 2016). However, the effectiveness of inquiry-based learning largely depends on the guidance provided by teachers. Poorly guided and unstructured inquiry may lead to unproductive floundering and frustration when students have limited prior knowledge and experience (Nesbit & Qing, 2014).
Scaffolds
During an inquiry, scaffolds can be used to enhance pupils’ understanding and engagement (Saye & Brush, 2002; Simons & Klein, 2007). According to Saye and Brush (2002), scaffolds can be divided into hard and soft scaffolds. Hard scaffolds are static supports that can be planned based on experience. On the other hand, soft scaffolds are dynamic and contextual. Soft scaffolds demand teachers to diagnose the understandings of learners and to provide feedback on the fly based on student responses. Effective scaffolds build upon pupils’ prior knowledge and to provide support in their zone of proximal development (Van Uum et al., 2017). Teachers need to support pupils’ progress in the different phases of inquiry, such as focusing on the acquisition of conceptual or procedural knowledge (Van Uum et al., 2017). Inquiry-based learning with appropriate guidance has a huge demand for teachers’ attention, which makes it challenging to adopt with a class of learners with various interests and needs.
Collaboration
Social interdependence theory (Johnson & Johnson, 2008) predicts that cooperative goal structures will result in higher achievement than competitive or individualistic goal structures. Cooperative goal structures tend to result in promotive interaction, as learners provide assistance, information, and resources needed to achieve their mutual goals. On the contrary, competitive and individualistic goal structures result in oppositional or no interaction. It may be expected that cooperative goal structures will result in higher achievement than competitive or individualistic.
Relevance
Keller (1987) identified the concept of relevance as a key component of motivation in his ARCS (attention, relevance, competence, satisfaction) model of motivation. Relevance is viewed as an important component of motivation because relevant tasks and content satisfy students' needs, such as the need for power, the need for achievement, and the need for affiliation. As these projects allow students to find solutions for real-world problems, work in cooperation and, perform realistic models, they can increase the interests of the students in the subject of learning (Blumenfeld et al., 1991). Whenever students are given choices and responsibilities to assume, they show progressive motivations (Paris & Turner, 1994).
THEORIES OF ACTION
FEEDBACK LOOP
Measure Student Engagement
We can observe behavior changes in students directly by analyzing their log files generated from Kaboom. For example, we expect to see an increase in the frequency of VLE visits after using Kaboom. We also expect to see more meaningful interactions among learners, such as their activities in the discussion board. Other possible indicators include:
• The number of projects created.
• The number of students on the platform.
• The number of tasks created and completed on the dashboard.
• How the docs and files sections are organized.
• The number of voluntary check-ins by each student over a course of a project.
• The number of hours of chat on the ‘campfire’ (informal chat).
Measure the Development of 21st Century Skills
Since the development of 21st-century skills takes time and is complex to measure, we can evaluate our impact by looking for proxies. For example, learners might describe how they leverage collective intelligence in a team to solve real-world problems. Parents might describe that their children apply design thinking principles in play and their daily lives. Teachers might report that students become more mindful of their actions in a group setting after they participate in a Kaboom project. We can measure the development of 21st-century skills indirectly by conducting in-depth interviews with students, teachers, and parents.
Measure Changes in Classroom Practice
Changes in classroom practice can be measured by direct classroom observations, surveys of students, self-report of teachers and school administrators.
Sample survey questions:
For schools
1. How is Kaboom used at your school? (e.g., as an extracurricular? as part of science classes?)
2. Has Kaboom changed the culture of learning at school? If so, how?
3. How hard has it been to integrate the Kaboom within the existing schedule?
For educators
1. How does Kaboom influence your workload?
2. Has Kaboom simplified the process of conducting inquiry-based learning?
3. Have the teachers experienced a change in classroom culture?
4. Have the student-teacher relationships improved?
5. Has the quantity and quality of student work improved?
6. What is your experience of having inquiry guides in your classroom?
For students
1. How does your teacher teach before using Kaboom? How does your teacher teach now?
2. Tell us what you like and dislike about Kaboom.
3. How does Kaboom change your learning experience?
4. Tell us one project you made in Kaboom and what you think.
We plan to use a mixed-method approach to measure our impact. Surveys would give us an overarching picture, and individual interviews and focus groups would provide us with a more detailed and precise representation of users’ perspectives. Triangulating the feedback from learners, parents, teachers, and school administrators, we will have a comprehensive picture of children’s learning experience in Kaboom to refine our design.
IMPLEMENTATION GUIDANCE
Conditions of Success:
• Access to computers and high-speed internet. ~100 MBS.
• Basic computer literacy.
• Access to data storage.
• Access to copyleft resources for advancing learning.
• Clear terms and conditions for operations, mutually agreed upon by all stakeholders.
• Clear channels of communication between all stakeholders (i.e. children, educators, parents, schools, and the product managers).
• Accommodations for working asynchronously and synchronously across time zones.
• Clear curriculum standards that can be aligned with the projects.
• Integration of the learning outcomes with the grade-wise progressions.
• The willingness and ability of the school/decision-making authority to learn in interdisciplinary settings.
• Availability of the time to incorporate this module within the school schedule/after school.
• Opportunities for teacher training and professional development.
• Willingness and ability of educators to participate in professional development.
• Appreciation of inquiry-based learning as an authentic learning experience
• Safe and secure learning environment. (i.e, access is limited to teachers and students, and every interaction is monitored and moderated.)
• Clear guidelines and systems for resolving conflicts.
• Involvement of field experts who continually evaluate and improve the offering.
• Systems of evaluation and assessment, to assess and improve teaching practice.
• Systems of evaluation and assessment, to assess and improve the progression of individual projects.
• Systems of evaluation and assessment, to assess and improve the learning model.
TWO INTERFACES OF KABOOM
Kaboom VLE has two interfaces:
1) Team building interface:
Learners can view the ongoing and past projects, initiate a new project, and recruit team members. The projects will be cataloged and the users can search them through a range of filters (shown in Figure 3).
Figure 3 shows the team building interface in Kaboom
2) Inquiry interface:
This is the collaborative space where inquiry takes place. The dashboard hosts a tool suite to help learners to manage group discussions, distribute tasks, organize documents, keep track of deadlines and make announcements. Figure 4 shows our first prototype for the inquiry interface. We decided after two iterations that we will piggy-back the basecamp to test our ideas first before developing our own platform, as shown in Figure
Figure 4 shows our first prototype for the inquiry interface
Figure 5 shows an example of a Kaboom project in basecamp
How it will work
1) Team building interface:
Schools who wish to use the platform will register their interest and sign terms of use contract.
They will then be given access to the first admin account, which can be used to create further user accounts - admin, teachers, students and parents. All accounts will have a different set of permissions that can be easily modified.
They will then be given access to the first admin account, which can be used to create further user accounts - admin, teachers, students and parents. All accounts will have a different set of permissions that can be easily modified.
• To implement the program, we will conduct the following training:
• System implementation training for the administrative staff at the school.
• Operations training for the educators, parents and students to navigate through the team building interface and the inquiry interface
• Professional development for teachers
Families who wish to use the program independently will be able to create personal accounts by navigating through the platform. We will create learning guides for parents who are keen to understand the pedagogical framework. Other than that, we will develop short ‘how-to’ videos for parents and students to make the best use of the program.
The helpdesk will curate frequently asked questions to create a database of queries that regularly emerge. Our team will use this information, along with site analytics and user experience surveys to iterate the design.
The helpdesk will curate frequently asked questions to create a database of queries that regularly emerge. Our team will use this information, along with site analytics and user experience surveys to iterate the design.
2) Inquiry interface:
How does inquiry work in Kaboom?
ITERATION HISTORY AND PLAN
Iteration 01: Testing with Students at Harvard Graduate School of Education (HGSE)
Our first design was tested with the most accessible audience, our instructors and cohort at HGSE. We learned valuable lessons not only about our first testable but also what makes a testable that is most helpful to guide our thinking. We summarized our first lesson as the following.
Less is more
We were very excited to make our ideas more concrete, and we rolled out a prototype with many pages and features without testing the validity of our ideas. We learned that it is easy to fall in love with the solution rather than the problem once we spend a lot of time working on a prototype. It is more helpful to make a very primitive prototype to test. Testing ideas and usability tests are two different things!
Start with one feature, test one button!
Talk less and listen more
The first time we did the test, we did most of the talking. We were excited to explain our ideas and the features, and the first thing we did is to explain everything. We learned soon that doing a test is quite the opposite – let the audience do the talking, so we can see from their point of view without justifying our solution.
After our first testable, we adjusted the way we do our testing. We learned a lot by listening rather than explaining. Things we learned:
• The audience is not sure about the purpose of our platform. Some thought we were organizing field trips.
• The vocabulary we used was not clear to our audience (e.g., mentor, mission)
• The value proposition is not clear
• The targeting audience is not clear
• Busy and overwhelming interface
Modifications:
• Modify the language (e.g., change mission to project)
• Add an intro video to explain the purpose of the site and make a value proposition
• Clarify targeting audience, i.e., students from age 10-16
Iteration 02: More Testing with HGSE Students and an Expert
Students:
Our cohort reminded us that the design of the product needs to be more children-oriented since they are our main targeted users. For example, the description of the project might be too long for children. They are also concerned about the safety of children’s personal information. They suggested that instead of the full name of the children, the page can only show their first name or initials. Furthermore, our cohort commented that we need more information on our landing page so the users will understand the purpose and function of the platform. Having a video is not enough since users might not see the video. They suggested having both text and video to accommodate the preferences of different users. Things we can add on include a summary of previous projects and a scrollable home page. They also pointed out that she was confused about our targeted age group because it was never clearly identified in our info video.
Expert:
We showed our prototype to an expert researcher in education. He thinks that the idea of Kaboom has potential to connect children around the globe to have meaningful interactions, which indirectly promote intercultural understanding and global peace building. He affirmed the overall idea of Kaboom and emphasized the importance of pedagogical framework underpinning the platform.
Iteration 03: Testing with Secondary School Students
Since the audience of our first two iterations are adults, we are curious about what children think about our design. We decided to not modify the design at this stage and test our idea with students.
Testing with two students from the US
• Students enjoyed the color of our site and found the design intuitive.
• Students enjoyed reading about other children’s project and would like to know more (e.g., why a particular child wants to start a project and more about other learners)
• Students prefer longer projects over shorter ones.
• Students wondered if there can be an option to quit if they are not getting along with the team.
• Students want additional features (e.g., a virtual whiteboard on the dashboard to keep a track of mind maps visually)
• In case of conflicts, the student wanted a third-party mediator to lead the reconciliation (not the inquiry guides)
Testing with two students in Pakistan
• Both students were able to understand the concept of the platform after viewing the video.
• They wanted to see some past projects before joining/initiating a new project.
• They were excited about the idea of pursuing a real-life project but were worried about the real-life implementation. They see real projects entail funding.
• One student suggested holding competitions within the platform to pick the best projects and raise funding for those projects. He made a reference to Elon Musk and how he often supports student-run projects.
• Both students asked whether projects can also be initiated by educators as well as some children feel more comfortable with learning when there are clear expectations from teachers.
• While the students seem to not mind the amount of text (as they preferred to have more information to make an informed decision), they wanted the text to be organized in a way that makes it easier to read (e.g., with headings and bullets).
• They are willing to work with peers from different parts of the world. They wanted to chat with their peers before starting the project. They wanted to know more about the strengths of each individual who had already joined the project before making their decision. They said they would prefer to read testimonials/reviews of people who had previously worked with that child.
• They did not appreciate the star rating system. They said it can cloud their judgment to see the overall number of stars.
• Students expressed willingness to use Kaboom but did make a reference to time constraints with existing homework and school-related commitments.
• The students didn’t find the dashboard to be overwhelming though they didn’t like the colors as they found it dull. They appreciated the widgets on the dashboard and their ability to move around/resize different cards based on their needs.
Testing with a student in China
• The student expressed a strong interest in using Kaboom for both in-class and extracurricular activities. She is interested in working with children from all over the world. She also prefers to work on a project initiated by children than those by adults
• On the project research page, she suggested adding a searching box so the users can directly search the projects they like with keywords. Moreover, she wants to see more instructions on each project such as possible solutions to the problem.
• In terms of the inquiry guide, the student pointed out that besides having group meetings, she also would like to have individual meetings with the inquiry guide. Based on her schedule, she wants to meet the inquiry guide once a week.
Feedback from children affirmed the validity of our ideas. The way they view Kaboom is quite different from what adults assume. Children seem to be very interested in working with their peers on an authentic task. However, we wonder what it would look like if we have a group of learners to try out a Kaboom project? Would the positive feedback uphold? What are the challenges and issues that we haven’t anticipated? So far, we have focused on testing the idea. To further test the validity of our idea, we should move on to a user trial. We can simulate a Kaboom experience for a group of learners and an inquiry guide and test how inquiry in Kaboom unfolds.
Iteration 04: Testing with Potential Inquiry Guides
Before conducting the user trial, we decided to test our ideas with potential inquiry guides. We had interviews with a junior lawyer and an architect. During the test, we showed them the intro video of Kaboom and our prototype.
The junior lawyer thinks that this is a great opportunity for professionals like her who would like to teach without necessarily obtaining a teaching certificate. Since the inquiry is conducted remotely, she found it easy to fit Kaboom into her schedule. She wondered how we could make it accessible to children who cannot afford it, and gave us the idea of offering free spots in every team. The junior architect was intrigued and expressed interest in being part of the beta version. She also found it easy to incorporate Kaboom into her schedule since the hours are flexible. She wondered if we will provide her with some training before working with children. Both of them said they were more motivated for the cause than for the remuneration.
During the gallery, one of our classmates also wondered if we could make the mentor system volunteer-based. He thinks a lot of parents would want to get involved if their children participate in projects in Kaboom. The expert also mentioned that there are many retired professionals who would volunteer their time to impact the next generation. The design for the equity panel in class made us realize that equity should be engineered into the features and incorporated into the design rather than staying as an aspiration. We are now thinking about whether we can have the platform free to children if we make the inquiry guides volunteer-based.
NEXT STEPS
The figure above illustrates our iteration history and next steps.
User Trial
So far, we have tested the viability of Kaboom among users (learners and inquiry guides). The feedback we have got so far is on the positive side. What would it look like if we have a group of learners to try out a Kaboom project? What are the challenges and issues that we haven’t anticipated?
We decide to further test the viability of our idea and improve our design by having a user trial. We have a group of volunteer children (mostly connections from our immediate circle upon their consent and their parents’ approval) from different parts of the world who would like to try Kaboom together. We are going to simulate a Kaboom experience without developing a fully-fledged platform. We are going to have a meeting with one of our classmates to brainstorm ideas to develop a low-fidelity model, which would allow us to simulate a Kaboom experience with authenticity. During the trial, we will be looking for things such as:
1) How do children interact with other children whom they haven’t met. How do they build connections and establish trust?
2) How do children use the platform? What features do they use/not use?
3) What are the scaffolds and support that children require? Additional features? Improving existing features?
4) How do children position Kaboom in their daily life and learning?
5) What qualities are required of the mentor to facilitate a child-driven inquiry?
Apart from targeting questions, we will make observations in an organic manner, keeping an open mind for surprises.
We would also like to interview the participating children’s parents when possible. We would like to know how they view Kaboom (e.g., suggestions and concerns) and whether it has brought any positive changes to their children, such as,
1) Do the children engage in deep thinking and conversations with their parents?
2) Do the children talk at home about the projects they are working on?
3) Would the parents want to volunteer to be a Kaboom inquiry guide?
Testing with Teachers and School Administrators
Incorporating a VLE like Kaboom into daily learning is beyond the pedagogical scope and user experience, which are what we have been focusing on so far. According to the EdTech Genome Project, factors such as competing priorities, professional development, school staff culture and adoption plan are among the top 10 critical considerations for a successful implementation of an EdTech product into school(Molnar, 2020).
The next step is to test with teachers and school administrators to gauge their view on a student-centered VLE like Kaboom.
During the testing with teachers and school administrators, we would like to understand:
1) school/district vision for teaching and learning with technology
2) current VLE adopted by school and reasons for adoption (e.g., advantages and disadvantages; pain point with the current VLE?)
3) current priorities for schools and teachers
4) alignment between Kaboom and the national learning standards (e.g., NGSS)
5) school and district policy for ed-tech (e.g., regulations and safety).
6) technology infrastructure and availability of operational tech support
7) teachers’ pedagogical and technology proficiency
8) teacher beliefs and attitudes towards inquiry-based learning and ed-tech
9) teachers’ pain points during inquiry-based learning
10) perspectives of teachers and schools on incorporating “inquiry-guides” into their class
Depending on the feedback from the two types of testing explained above 1) user trial; 2) testing with teachers and school administrators, we will then decide whether to persist in the current direction (i.e. have Kaboom as a new model of VLE) or pivot.
MOST RECENT PROTOTYPE
REFERENCES
Blumenfeld, P. C., Soloway, E., Marx, R. W., Krajcik, J. S., Guzdial, M., & Palincsar, A. (1991). Motivating Project-Based Learning: Sustaining the Doing, Supporting the Learning. Educational Psychologist, 26(3–4), 369–398. https://doi.org/10.1080/00461520.1991.9653139
Bransford, J., Brown, A. L., & Cocking, R. R. (Eds.). (2000). How people learn: Brain, mind, experience, and school (Expanded ed). National Academy Press.
Johnson, D. W., & Johnson, R. T. (2008). Social Interdependence Theory and Cooperative Learning: The Teacher’s Role. In R. M. Gillies, A. F. Ashman, & J. Terwel (Eds.), The Teacher’s Role in Implementing Cooperative Learning in the Classroom (pp. 9–37). Springer US. https://doi.org/10.1007/978-0-387-70892-8_1
Keller, J. M. (1987). Strategies for stimulating the motivation to learn. Performance + Instruction, 26(8), 1–7. https://doi.org/10.1002/pfi.4160260802
Lazonder, A. W., & Harmsen, R. (2016). Meta-Analysis of Inquiry-Based Learning: Effects of Guidance. Review of Educational Research, 86(3), 681–718. https://doi.org/10.3102/0034654315627366
Molnar, M. (2020, March 18). What Factors Make-or-Break Ed-Tech Implementation? Study Takes a Look. Education Week. https://www.edweek.org/technology/what-factors-make-or-break-ed-tech-implementation-study-takes-a-look/2020/03
Nesbit, J., & Qing, L. (2014). Is Inquiry-Based Learning Effective? EdCan Network. https://www.edcan.ca/articles/the-facts-on-education-is-inquiry-based-learning-effective/
Paris, S., & Turner, J. (1994). Situated Motivation. In P. R. Pintrich, D. R. Brown, & C. E. Weinstein (Eds.), Student Motivation, Cognition, and Learning: Essays in Honor of Wilbert J. Mckeachie. Routledge.
Saye, J. W., & Brush, T. (2002). Scaffolding critical reasoning about history and social issues in multimedia-supported learning environments. Educational Technology Research and Development, 50(3), 77–96. https://doi.org/10.1007/BF02505026
Simons, K. D., & Klein, J. D. (2007). The Impact of Scaffolding and Student Achievement Levels in a Problem-based Learning Environment. Instructional Science, 35(1), 41–72. https://doi.org/10.1007/s11251-006-9002-5
van Uum, M. S. J., Verhoeff, R. P., & Peeters, M. (2017). Inquiry-based science education: Scaffolding pupils’ self-directed learning in open inquiry. International Journal of Science Education, 39(18), 2461–2481. https://doi.org/10.1080/09500693.2017.1388940
Weller, M. (2007). Virtual Learning Environments: Using, Choosing and Developing your VLE (0 ed.). Routledge. https://doi.org/10.4324/9780203964347