Session S1F

MANIC Content Delivery in a Mobile Application

Eric Ssebanakitta , Paul E. Dickson , and W. Richards Adrion
University of Massachusetts Amherst and Hampshire College essebana@student.umass.edu, pedCS@hampshire.edu, adrion@cs.umass.edu Abstract - The University of Massachusetts Amherst RIPPLES group has developed and deployed classroom content creation/capture and delivery systems in the Multimedia Asynchronous Networked Individualized Courseware (MANIC) family for over a decade. The latest of these content delivery systems, iMANIC, is an application for the iPhone/iTouch. The iMANIC mobile platform makes the large variety of data in our complete catalog of courses, seminars, and lectures more accessible to the handheld device-wielding students of today. iMANIC flexibly delivers content from many different methods of content capture/creation, including content captured in fully instrumented classrooms, partially instrumented classroom, captured by handheld camcorders, and uploaded from earlier handcrafted multimedia and slide deck lecture presentations. The RIPPLES Group is transitioning content to the iMANIC iPhone/iTouch mobile application, and adapting the interface to the smaller screen size and other limitations of the device. We have plans to support other mobile devices in the future. Retrospective studies compared iMANIC with earlier MANIC delivery modes and evaluated iMANIC for automatically generating indexed presentations that include videos and simple PDF slide presentations. Index Terms Distance education, Lecture capture, Content delivery. INTRODUCTION As we move further into the 21st century, the world is faced with significant changes in higher education. Rapid changes in pedagogy coupled with an explosion of knowledge force colleges to consider how to increase productivity and effectiveness. Faced with rising costs, many institutions have limited the size of faculty and depended more heavily on blended (on-line plus in-class) education. These changes suggest an increased use of educational technologies, which must be coupled with cost- and learning-effective strategies. The RIPPLES group at the University of Massachusetts Amherst has been developing the MANIC [1] content delivery framework for more than a decade and evaluating its use in a variety of educational and pedagogical settings. Instantiations of the MANIC content delivery system provide a platform [2] by which content can be delivered in a format that includes features proven effective in distance education. Several of these features were developed to support on-campus teaching and learning, including search, navigation, notation, and collaboration tools. Initially, content delivered using the MANIC technologies was provided by human operators and required production crews to create high-quality presentations. While RIPPLES tried a few approaches to automate content creation and capture, the demand for quicker and less expensive production and more flexibility in content delivery led to the development of PAOL [3]-[8], a content capture system, that is used fully in instrumented classrooms to capture video/audio of the lecturer, any content (such as slide presentations or software demonstrations) displayed on the instructors laptop, and any content written on whiteboards or projected on screens. PAOL emphasizes transparency to the lecturer, a key tenant of the RIPPLES group. Transparency is achieved by using a computer screen capture device, wireless microphones, and, in the fully instrumented classrooms, high-resolution cameras, all of which function unobtrusively. To move beyond the instrumented classrooms, PAOL has been used with a second dedicated computer or laptop to capture presentations and audio. Recently, we have experimented with a low-cost Flip camera to capture lectures.

FIGURE 1 PAOL/MANIC INTERFACE

Content captured by PAOL is automatically compiled into a multimedia presentation that can be viewed via a web browser as shown in Figure 1. Similar interfaces are used for the limited PAOL and Flip camera, as we describe in the CONTENT DELIVERY OVERVIEW Section. Recognizing that many, if not most, students have access to mobile devices such as smart phones, we have begun to experiment with delivering content to mobile devices. In this paper, we describe our early efforts to deliver classroom content, 978-1-4244-6262-9/10/$26.00 2010 IEEE October 27 - 30, 2010, Washington, DC 40th ASEE/IEEE Frontiers in Education Conference S1F-1

Session S1F
captured by PAOL in both instrumented classrooms and using a second computer/laptop, captured using Flip cameras, and uploaded as simple slide decks, using a iPhone/iTouch application called iMANIC. In the following sections, we describe the capture processes, both web-based and mobile content delivery, and an initial experiment to evaluate and compare each approach. CAPTURE OVERVIEW As noted, lecture content is captured by the PAOL capture system in both instrumented classrooms and using a second computer/laptop. It is also captured using Flip cameras and/or uploaded slide presentations. PAOL Presentations Automatically Organized from Lectures (PAOL) is a complete lecture-capture system that is transparent to the lecturer. The principle behind the system is that for a capture system to gain acceptance by lecturers it must require no special effort on the part of the lecturer and be able to capture any type of content the lecturer chooses to use; whiteboard, computer, etc. Figure 2 shows the PAOL deployment in an instrumented classroom that uses a pair of cameras to capture the entire front of the room including whiteboards and projection screen. The computer content is captured using an analog capture device that grabs the signal sent to the projector and therefore enables to lecturer to use a computer of their choosing without installing any special software. Audio is captured with a wireless microphone, the wearing of which is the only restriction placed on the lecturer. and a small video frame that surrounds the lecturer location is extracted from the large field of view captured. These extracted frames are then combined with an audio track to create a virtual pan camera generated video of the lecturer.

FIGURE 3 PROCESSING STEPS OF WHITEBOARD CAPTURE. FIGURE 2 COMPLETE PAOL CAPTURE SYSTEM.

PAOL captures lecturer audio, lecturer video, and still frames of all content shown on the whiteboard and computer. The cameras are used to create both the lecturer video and still images from the whiteboard material [3]. The fixed view cameras run at 15 frames per second and capture the area of the front of the room where the lecturer presents material. An algorithm is used to identify lecturer location

The camera images are also processed to create still images of material presented on the whiteboard. One in ten frames is processed to remove the lecturer and raise image contrast to make the text easier to read (Figure 3). Once the processing is complete consecutive images are compared and if sufficient differences are found the new image is stored along with the time that it occurred. This capture of changed images means that progression of material presented is captured as well as the end results (Figure 4).

978-1-4244-6262-9/10/$26.00 2010 IEEE October 27 - 30, 2010, Washington, DC 40th ASEE/IEEE Frontiers in Education Conference S1F-2

Session S1F
The saved image is considered to contain significant information and the time enables the image to be used as part of an index into lecture material. While the PAOL system, particularly in instrumented classrooms, can capture almost all media that might be used, we felt it useful to experiment with low-cost, high definition, portable cameras. A single operator of any skill level can capture virtually all presentations quickly and effectively with a hand-held digital camera. Neither the operator nor the lecturer is required to perform any logistical and set-up tasks. With the FlipPresentation system [7], video and the presenter's slides can be uploaded to a server with a relative ease. The FlipPresentation system works with a slides-only presentation, a whiteboard-only presentation, an audio/video-only presentation, or any combination. Several computer vision and machine learning techniques [8, 9] are used to process the captured video as well as several open source tools like FFmpeg to create videos. The lecture video is digitally enhanced to improve audio, remove noise, correct radial distortion and perspective distortion in the video frames, and improve video quality before it is converted to Adobe Flash format. Most web browsers can handle Flash video better than other formats. Crucial points of presentation in the lecture are identified and extracted to form an in index for navigating the presentation. If the lecture slides (which can be in any format including PPT and PDF) are uploaded, they are processed into an image slide show to be used in the web application. The presentation is captured using a single camera and there is no sophisticated media synchronization with presentation slides, whiteboard/blackboard content and audio/video streams. As part of our optimization process, we employ multi-pass processing technique such that a presentation that is uploaded is ready within minutes for viewing with minimal processing and is progressively improved during subsequent passes. CONTENT DELIVERY OVERVIEW The RIPPLES Group has carried out research on multimedia learning technologies and developed and deployed multimedia materials for on-campus and distance education since 1996. The MANIC [2], [20] framework is used to deliver both live and authored presentations, coupling them with text, graphics, and search/index mechanisms and delivering the content through streaming servers and downloads and on CD/DVD. MANIC initially was developed as a platform for testing streaming multimedia network protocols. The initial incarnation of MANIC involved synchronizing HTML slides with streaming audio. CD-MANIC was developed as a mechanism for delivering high-quality video, synchronized and indexed with lecture slides, to users who faced bandwidth limitations. As high-bandwidth access became more widely available, RIPPLES began to migrate multimedia course content back to the Web as a means of delivery. Various courses were encoded using Synchronized Multimedia Integration Language (SMIL) and a crossplatform Java version called jMANIC.

FIGURE 4 PROGRESSION OF MATERIAL ON THE WHITEBOARD.

Computer screen capture functions in a similar manner [4]. The algorithm grabs each frame sent to the projector as an image and then does pixel-by-pixel comparison to look for changes. When changes are found between consecutive images an image and the time that it occurred is saved for later index into a presentation. By using image-processing techniques on captured images this system is able to capture content regardless of the software used by the lecturer for

FIGURE 5 SAMPLE COMPUTER CAPTURED IMAGES.

presentation (Figure 5). Portable PAOL The screen capture system was the first component of PAOL to be developed and was used in the initial evaluation [3], [5] of the system. Recently a substantially updated version of the PAOL screen capture system was used as a stand-alone system as part of an extensive evaluation at Hampshire College [6]. Flip Capture System PowerPoint, Keynote, SliTeX and PDF are commonly used classroom/lecture presentation tools, while many instructors still prefer overhead or document projectors and blackboards/whiteboards. Most use some mixture of these tools with the result that classroom capture is a challenge.

978-1-4244-6262-9/10/$26.00 2010 IEEE October 27 - 30, 2010, Washington, DC 40th ASEE/IEEE Frontiers in Education Conference S1F-3

Session S1F
To exploit the availability of PAOL captured content, several new versions of MANIC have been implemented. Most of these are Adobe Flash-based for portability. PAOL/MANIC Content Delivery The PAOL capture system automatically creates MANIC presentations. PAOL uses a class schedule to automatically capture the camera and screen data before beginning image analysis to create the audio/video stream, and content images of the computer screen and whiteboard. This content is then compiled into a presentation, and uploaded to a server for distribution. This content is delivered using most recent web-based MANIC interface (Figure 1) that provides 3-4 re-sizable and movable windows, one each for lecture video, screen capture and whiteboard images. The thumbnail index on the right expands on mouse over and indexes points where either a new screen or whiteboard image is captured. The index allows for easy navigation. In the portable version of the PAOL system, files are uploaded to a content management system and viewed through a similar interface. We are currently moving all PAOL capture to this content management system. FlipPresentation Content Delivery The FlipPresentation web-based content delivery interface (Figure 6) is quite similar to the PAOL/MANIC system. It is as a Web application with several movable and re-sizable windows, including a video window, a notes window, a slide window, and sampled video and slide indexes for navigation. The primary difference with PAOL/MANIC is that the video thumbnail indexes on the right are sampled, rather than representing slide changes. The slide indexes on the left allow the user to advance the slide window, but the user can also advance the slides using controls in the slide window. At present, slides and video are not synchronized, but we are working to add that capability.
(A) VIDEO

overcome the limitation that video and still images could

FIGURE 7
IMANIC INDEX

not be synchronized. An iMANIC user begins with an index (Figure 7) that provides access to class content. In this example, the first two presentations are FlipPresentations adapted to the iPhone/iTouch, while the rest are lectures captured by the

(B) VIDEO INDEX FIGURE 8 IMANIC PAOL CONTENT

FIGURE 6 FLIPPRESENTATION INTERFACE.

iMANIC Content Delivery The iMANIC content delivery system had to address some limitations of the iPhone/iTouch. While the screen size is the obvious limitation, a significant challenge was to

(B) VIDEO INDEX

(C) SLIDE INDEX FIGURE 9 IMANIC FLIP VIDEO CONTENT

978-1-4244-6262-9/10/$26.00 2010 IEEE October 27 - 30, 2010, Washington, DC 40th ASEE/IEEE Frontiers in Education Conference S1F-4

Session S1F

FIGURE 10 EXPERIMENTAL RESULTS

PAOL system. Selecting a presentation leads to a second index (Figure 7). For PAOL lecture content, this screen provides a choice (A) V between Video IDEO (Figure 8a), Video Index Slideshow (Figure 8b) and Notes. Video provides a standard iPhone/iTouch video interface for the PAOL-captured video (similar to a podcast). The Video Index Slideshow provides sampled images that are synchronized with audio. Touching the screen brings up an automatically generated thumbnail index of the video that supports navigation not provided by standard podcasting. For the evaluation described below, we did not include screen or whiteboard content due to image size impacting download times, but we are adding this content. The FlipPresentation lecture content screen provides a choice between Video (Figure 9a), Video Index Slideshow (Figure 9b), Instructors Slides (Figure 9c) and Notes. Again Video provides a standard iPhone/iTouch video player interface and the Video Index Slideshow provides sampled images from the video synchronized with audio and a popup index. As with web-based FlipPresentation content, slides and notes are not synchronized with the audio, however the slides are indexed similarly to the video an index pops up when the current slide image is touched. RELATED WORK Lecture capture systems have been under development for years. These systems range from the basic seminar-style capture of AutoAuditorium [10] to eClass [11], a comprehensive capture system that indexes computer and whiteboard material by using electronic whiteboard and

special computers to capture all material. In between are such systems as the University of California Berkeley's system [12] that captures a video of the lecturer and any slides presented and Authoring on the Fly [13], which records lectures but focuses on methods of indexing captured material. Many other research [14]-[16] and commercial [17]-[19] capture systems exist, and all limit the lecturer in some way, either by restricting presentation modality or requiring special software or equipment. PAOL and FlipPresentations require neither and compare very well with the most similar systems. EXPERIMENTAL RESULTS During Spring 2010, we deployed the iPhone/iTouch content delivery system in an advanced software engineering class at the University of Massachusetts Amherst where we were already evaluating both web-based content delivery systems. About one third of the class had access to iPhones or iTouches. We developed a survey form and asked these students to evaluate each web-based interface and the iPhone/iTouch interface and to compare each. The results are summarized in Figure 10. One interesting outcome was the general consensus that all of these content delivery systems were no substitute for attending class. Less than 20% of the students indicated that the availability of class content would result in lower attendance, but many suggested that they are great to have in case one missed class for illness or other reasons. For the two web-based interfaces, students liked the navigation and, for the PAOL capture, the synchronization of slides, whiteboard and video/audio. Interestingly, the

978-1-4244-6262-9/10/$26.00 2010 IEEE October 27 - 30, 2010, Washington, DC 40th ASEE/IEEE Frontiers in Education Conference S1F-5

Session S1F
sound quality was criticized in both web-based versions, but not in the iPhone/iTouch versions. Overall, the students preferred the web-based versions, but commented on the advantages the iMANIC content delivery had for ease and portability. There was little criticism of the compromises imposed by the limitations of the iPhone/iTouch devices (video/image synchronization, small screen size). The results shown in Figure 10, indicate that about 40% of the students preferred the iPhone/iTouch to the web-based versions, but the vast majority would recommend the iPhone/iTouch application. 40% would like to see it on Android and other smart phones. CONCLUSIONS AND FUTURE WORK While this is a limited trial of the iMANIC application, it does show that mobile delivery of classroom content holds great promise. It seems clear from this experiment that for the near present, students would prefer a variety of options including both web-based and mobile devices. We continue to deploy and evaluate both web-based and iPhone/iTouch content delivery at Hampshire College and the University of Massachusetts. We are adding screen and whiteboard content to iMANIC. We plan an additional evaluation at the end of the Spring 2010 semester. As we transition captured content to a content management system, we believe that automatic compilation of multimedia content will be supported easily for a variety of delivery media. ACKNOWLEDGMENTS This work was partially funded by the National Science Foundation under grants CNS-0837739 and CNS-0755376 and by a University of Massachusetts ITC Strategic Initiative Grant. REFERENCES
[1] Agustn Schapira, Kimberly de Vries, Cris Pedregal-Martin, MANIC: An Open-Source System to Create and Deliver Courses over the Internet, Applications and the Internet Workshops, IEEE/IPSJ International Symposium on, p. 21, 2001 Symposium on Applications and the Internet-Workshops (SAINT 2001 Workshops), 2001 W. Richards Adrion, James Kurose, Edward Riseman, Zhigang Zhu. A Multimedia Virtual Classroom, Proceedings of the International Conference on Information Communications Technologies in Education (ICICTE-04), July 2004 Paul Dickson; W. Richards Adrion; Allen Hanson; , Automatic Capture of Significant Points in a Computer Based Presentation, Multimedia, 2006. ISM'06. Eighth IEEE International Symposium on, vol., no., pp.921-926, Dec. 2006 Paul E. Dickson, W. Richards Adrion, Allen R. Hanson. Automatic creation of indexed presentations from classroom lectures, ITiCSE 08: Proceedings of the 13th annual conference on Innovation and technology in computer science education, pp. 1216, 2008 Paul E. Dickson , W. Richards Adrion , Allen R. Hanson , David T. Arbour, First experiences with a classroom recording system, ACM SIGCSE Bulletin, v.41 n.3, September 2009 Paul E. Dickson, David T. Arbour, W. Richards Adrion, Amanda Gentzel. Evaluation of automatic classroom capture for computer science education. To appear in ITiCSE 10: Proceedings of the 15th annual conference on Innovation and technology in computer science education [7] Eric M. Ssebanakitta, W. Richards Adrion, and Paul E. Dickson. FlipPresentations.com - Lecture capture with portable ("Flip") camera, intelligent automated processing, and a customizable Web2.0 presentation. http://www.flippresentations.com/iphone/FlipPresentationsPaper.pdf March 2010. P.E. Dickson; W.R. Adrion; A.R Hanson. Automatic capture and presentation creation from multimedia lectures. Frontiers in Education Conference, 2008. FIE 2008. 38th Annual , vol., no., pp.T2A-14-T2A-19, 22-25 Oct. 2008 Weihong Li; Hao Tang; Zhigang Zhu; Automated Registration of High Resolution Images from Slide Presentation and Whiteboard Handwriting via a Video Camera. Computer Vision and Pattern Recognition Workshop, 2004. CVPRW '04. Conference on , vol., no., pp. 168- 168, 27-02 June 2004

[8]

[9]

[10] M. Bianchi, AutoAuditorium: a fully automatic, multicamera system to televise auditorium presentations. Proc. Of Joint DARPA/NIST Smart Spaces Technology Workshop, July 1998. [11] Brotherton, J.A. and G.D. Abowd, Lessons learned from eClass: Assessing auto-mated capture and access in the classroom. ACM Transactions on Computer-Human Interaction, 2003. [12] E. Machnicki and L.A. Rowe, Virtual Director: Automating a Webcast, SPIE/IS&T Multimedia Computing and Networking Conference, Vol. 4673, San Jose CA, Jan 2002. [13] W. Hurst, G. Maass, R. Muller, and T. Ottmann. The 'authoring on the fly' system for automatic presentation recording. CHI 01: CHI 01 extended abstracts on Human factors in computing systems, pp. 56, New York, NY, USA, 2001. [14] G. Cruz , R. Hill, Capturing and playing multimedia events with STREAMS, Proceedings of the second ACM international conference on Multimedia, p.193-200, October 15-20, 1994. [15] Y. Rui, A. Gupta, J. Grudin, Videography for Telepresentations, Proc. of the ACM Conference on Human Factors in Computing Systems, pp. 457 - 464, 2003. [16] Y. Rui, L. He, A. Gupta, Q. Liu, Building an intelligent camera management system, Proc. ACM Conf. on Multimedia, pp. 211, 2001. [17] Sonic Foundry. Mediasite platform. http://www.sonicfoundry.com/ Mar. 2010. [18] Panopto. Redefining Presentation Capture..http://www.panopto.com/ Mar. 2010 [19] TechSmith. Camtasia studio screen recorder for demos, presentations and training. See http://www.techsmith.com/camtasia.asp Mar. 2010. [20] S.R. Thampuran, W. Burleson, K. Watts, Multimedia distance learning without the wait, Frontiers in Education, Annual, pp. T2F-T21 vol.1, 31st Annual Frontiers in Education Conference, 2001.

[2]

[3]

[4]

[5]

[6]

978-1-4244-6262-9/10/$26.00 2010 IEEE October 27 - 30, 2010, Washington, DC 40th ASEE/IEEE Frontiers in Education Conference S1F-6