1

Formative Report for Educational Resources

Prepared for: ThinkHard Design Studies, LLC

Group Members: Daniel Flynn Tony Hetrick Jennifer Hornyak

EDTECH 505-4172 April 19, 2011

Table of Contents
Table of Contents Executive Summary Results and Interpretations Usability Content Educational Value Instructional Styles Instructional Setting Instructional Application Vividness Page Comparisons Site Rankings Highest Rankings - Most Appropriate Instructional Activities Highest Rankings - Common Elements Lowest Rankings - Common Elements Overall Recommendations References Appendices Charts / Graphs Average Scores among All WBLES A rubric for evaluating Web-based learning environments (Baya'a, Shehade, & Baya'a, 2009) Usability Criterion Content Criterion Education Value Criterion Vividness Criterion

Executive Summary
A small team was contracted by ThinkHard Design Studies, LLC to conduct a detailed analysis of 29 educational websites (WBLEs). In order to analyze each website, a rubric created by Baya'a, Shehade, & Baya'a (2009) was distributed in electronic form to experts in the field of education. Both quantitative and qualitative data was collected based on the expert feedback. The rubric helped to evaluate each website in the following areas: usability, content, educational value, and vividness. Overall, the WBLEs were only rated as adequate with the educational value category receiving poor reviews. However, the content category received relatively high reviews. Of the evaluated WBLEs, 44% contained an informational instructional style while only 9% used a drill-andpractice instructional style. To improve the WBLEs, it is recommended that both the usability and educational value aspects are modified to meet the rubric criteria.

Results and Interpretations

Usability
A majority of the WBLEs that were reviewed contained significant usability issues. The main issue had to do with navigation, which allows users to tell where they are, where they have been, and where they need to go next. Also, design and lack of enjoyment contributed to the low score. The below graph displays the average score of all WBLEs reviewed.

The data showed a strong correlation between page design and navigation. When a site was evaluated on its design, only 12.5% of the time was there a navigation score that ranked higher than the design score. This was also evident from the review comments. For example, a reviewer of Zebra Keys stated that the site is too busy. It has too many advertisements for the audience to navigate with flow. Another reviewer for Kids Biology stated that the ads and extra links were constantly interfering with the instruction making it nearly impossible to process the navigation or efficiently gain knowledge from the text. Other than interfering with the navigation, the design of the WBLEs contributed to other usability issues. The reviewers stated that the sites were overwhelming by being too busy, used inappropriate or too small font size, or were lacking contrast between the text and the background. The lack of good design contributed to making certain sites difficult to use, despite good content. A reviewer of ESL Go explained that while the site provided good grammar help, it was poorly designed which led to confusion and a jumbled mess.

While the majority of the sites contained usability comments that listed problems, not all sites were poorly designed. For instance, a reviewer stated that The Musical Mind was very clean and professional looking with no ads and nice use of color. To improve usability, CARP (Contrast, Alignment, Repetition, and Proximity) design principles should be followed. These principles are the guidelines that will enable the WBLEs to look clean, neat, and professional (Williams & Tollett, 2006, p. 113) and the WBLEs will better communicate their content and they will be more enjoyable (Williams & Tollett, 2006).

Content
A majority of the WBLEs that were reviewed contained significant doubt as to content authority. For the most part, however, the WBLEs content for relevance, sufficiency and appropriateness are quite relevant for users and show adequate ratings for the most part. The following chart shows the averages of the results of the WBLEs.

The results showed that reviewers found most of the websites relevant in terms of content. However, there is a high doubt on the authority of the resources used in creating the content of the site. Reviews of some the WBLEs authority (Q4.2) were mentioned in the comment sections (Q4.6) and they included:

6 WBLE Camera Work Tutorial Comment The site is run by experts in their fields, (based on information of profiles) however, not sites or other resources are mentioned to back up their claims of information

Franchise Webquest Kids Biology

I would have given the authority a better mark if the links to back up information were working. There are no sources, so it is unclear where the information is coming from. Overall, I felt the level of information provided was not sufficient

There are 43% of sites that have an excellent rating for authority of websites. However, this leads to an issue of inclusion of sites not based on authority in the portal page since the sites that dont have proper resources will likely lose credibility with potential users. Most of the reviewers did like the overall content but lacked the confidence in referencing valid sources that would give an authority for inclusion in the portal. This being said, reviewers believe that the WBLEs are relevant in their appropriateness with a strong level of difficulty and sustainability. For the sites that have content that is backed by proper resources, they often score high on relevance, sufficiency, and appropriateness. Reviewers often comment positively about the overall content of the site. For example, The Genius of Leonardo da Vinci site, a reviewer commented that the content may have been slightly excessive, but I'd rather have that than insufficient. These comments were frequent thought out the WBLE and show a general concern of how a site should be constructed with proper resources. Overall, content scored well, except for the amount of sites that lack proper authority (resources, working links to resources, dependable, etc.) that would bring credibility to the portal. One possible solution could be to create a second section within the portal for sites that lack proper resources and label it something other than primary sites. After analyzing the content for the various age groups, there seems to be a heavy disposition of sites for 4-6th grade and 7-9th grades, making up 56% of the sites. 16% of the sites were made for 10th-12th grade and 9% of the sites were for the 1st-3rd grade. Kindergarten ages sites represented 3%, or only 1 WBLE. See the below chart for additional detail.

Educational Value
Overall, reviewers found the WBLEs to have varying degrees of educational value. The rubric addressed each website in the areas of: instructional style, instructional setting, and instructional application. While some variance was found in these areas, it would be beneficial to expand the portal to include WBLEs that represent a wider variety of instructional styles and instructional settings as well as address a wider range of instructional applications. Instructional Styles The WBLEs were evaluated for the following instructional styles: Informational, Drill & Practice, Tutorial, Simulation, Instructional Game, and Mixed Mode which is a WBLE that includes a combination of one or more of the above instructional styles. According to the data, the majority (44%) of the sites represents WBLEs that are informational and contain little or no student interaction. None of the WBLEs were found to be of the simulation instructional style. Please see the below chart for additional information.

Instructional Setting The WBLEs were rated by the expert reviewers in terms of which setting each WBLE would be most effective. The majority of the sites, 53% or 17 WBLEs, in the list were rated to be most effective for individuals to use outside of class (at home or after school). 25% of the sites were found to be appropriate for a whole class instruction using a projector or large display. It would be beneficial to include a wider variety of WBLEs appropriate for all settings on the portal so that educators would be able to utilize the portal to a greater extent. Instructional Application The instructional application that was determined to be most appropriate was basic instruction for all learners. The graph below shows that while there are a range of instructional applications among the WBLEs evaluated, the majority would be most advantageous to be used in basic instruction. To improve the educational value of the WBLES on the list, there should be some consideration to including a wider variety of instructional activities.

Vividness
Overall, the WBLEs ranked as adequate in the category of vividness. According to the rubric, vividness includes sufficient links to other pages as well as updated and current content and links. Current information and workable links are important for a WBLE to obtain a high scoring review. Expert reviewers rated the links found in the WBLEs that point to other pages as adequate with a mean result of 2.06 on a scale of 1 to 3, with 1 being the best ranking possible. In addition, experts rated the WBLEs in terms of updated content and links as adequate with a mean result of 2.56. It is recommended that additional sites be added to the portal which includes updated links that point to appropriate and supplementary content, as well as sites that contain current content.

Page Comparisons
Upon comparing the sites on the two different pages, they were found to be comparable. Overall, there was only 2.6% separating the two pages, with page #2 performing slightly better. The mean result for page #1 was 1.94 and the mean result for page #2 was 1.89.

10

Site Rankings
Highest Rankings - Most Appropriate Instructional Activities Among the 10 sites that were rated the highest by the expert reviewers, the most appropriate instructional activity is basic instruction for all learners. Below is the list of the instructional values from the top 10 sites: 1. 5 WBLEs - Basic instruction for all learners 2. 3 WBLEs - Enrichment instruction for students who will benefit from more advanced activities 3. 1 WBLE - Differentiated instruction for students with varying learning styles 4. 1 WBLE - Remedial instruction for students who did not initially grasp the concept Highest Rankings - Common Elements Among the 10 sites that were rated the highest by the expert reviewers, their common elements were from the categories of usability and content. Below are their rankings in order:
1. Enjoyment (Usability Q3.6): Multimedia and humor that are relevant, clear and

appropriate to the subject matter 2. Readability (Usability (Q3.7): Users can easily view contents of the web pages 3. Relevance (Content Q4.3): Information focuses on the main topic and does not include irrelevant or marginal data 4. Authority (Content Q4.2): Information relies on authentic organizations and dependable resources

11

Lowest Rankings - Common Elements Among 10 sites that were rated the lowest by the expert reviewers, their common elements were from the categories of educational value and vividness. Below are their rankings in order:
1. Help Tools (Educational Value Q5.8): Help tools to assist users in solving problems 2. Updating (Vividness Q6:3): Designers continuously update the information, content and

links
3. Communication (Educational Value Q5.5): Users are able to interact with designers of

the WBLE and other peers 4. Rubric (Educational Value Q5.7): Activities include rubrics to evaluate user performance

Overall Recommendations
In order to improve the web portal, it is recommended that the data presented in this report be used to provide additional WBLEs on the portal that would rank highly amongst experts especially in the area of educational value and usability. The rubric, provided in the appendix, should be used in choosing any additional sites. It may also be considered that the lowest ranking WBLEs be removed from the portal and replaced with higher ranking WBLEs. The chart below depicts the WBLE in terms of overall ranking of the WBLEs. Sites that should be considered to be replaced in the portal include: Kids Biology, ESL Go, ZebraKeys, and Free Drawing Lessons.

12

When choosing new WBLEs for the portal, a specific focus should be placed on the portal including WBLEs that are strong in the area of content so that educators will be able to use the WBLEs across the curriculum. Based on the data, an additional recommendation is to add more websites that are geared towards the Kindergarten grade level. It is also very important the portal includes websites that address a variety of instructional applications across many different educational settings. In addition, it would be beneficial to categorize the websites on portal by grade level, allowing the users to easily access proper content immediately.

13

References
Baya'a, N., Shehade, H. M., & Baya'a, A. R. (2009). A rubric for evaluating Web-based learning environments. British Journal of Educational Technology, 40(4), 761763. Williams, R., & Tollett, J. (2006). The non-designers web book: An easy guide to creating, designing, and posting your own web site (3rd ed.). Peachpit Press CA.

14

Appendices
Charts / Graphs
Average Scores among All WBLES

15

A rubric for evaluating Web-based learning environments (Baya'a, Shehade, & Baya'a, 2009)
Usability Criterion
Purpose The purpose, benefit and importance of the WBLE should be presented clearly. The homepage should include a well-labeled, clearly defined table of contents. Users need to know where they are, where they have been and where to go next. Web pages have to be neat, simple and not overstuffed, and background should not interfere with text. Multimedia and humor are recommended, but should be relevant, clear, appropriate to the subject matter, and should not be heavy in a way that slows down the presentation of the web pages. Users should be able to read the content of the web pages easily, by proper use of text, sounds and images.

Homepage

Navigation

Design

Enjoyment

Readability

Content Criterion
Authority Information in the WBLE should rely on authentic organizations and dependable resources in the field of the presented material. Information should be accurate and based on factual knowledge and professional thinking. The information provided ought to focus on the main topic of the WBLE, and should not include irrelevant or marginal data. The amount of information in the WBLE should be sufficient but not excessive. The information should be presented in an appropriate method and suitable level of difficulty to the target users.

Accuracy

Relevance

Sufficiency

Appropriateness

16

Education Value Criterion

Learning activities WBLE should provide learning activities that expose the users to new information and encourage them to construct new knowledge and educational substance. Each learning activity should be accompanied with a clear and suitable activity plan. The activities should include well prepared content presented in various ways, as well as references to additional resources on the Web. Users should be able to interact with the designer of the WBLE and with their peers through different communication tools (email, chat and forums). Users performance and outcome of each activity should be tracked and evaluated through a mechanic or human feedback. Each activity ought to include a rubric to evaluate the users performance. WBLE ought to include assistant tools that help the users solve problems (technical, contextual and didactical) that they might face in the process of performing the activities.

Activity plan

Resources

Communication

Feedback

Rubric

Help tools

Vividness Criterion
Links WBLE should direct users to other web pages: in other sites or in the site itself. These links should always be active, adequate and enriching. WBLE designers should update the information, content and links of the WBLE continuously.

Updating