ABSTRACT

Given the phenomenal changes in the use of network computer technology in education, it has become important to better understand teachers' conceptions of this technology and its uses. This is particularly true in the case of conceptions, or "mental models", of the Internet and its uses. Two questions are addressed: How do novices and experts think about the Internet and its applications (e-mail and WWW)? Does a person's conceptual model have an impact on network use? To examine the role of these mental models in more detail, a survey and eight case studies were conducted among pre- and in-service teachers enrolled in University courses. The survey solicited mental models of the Internet, e-mail and the World Wide Web. The case studies consisted of a network problem-solving task using think-aloud protocols. All subjects identified themselves as to their technical expertise. The results show a surprising diverse set of plausible models of the Internet and the Web, not related to their expertise level. Epistemic relationships to mental models are also discussed. The case studies reveal some relationship between model and navigation strategies, where experts employ multiple, structurally and/or epistemologically different models at different stages in a network task. These findings are discussed with respect to educational applications of the Internet.

INTRODUCTION

It has been four years since the Center for Technology and Education's (CTE) national survey of K-12 telecommunications was conducted (Honey & Henriquez, 1992). One important implication from this survey was that there is a need to better understand teachers' conceptions of technology and its uses. Honey and McMillan followed up this survey with in-depth interviews of 18 teachers, and found different "mental models" ("superhighway" vs. "ocean") among the teachers. Previous to and since these studies there has been little research in how mental models might impact network usage, navigation, and learning. Research of this type is vital if we are to understand how to use network learning environments productively in schools by teachers and students.

To examine the role of these mental models in more detail, we have conducted an exploratory study of mental models of the Internet, E-mail and the World Wide Web. These studies consisted of surveys of mental models and case studies of network problem solving using think-aloud protocols. This paper will address the following questions: How do novices and experts think about the Internet and its applications (e-mail and Web)? Does a person's conceptual model have an impact on network use? What are the implications for interface design, network training, server development, and other educational uses of the Internet?

We are interested in the "mental models" users have for using computer networks. Mental models are those internal, often informal and naive, representations that guide the user's actions on a task, creation, or problem (Gentner & Gentner, 1983; Johnson-Laird, 1983; McCloskey, 1983; Norman, 1983; Norman & Draper, 1986). Mental models often have strong visual images of the system to explain its function, or they can be metaphorical explanations as to its meaning in the mind of the individual. Together, these two features of mental models provide guidelines for acting upon the system, especially when no explicit strategies exist (Norman, 1988; Norman & Draper, 1986).

The goal of this study was not to find the most plausible models that explain the Internet or Web, but to empirically generate a categorization scheme for common mental models and to determine if those models are employed in any systematic manner in network tasks. It was also our goal to isolate the components and characteristics of mental models, their points of failure, and their adaptability to problem solving and network navigation. If a particular class of mental model predicts performance on a relevant task, then it might be used as an instructional model for training and designing network learning environments. The dimensions of this analysis include the classification of the models, their dynamic attributes over time and in relation to expertise, the extensibility of the models to other settings, and the epistemic qualities of the models.

METHODOLOGY

This is a qualitative study using data from surveys, interviews, and observations. The subjects consisted of pre- and in-service teachers from the University of Illinois at Urbana-Champaign (UIUC) and the University of California, San Diego (UCSD). Other volunteers from UIUC were also studied. Three instruments were used to collect data. The first was a questionnaire to elicit self-descriptions of mental models from the subjects. Before eliciting the individual's mental model, we first provided an example of a mental model:

"For example, people often think about the atom as being like a miniature solar system. The solar system is then a mental model of the atom."

As shown in data from this study, this analogy was usually sufficient for eliciting mental models from the subjects. The questionnaire was administered at the beginning and end of the courses, and included interval scale assessments of the subject's computer and network experience and as well as their teaching experience. The remainder of the survey were open-ended questions about mental models of the Internet, Gopher, and World Wide Web.

The second instrument was a task analysis of the problem solving behavior and analysis of think-aloud protocols of subjects tackling two problems using the WWW on the Internet. The third instrument was a trace of network navigation during the think-aloud sessions. Those traces were used to augment the transcriptions from the think-alouds and to confirm navigational patterns. The variables in this study are: (a) experience of user, (b) the network task or situated problem, and (c) the robustness of individual models as they relate to actual use of the Internet in the context of the task. We are interested in the mental models of novices and experts, measured in terms of technical expertise, familiarity with the Internet and telecommunications, and professional expertise.

A pilot study was conducted in the Spring of 1995. Teacher education students enrolled in courses at UIUC responded to a survey of their mental models. Data from that pilot study were used to refine the survey and design a think-aloud protocol for observing subjects during a network problem-solving task. From the pilot study, we discovered a systematic problem with the phrasing of the questions. We were concerned that the subjects would not know what the term mental model meant, so we asked the subjects:

"What is your image of the Internet?"

This generated unpredictable responses, usually about the subject's opinion of the Internet. As a result, we re-worded using the term "mental model" the survey and provided an example of one (as mentioned above). The final survey is shown in the Appendix A. The network task and the think-aloud protocol is shown in Appendix B. This revised survey was used during the Summer and Fall of 1995 with three subject groups selected for their levels of network experience.

For the survey portion of this study, three groups of network users were studied. The first group was a math education class (C&I 336) at UIUC. The students in that class were mostly in-service math teachers working towards a Masters degree during the Summer 1995 session. As part of this course, the students were provided with laptop computers equipped with Internet tools. One of the goals of this course was to expose the teachers to math education resources on the Internet. The second group was a teacher education class (TEP 231) at UCSD. The students in that class were all practicing K-12 teachers enrolled in a master's program, also enrolled during the Summer 1995 session. The class focused on technology and education. The third group was an educational computing class (EDPSY 387) at UIUC enrolled during the Fall of 1995.

For the case studies, volunteers were solicited from each class, as well as from expert network users within the College of Education at UIUC. The subjects were informed that the interview would be recorded on audio tape, and that it would take less than one hour.

RESULTS

Survey Findings

The survey was repeated at the end of each course, but without the questions regarding self-evaluation of skills. In all, 61 subjects from the three classes responded to at least one survey.

Responses to the self-evaluation were converted into scalar values from 0 (none) to 4 (lots). These indices were used to determine an overall measure of expertise. This is referred to as the Expertise Factor and is equal to the average of the six expertise questions (experience with computers in general, networks, e-mail, newsgroups, gopher, and Web). However, to better tune the Expertise Factor for the purposes of this study, the general computing value was weighted twice and the e-mail and Web values weighted 1.5 times. This measure was used to rate the subjects' overall computer and network expertise into three classes: novice, intermediate, and expert. Possible values for the Expertise Factor ranged from 0 to 5.67.

With respect to the questions about mental models, not all subjects responded to all the questions. Some subjects drew diagrams of the models, while others pointed to earlier responses. Responses from all three groups were collapsed into a single matrix. Table 1 shows frequencies of responses for each group for the pre- and post- experience questionnaires. Non-responses include both blank responses and those explicitly not responses (responses of "don't know", "?", etc.)

Table 1. Response frequencies

	Pre-Experience				Post-Experience
	Internet	E-mail	Web		Internet	E-mail	Web
TEP 231	18	17	15		18	18	19
CI 336	10	9	8		6	6	6
EDPSY 387	15	16	17		12	10	12
TOTAL MODELS	43	42	40		36	34	37
TOTAL NON-RESPONSES	18	19	21		25	27	24
TOTAL RESPONSES	61	61	61		61	61	61

Diversity of models

The most striking result was the diversity of models. Appendix B shows the responses from all three groups. The responses were then shortened into a few-word statement about the model. Table 2 is a summary of all the unique responses to the Internet and Web questions that could be interpreted as models.

Table 2 (a). Internet models.

Count	Pre-Experience Model		Count	Post-Experience Model
7	web		6	highway
6	highway, interstates		6	web, spider web
3	communication		2	encyclopedia
2	brain		1	city
2	library		1	community
3	phone calls, lines		1	computer chip
2	solar system		1	connection of information
2	universe		1	filing cabinet
1	BBS		1	foggy world
1	encyclopedia		1	fungus
1	filing cabinet		1	funnel
1	fishnet		1	Holodek
1	fungus with tentacles		1	Lattice - Interconnected
1	lattice		1	Library
1	Maze		1	network of clients and servers
1	nervous system		1	neural networks
1	schema		1	octopus
1	sea		1	solar system
1	Star Trek Enterprise		1	teleconnections
1	toy jacks		1	telephone system
1	tree		1	water molecule
1	wave - interactive		1	wave - surfing
39	22 Total Unique Responses		32	22 Total Unique Responses

Table 2 (b). Web models.

Count	Pre-Experience Model		Count	Post-Experience Model
12	Web, Spider Web		5	Web, spider web
6	library		2	freeway, highway
2	BBS		2	information flow, sharing
2	information, storerooms		1	ant trails
2	link - international		1	BBS
2	multimedia book, lines		1	city
2	streets, roads, highways		1	communication
1	Access to Internet		1	community
1	brain synapses		1	conference room w/ encyclopedias
1	encyclopedia		1	connected computers
1	fishnet		1	fungus
1	fungus		1	funnels - visible
1	graphical Internet		1	global village
1	Homebase		1	grain of sand
1	jacks		1	library - floating pages
1	nervous system		1	nebula - amorphous
1	solar system		1	network operating system
1	visual model of culture		1	neural networks
			1	publishing
			1	Star Trek computer
			1	supermarket of ideas
			1	supper table
			1	tangle of connections
			1	telephone system
			1	TV channel
			1	USA Today
			1	window on the world
39	18 Total Unique Responses		33	27 Total Unique Responses

As shown in Table 2 (a and b), there are similarities in how subjects perceive the Web and the Internet. For some, the Web and the Internet are the same thing. For others who may be more expert, the difference between the Internet and the Web are represented in different models of the two. For still others, the name "Web" evokes a model that is sufficient. Indeed, "Web" and its variants was the modal response for the Internet question pre and post. It was expected that Web would be the modal response for the Web question, and Table 2(b) bears this out. What is interesting in Table 2(b) is how the subjects discovered for themselves more salient models for the Web, than "Web". "Web" is still the modal response in the post column with 5 responses, but this is down from 12 in the pre column.

Expert Versus Novice Models

Table 3 shows the frequency of responses in which the subject replied that he or she did not have any model, or that the question was not answered. These are referred to as non-responses. The non-responses were sorted by expertise. Levels for expertise were determined by values from the Expertise Factor:

Novice 0 to 1.99

Intermediate 2.00 to 3.99

Expert 4.00 to 5.67

Table 3 shows a definite pattern that indicates that novices are less likely to report having a model than more experienced users. The totals are represented in the total responses in Table 1.

Table 3. Percentages of non-responses.

	Internet					Web
	Pre	%	Post	%			Pre	%	Post	%
Novice	8	40%	1	9%		Novice	10	50%	1	9%
Intermediate	5	20%	2	10%		Intermediate	4	16%	1	5%
Expert	2	17%	1	11%		Expert	3	25%	1	11%
TOTAL	15		4			TOTAL	17		3

Impact of experience on model change

The diversity of models increases with experience, especially for the Web models. Table 4 shows all of the subjects who responded to the Web model question pre- and post-training, sorted by expertise. Thirteen (13) models show a substantive change in model, seven (7) show no substantive change, and four (4) show a similar model. It is the group of subjects (subject numbers: 2, 11, 35, 44) whose models were similar that is most interesting. Table 5 explores their actual responses in detail.

Table 4. Response differences between pre-and post Web models, sorted by increasing expertise.

Subject	Group	Experience Factor	Pre-Web Model	Post-Web Model	Difference
2	TEP 231	0.33	encyclopedia	information flow	similar
10	EDPSY 387	1.08	library	USA Today	change
11	CI 336	1.17	library	encyclopedia - 24-hour	similar
14	CI 336	1.42	Web	Web with strands	no change
21	EDPSY 387	2.00	information	highway	change
25	CI 336	2.25	BBS	international BBS	no change
27	TEP 231	2.50	Access to Internet	tangle of connections	change
31	TEP 231	2.75	fungus	fungus	no change
33	TEP 231	2.83	streets	freeway	no change
35	EDPSY 387	3.00	information storerooms	publishing	similar
36	EDPSY 387	3.08	multimedia book	TV channel	change
38	TEP 231	3.08	spider web	library - floating pages	change
41	EDPSY 387	3.17	spider web	web	no change
42	EDPSY 387	3.17	library	network operating system	change
43	EDPSY 387	3.33	link - international	information sharing	change
44	EDPSY 387	3.50	brain synapses	neural networks	similar
45	EDPSY 387	3.58	BBS	window on the world	change
46	TEP 231	3.75	spider web	spider web	no change
47	TEP 231	3.75	[drawing]	[drawing]	change
52	EDPSY 387	4.33	link - pictures	nebula - amorphous	change
53	TEP 231	4.67	library - linked	supper table	change
57	TEP 231	5.17	multimedia lines	connected computers	no change
59	EDPSY 387	5.50	library	city	change
61	TEP 231	5.67	web	community	change

Table 5. Actual survey responses for subject whose Web models were rated as similar pre and post.

Subject	Pre-Web Response	Post-Web Response
2	Huge encyclopedia with unlimited information available to everybody	Information going back and forth in all directions. No beginning/no end [a drawing of a network of nodes with interconnecting arrows]
11	Sub-Libraries	24-hour encyclopedia
35	Multi-faceted, Individually Based Storerooms Of Information	Opportunity for Individuals To Immediately "Publish" + Utilize Information
44	something like the brain, with tons of synaptic connections	neural networks

Table 5 shows an emergence of more descriptive models, each in unique ways. For example, the models for subjects 2 and 35 take on a publishing component. This is an indication of their expertise gained in creating Web pages in their courses. Subject 11's model goes from a spatial library model to one of an encyclopedia, which represents the (graphical) information contained in the library. The models for subject 44 stay aligned toward the brain model, but change in structural specificity from the randomness of synapses to the structure of neural networks.

Case studies

Ten volunteers were obtained to participate in an in-depth interview of using the Web on a task: two (novices) from CI 336, four (intermediate) subjects from TEP 231, one (expert) from EDPSY 387 and three (experts) from within the UIUC College of Education (two graduate students and one faculty member).

As in the survey study, we found diversity among the models of these subject. Initially we looked for a different in the kind of mental models used by experts from those used by novices.

In our initial studies of the case studies, there are two major phenomena that stand out. Both of these are seen most clearly by contrasting the case studies of the two novices with the case studies of the four experts.

NovicesFor the two novices, the transcripts of their network problem solving efforts indicate relatively vague and undifferentiated models for the Internet and the Web. Subject N1 described a "superhighway" model for the Internet, and described the Web as "sort of like the Internet." Subject N2 described both the Internet and the Web as "kind of like a spider web." This lack of specificity is not surprising, since both reporting only having seen a demonstration of the web during class the previous week.

ExpertsFor the four case studies of people with considerable expertise with the Internet, Email, and the Web, their models are much more articulated, especially in contrast to the novices. In addition, in their transcripts, three of the four experts described using at least two different models, and in fact in two cases, articulated reasons for using the different models in different situations.

Here is an example of what one expert had to say in the case study:

Expert 1: "The overwhelming mental model I have for the Internet is what is called the rhizome...[the web is] also pretty rhizomatic. Rhizome is contrasted with the tap root model ...central tap root"

[describing his search strategy] "But I think although theoretically, I think that rhizomatic structures are interesting I still think....tap root hierarchical fashion. Let's go to the central sources first - let's get those names things and then.. "

So the two main phenomena are:

1. Experts' mental models of the Internet and the Web are much more elaborate and detailed than novices'.
2. Experts have multiple mental models of each area, and use the different models at different time when engaged in problem solving.

These findings are similar to the outcomes of previous studies contrasting experts and novices (Larkin, McDermott, Simon, & Simon, 1980; Chi, Feltovich, & Glaser, 1981). Novice physics problem solving have only one way to think about physics problems - they start writing equations, while expert physics problem solvers have multiple ways which they select among and switch between in the problems solving process.

In a sense, we were asking the wrong question earlier in our study. We wanted to know if expert models were different from novice models. They are different only in the degree of elaboration and detail, but not different in kind. Instead, the key difference is that experts have multiple mental models.

Summary

There are two main findings of these studies. The most striking is the diversity of mental models that people have, both for the Internet and for its uses. The second is that experts differ from novices not in the kinds of models they have, but in having multiple, detailed models that they switch between in the process of using the network to accomplish tasks.

These finding have obvious implications for teaching people how to use the Internet. The selection of any one mental model as a framework for training may be helpful to some students but not at all helpful to others. Teachers should be aware of the diversity of models, and build upon this diversity rather than trying to destroy it. The goal of instruction, instead, should be to help learners develop multiple, coordinated mental models that they can use at appropriate times to help them achieve their goals using networks.

The ability of the Internet and its most common applications to support a diversity of views may be to a considerable extent the basis of the "ease of use" that people report. As Internet applications become more heavily graphic, we may see the unexpected result that these applications become easier to use for some people (those who have and use mental models that contain the same visual elements), but actually harder for many others to use. This is one of the reasons that such studies of the mental models that people actually have of telecommunications may be very useful to the developers of the next generation of network tools.

Network Mental Models Questionnaire

We are conducting a study of how best to teach people about computer and information networks. We'd like you to fill out the questions below. We will protect your identity in any reports of our findings. If you agree to participate, please sign below.

Signature: Date:

Please contact me if you have any questions. Thanks!

Jim Levin 244-0537 Room 130 Education Building

Please circle the appropriate choice below:

My experience of teaching: none 1-5 years 6-10 years 11-20 21 or more

My previous experience with computers: none little some quite a bit lots

My previous experience with networks: none little some quite a bit lots

Previous experience with email: none little some quite a bit lots

Previous experience with newsgroups: none little some quite a bit lots

Previous experience with Gopher: none little some quite a bit lots

Previous experience with WWW: none little some quite a bit lots

Other network experiences:

We are interested in what "mental models" you have for using computer networks. For example, people often think about the atom as being like a miniature solar system. The solar system is then a mental model of the atom.

What "mental model(s)" do you use to think about the Internet?

What "mental model(s)" do you use to think about electronic mail?

What "mental model(s)" do you use to think about the World-Wide Web?

Would you be willing to talk with us in more detail about your mental models?

Yes No

If yes, please write your phone number here:

What days and times would you be available this week to meet for about 30 minutes?

Network Mental Models Questionnaire

[the first page of the case study was a repetition of the survey shown in Appendix A]

[Page 2]

Network Task: Lesson Plan with Network Resources

Imagine that you have a class of your choice to teach tomorrow and that you plan to use resources for the class from the "Web". You will have a total of 20 minutes to find relevant materials on the Web. Add the network documents you feel are relevant to the "Bookmark" list. Also, please tell us:

¥ the main topic or topics of the lesson plan,

¥ a general outline for the lesson,

¥ how the network materials are superior or inferior to non-network materials you would typically have available for a lesson such as this.

We want like you to "think aloud" as you do this task. Just say aloud the thoughts that come to you as you work on the Web and make your selections. Do you have any questions?

Feel free to write down any notes or thoughts you have on the front and back of this page.

[Page 3]

Experimenter Instructions for Think Alouds

Read the following instructions to the subjects:

In this study we are interested in what you think to yourself as you perform some tasks that we give you. In order to do this, we will ask you to think aloud while you are doing the network tasks on the computer. What I mean by think aloud is that I want you to say out loud everything you would ordinarily think to yourself silently. Just act as if you are alone in the room speaking to yourself. If you are silent for any length of time I will remind you to keep thinking aloud. At some points during the session, I might ask you about what you are doing.

Any questions?

Why don't we start with a practice problem for thinking aloud.

Think aloud while you tell me how many windows there are in your house.

After they are done, ask them how it was. Then ask if they would like to practice another think aloud. If so, then:

I want you to multiply two numbers in your head. Think aloud while you multiply 24 times 34.

End of Session Questions

I am interested in having you reflect on the process of doing the network task you just completed.

Were there any general thoughts you had while doing the task that you did not mention while you were thinking aloud?

At the beginning of our session, you indicated a mental model [or models--check their responses] you had about electronic networks. In what ways do you think your mental model [or models] of electronic networks influenced you while you were doing the task?

Next, ask some probing or clarifying questions based on what she or he says.

Any other comments you would like to make?

Be sure to thank them for participating!

E1:

The overwhelming mental model I have for the Internet is what is called the rhizome...[the web is] also pretty rhizomatic. Rhizome is contrasted with the tap root model ...central tap root"

[describing his search strategy] "But I think although theoretically, I

think that rhizomatic structures are interesting I still think....tap root

hierarchical fashion. Let's go to the central sources first - let's get

those names things and then.. "

E2:

"My model for the Internet that I've come up with just recently has been

sort of a salvage ..... which comes from experience. And, I don't want to

say junk yard cause that -.... talk about .. that's now what I want ....

but rather it's a place where is a lot of stuff there that I don't want

and if I need a part for my car I'm going to go to the Ford section and

there might be a light working on one and a clutch cable on another and

just kind of strip from it what I want and that view of it has evolved and

I think away from the one I was given by the name web. And, thinking that

- a place I want to be and I jut need to find my way there and once I'm

there I'm - I don't think that way anymore because I'm going to have to

pull - I'm going to have to call it's much more that I'm going to have be

an active person....not trying to get somewhere..... ... salvage .... pull

the information .... I think that does have an effect on the way I use it."

"My model - well, let's see, how do - which - the evolved together. I

think it's really the web. "

E3:

"Well, I was reading yesterday that someone compared the WWW to a city. "

"the Internet .... as a spider Web than anything else cause I had that

image before the WWW came out so, I feel funny when I say when I think of

the Internet I think of the Web"

"oh another one that I use to explain is, I just use the phone company, how

we are all connected ...... and sort of works."

[on the relation between her mental models and her actions]: "I don't think

that there's any relation at this point - maybe when I was starting out.

But at this point I've got a pretty good concept of what's there that I

don't think about it anymore. Maybe subconsciously I do, but I don't think

about it really."

E4:

Internet as "computer network - all the computers are connected"

"All the books ... will be there and anyone can access"

References

Chi, M. T. H., Feltovich, P. J., & Glaser, R. (1981). Categorization and representation of physics problems by experts and novices. Cognitive Science, 5, 121-152.

Gentner, D., & Gentner, D. R. (1983). Flowing waters or teeming crowds: Mental models of electricity. In D. Gentner, & A. L. Stevens (Eds.), Mental models. Hillsdale, NJ: Erlbaum.

Johnson-Laird, P. N. (1983). Mental models: Towards a cognitive science of language, inference, and consciousness. Cambridge, MA: Harvard University Press.

Larkin, J. H., McDermott, J., Simon, D. P., & Simon, H. A. (1980). Expert and novice performance in solving physics problems. Science, 208, 1335-1342.

McCloskey, M. (1983). Naive theories of motion. In D. Gentner, & A. L. Stevens (Eds.), Mental models. Hillsdale, NJ: Erlbaum.

Norman, D. A. (Eds.). (1983). Some observations on mental models. Hillsdale, NJ: Erlbaum.

Norman, D. A. (1988). The design of everyday things. Reading, MA: Addison-Wesley.

Norman, D. A., & Draper, S. W. (Eds.). (1986). User centered system design: New perspectives on human-computer interaction. Hillsdale, NJ: Erlbaum.