Abstract 1. Introduction 2. Evaluation description 3. Findings 4. Discussion 5. Conclusions 6. References 7. Acknowledgements

CTER OnLine: Evaluation of an online master of education focusing on curriculum, technology and education reform Sandra R. Levin, James A. Levin, and Gregory L. Waddoups University of Illinois & Brigham Young University slevin@uiuc.edu, j-levin@uiuc.edu, greg_waddoups@byu.edu Copyright © 2001 IEEE. Published in the Proceedings of the Hawai'i International Conference on System Sciences, January 3-6, 2001, Maui, Hawaii. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.

In the summer of 1998, the College of Education at the University of Illinois in Champaign-Urbana began offering courses in an online format that would culminate in a Master of Education (Ed.M.) degree (see Ed-Online at http://www.ed.uiuc.edu/ed-online/). With financial assistance from the UI OnLine initiative http://www.online.uillinois.edu/ , courses were redesigned and offered to education professionals. Initially, the intention was to fill the needs of two groups of professionals. CTER OnLine focuses on topics related to Curriculum, Technology, and Education Reform http://cter.ed.uiuc.edu, and is geared toward practicing K-16 teachers and administrators while the other initiative, HRE OnLine, targets human resource educators.

For the past two years, we have conducted both formative and summative evaluations of the Ed-Online initiatives. This paper focuses on the evaluation of CTER OnLine from both program development and student learning perspectives.

We identified five groups of stakeholders and specified their interest in the development and conduct of online degree programs. These stakeholders include administrators at the university, college and department level, college faculty, and students.

University administration was interested in creating online initiatives. At the college level, there was an interest by administration in increasing departmental and college capacity, maintaining high academic standards, providing programs that are self-supporting, and utilizing college resources wisely.

Department administrators share many of these concerns and must also deal with issues within their departments; for example, departments must be concerned about the amount of money spent on personnel (e.g., faculty and staff), and on computer technologies (e.g., hardware and software). They must also be concerned about providing quality programs and faculty satisfaction with their online teaching experiences.

Faculty members who teach online courses have a stake in the design, development, and conduct of online graduate programs. They must consider the amount of time it will take to learn new technologies, develop course content, and interact with students via the use of information technologies. College faculty who are not participating in online courses are also stakeholders because the decisions made will set precedents in the college that influence the allocation of funds for teaching online classes and the use and purchase of computer technologies within the college.

Lastly, students have a vital interest in the development and conduct of these online Master of Education degrees. Students expend a great deal of time, money, and energy completing a degree program. They are concerned with a) the quality and usefulness of their learning, b) the quality of interactions between themselves and the instructor, c) the quality of interaction between and among their classmates, d) the ways in which this learning is useful for their current jobs, and e) how the program is viewed by their administrators. Potential students are also watching the outcomes as one online program after another reports on its successes and failures.

The stakeholders’ concerns framed our selection of evaluation questions and the selection of data collection procedures. Evaluation questions were classified into six areas: student satisfaction, faculty satisfaction, economic viability, departmental capacity, interdepartmental collaboration, and college level infrastructure. A combination of survey data, interviews, and observations were used to determine the effectiveness of the program and the extent to which it is developing capacity that promotes its long-term viability [1].

Formative and summative data were collected from surveys administered online. The twenty-six CTER students were asked to complete three program surveys throughout their two-year program. During an on campus orientation meeting, a pre-survey was administered that assisted faculty in determining the students' level of computer expertise and their attitudes toward learning in an online environment. Mid- and post-surveys were administered after the first year and at the end of the two-year program, respectively. These surveys tracked students' progress and changes in attitude over the two-year period.

Course surveys were also administered at the end of each semester and provided faculty members with information about their instruction, course content, and use of technologies. In some cases, faculty members implementing innovative concepts into their teaching (for example, online simulations) administered mid-course surveys to obtain student feedback so that they could modify their courses in progress.

In addition, during the same two-year period, two college-wide surveys were administered that asked faculty about their use of technology and their beliefs/reactions/opinions concerning online programs. The College also conducted an evaluation of the Office of Educational Technology, the office that coordinates Ed-Online and manages technology resources in the college. These detailed formative evaluations were designed to improve the support for online education in the College of Education.

Faculty members teaching CTER OnLine courses were interviewed after their courses were taught. CTER support staff were also interviewed to determine the type and amount of technical support needed both for students to learn and for faculty to develop and teach online courses.

Four CTER students were interviewed and observed as a part of four in-depth case studies [2] [3]. Eight interviews were conducted with each of the case study students over the two-year period. Exit interviews are still in the process of being analyzed. Periodic observations were also made of the case study students in the context of their online classroom participation, in the context of working online and face-to-face in small groups with other CTER students, and in the context of their own classroom teaching.

A combination of synchronous and asynchronous technologies were used in these courses. Course descriptions, syllabi, grading procedures, student participation and expectations were available on the web and a variety of communication tools were used depending on the type of interaction needed. Instructors used the WebBoard® conferencing system for asynchronous discussions and for synchronous text chat sessions. Individual email and group email reflectors were also used for one-to-one or one-to-group correspondence. RealPlayer® was used to stream and archive audio and video files and text transcriptions of audio files were provided for hearing impaired students. Audio-narrated PowerPoint® presentations were created for some topics. Tapped In, a multi-user object-oriented environment (MOO) coordinated by SRI, International [4] [5] provided interactions between the CTER students and educators across the country. A specialized tool, known as CTERbase/TEbase, was developed to help faculty provide individual feedback to students, grade assignments, and publish exemplary work on the web [6]. In some courses, faculty members used Blackboard® (a.k.a. CourseInfo), a course management system. In each course, the instructor first thought about the type of interaction he/she wished to have with the students, then picked technologies that best handled the interaction.

CTER OnLine faculty members found that utilizing multiple technologies and multiple learning frameworks allowed them to create a learning environment in which web pages, communication tools, and electronic portfolios were woven together relatively seamlessly [7] [8]. The instructors combined multiple methods of instruction to enhance student learning [9]. The frameworks developed for online asynchronous discussions allowed students flexibility to participate on their own time. Students currently teaching in K-12 and college settings were given the opportunity to incorporate their coursework into their own classroom practice. Simulations were also used to provide experiences dealing with sensitive topics and issues. Various grouping arrangements allowed students to learn from one another. These multiple methods also demonstrated how current educational theories on learning can be incorporated into practice.

Findings are reported on the six areas identified as important by the stakeholders in the development and delivery of online degree programs: student satisfaction, faculty satisfaction, economic viability, departmental capacity, interdepartmental collaboration, and college level infrastructure.

Students were asked to complete course and program evaluation surveys. For course evaluations, student satisfaction was assessed using the University of Illinois’ Instructor and Course Evaluation System (ICES) that is used to evaluate all campus courses. Students were asked to rate the overall quality of the course on a five point Likert type scale (1 = exceptionally low rating; 5 = exceptionally high rating).

In general, CTER students were satisfied with the CTER OnLine courses. Of the eight courses taken by the first set of CTER students, the ratings for the courses ranged from 4.80 to 3.05, with a mean rating of 3.83. This pattern of course evaluation rating is similar to the general pattern of course evaluations for on-campus courses at UIUC.

In addition to course evaluations, the CTER students filled out three surveys designed to help evaluate the overall program. A pre-survey was completed in June 1998, a mid-survey in June 1999, and a post-survey in May 2000. Part of each survey was a self-evaluation of the students' technical skills. As shown in Table 1 below, their self-ratings of skills rose in all cases as the program progressed.

Table 1. CTER student self-rating of technical skills at the beginning and end of the two year program.

CTER students were also asked to rate their attitudes toward technologies. As shown below in Table 2, they started out largely positive toward technology and remained positive throughout the program with only minor changes over the course of the program. Questions 2 and 3 were written in the negative form and therefore show low numbers on the five point scale.

Table 2: CTER student attitudes toward technologies at the beginning and end of the two year program.

As shown in Table 3, students were asked to rate their attitudes on relevant issues related to different approaches to learning. For all questions except 3, 4, and 7, there are mostly insignificant changes in the mean differences. The largest changes were noted in question 3, suggesting they do not learn as well in a lecture format; question 4, showing a perceived decrease in there learning ability through online projects; and question 7, realizing the importance of time management skills when taking an program online. The questions in Table 3 were dropped from the post-survey in lieu of the exit questions discussed in Table 4 and open-ended questions about the program in general that required extensive responses.

Table 3: CTER student attitudes toward their own learning approaches at the beginning and middle of the program.

Finally, in Table 4, the CTER students were asked to directly rate their satisfaction with the CTER OnLine program. More students responded that the program exceeded their expectation than those who were disappointed. More said that they spent more time working on their Master degree than they expected. Finally, the students gave the program a grade of A-/B+.

Table 4: CTER student ranking of the CTER OnLine at the end of the two year program.

The final measure of student satisfaction in the program was found in the drop out rate. Currently, the drop-out rate of CTER students is zero. All 26 of the students who started in the summer of 1998 graduated in May 2000, and all 29 current students who started in the fall of 1999 have remained in the program as of September, 2000. While we do not expect this rate to remain at zero forever, it is considerably lower than most other online programs (and considerably lower than most face-to-face programs as well). Why has this drop-out rate been so low? Based on our evaluation so far, there are four contributing factors: 1) Students formed informal support groups, so that they could work together on course work. Many students said that they would not have made it without these informal support groups; 2) The program director and program coordinator have provided another avenue for communication between the students and instructors. This gave the students a feeling of continuity over time and a feeling that if they had a problem they had a potential outlet; 3) The ongoing evaluation provided feedback to improve the communication and support between students and faculty; and 4) Flexibility in the program has allowed students to shape the program somewhat to their own specific needs. This flexibility is being institutionalized in the program, which is now defining a set of core required courses and elective courses.

Faculty were interviewed over the course of the first two years of CTER OnLine. They generally expressed satisfaction and excitement about being involved. For several of the faculty members with considerable technical expertise, this was a chance to put into practice innovative ideas that they had previously developed on a smaller scale. For faculty members with less technical expertise, it was an opportunity to learn a range of educational technologies in a supportive context.

Four factors contributing to faculty satisfaction were identified during the interviews: 1) CTER faculty meetings, 2) Ed-Online meetings, 3) technical support, and 4) alignment with faculty research goals. Monthly CTER faculty meetings were held throughout the two-year period that allowed an open discussion of experiences, ideas, and plans. The faculty members with less expertise used these meetings as a way to start thinking about which of the range of technologies to plan to use; those with more expertise used it as a way to reflect on experiences and to plan for future uses. Out of these meetings came a sense of an overall "CTER philosophy," and shared lessons about how online courses, especially with adult learners, are different from face-to-face courses; for example, faculty discovered that CTER students were generally more willing to challenge faculty members than on-campus graduate or undergraduate students. Each faculty member initially thought that the challenge was unique to his/her own course, but was surprised to learn that student challenges of various sorts occurred across most courses. The "consumer mentality" of many of the students was a source of considerable discussion, especially when thinking about the future directions of education and of universities more specifically. This refers to the ability of students to shop around for courses and programs and the tendency for this to increase the likelihood that students will judge a course or program on its ability to provide immediate advantages for their current needs, while ignoring possible long-term benefits.

A second factor contributing to faculty satisfaction was the interaction that occurred between faculty members in different online programs in the College plus others at the College and university level interested in the development and coordination of online programs. The Office of Educational Technology in the College of Education coordinated these monthly Ed-Online meetings. Faculty members were given the opportunity to discuss their experiences, demonstrate technologies and techniques, and learn about new technologies coming to market to improve online instruction. This sharing between programs provided a broader context for faculty members to understand their teaching and the nature of online media.

CTER teaching assistants provided technical support to CTER faculty members and students. This support was crucial for faculty satisfaction with course development and course implementation. Technical problems occurred, but the technical support staff was able to solve these problems quickly and with minimal disruption for the faculty members and their courses.

Lastly, many of the initial faculty members participated and were satisfied with teaching CTER courses because it provided a basis for an extension of research in the use of educational technologies. This factor may be less salient as more faculty members become involved whose research interests lay in other domains.

The economic viability of online programs is directly related to its sustainability. Therefore, the cost of providing an online program has been a major concern for all levels of administration. A number of issues related to viability were raised: 1) How much does it cost to deliver a course using distance learning technologies vs. the traditional face-to-face method? 2) How much demand is there for the online courses by on-campus students as well as by off-campus students? 3) To what extent do costs change over time? 4) How much additional revenues do online programs bring in? 5) Do these programs provide opportunities for gaining additional revenue for the College and the university?

The University of Illinois administration provided "seed" funding to colleges and departments to begin developing online programs with the understanding that the programs be self-sustaining after the initial funding expired. Given this stipulation, the Dean, department chair and program director developed budgets and estimated tuition revenues. It quickly became apparent that the cost of developing and sustaining an online program was not possible to support under the current off-campus tuition rate, and it was suggested that a professional tuition rate be established for online programs. The University trustees approved a new tuition rate that eased the budget burden.

While it was difficult to estimate all costs and the percentage of costs for some services used by online courses, college and department administration came to an agreement about online program budgets. Program budgets included the cost of program administration and secretarial staff, technical and course development support, equipment and supplies. Some of these costs are expected to change over time, and budgets must be reviewed annually. Faculty members have been expected to teach courses on-load or as part of the department’s summer budget.

In the past two years, online education has been highly publicized. Virtual universities are sprouting up, higher education institutions are jumping on the bandwagon, and private companies are advertising online courses and degreed program. While faculty members have received increasingly frequent requests from prospective students interested in a masters degree program online, the question remains unanswered as to how much of a demand there is for an online program.

The College announced the development of CTER OnLine to several school districts throughout Illinois that had previous ties to the university. The announcement was sent in March 1998 that we would be accepting students beginning in the summer of 1998. Forty-five students applied to the program by May 1998, and 26 students were accepted. Due to the initial online course offering and the evaluation research component, faculty members insisted that class size be kept to a reasonable level. Since the initial announcement in 1998, we have received over 700 requests for information about the program from all over the world. During the second year of the program, 29 new students were admitted. All 26 students in the first cohort stayed in the program and graduated in May 2000. We have not had any students drop out of the second cohort and we have admitted a third cohort of twenty-six students in the Fall, 2000 semester.

In terms of department capacity, we attempted to answer the following questions, relating to issues in developing and conducting an online program: 1) What does it take to build capacity? 2) What are the barriers and possibilities? 3) Is there a meshing of these online course offerings with other department programs, on campus courses and the like?

Department faculty members play a vital role in how online courses and programs are developed and offered. CTER OnLine was fortunate to have a group of faculty members who were willing and able to get this started. At the department level, faculty members were given permission to teach an online course on-load. This is not necessarily the case for each department in the College or for other Colleges with online programs at the university. Issues of equity arose when some departments offered faculty members release time for course development while other departments insisted that faculty members continue to teach their normal course load plus teach the online class as over-load. The same issues of equity arose across the university as different colleges offered a variety of incentives for faculty. A university-wide committee was formed to discuss the issue of faculty salaries and incentives and to make recommendations to the administration.

After the first year, CTER OnLine began to offer additional courses, and new faculty members participated. These additional courses not only included more faculty members in the online initiative, but also allowed students more flexibility in the courses they were able to take. As the initiative moves into its third year, we continue to develop and teach new or revised courses in an online format. From our initial eight courses, we have now expanded to offering five required and nine elective courses.

Faculty members from four departments volunteered to develop and teach courses in the online format. This directly affected the sustainability of online efforts in the College which led to another set of questions: 1) What department, college, and university resources are being used and created to insure ongoing capability to conduct online programs? 2) How do departments who offer foundation classes feel about putting their classes online? and 3) Are collaborative relationships being developed between and among faculty and staff to facilitate the development and conduct of online classes?

CTER OnLine drew support from resources at the department, college, campus, and university level. The Office of Academic Outreach provides assistance with course registration, marketing, off-campus library services, server space and student accounts. The UIUC campus provides these services for all off-campus courses. UIUC also supports the Center for Educational Technology which provides course management systems and maintains course information for both on-campus and online courses.

The Office of Educational Technology in the College holds monthly Ed-Online meetings for all faculty and staff involved in online courses throughout the College. These meetings are an opportunity for different online initiatives to share their experiences, both good and bad, with one another. This office also maintains a Faculty Development Center, providing all College faculty members with training on a variety of educational technologies.

Most of the initial faculty members in CTER OnLine were proficient using technologies and assisted other faculty members who were less experienced or uncomfortable with teaching in an online format. Monthly scheduled meetings of CTER OnLine faculty and staff were also held to discuss the technologies being used and problems that had arisen, in order to jointly find solutions to the problems. As an incentive, CTER OnLine faculty members were given additional teaching assistant support for two semesters in order to develop and teach the online course. In addition, faculty members who wanted to work from home were given higher speed modems and an outside Internet Service Provider to use when university access was slow.

Instead of offering four half-unit foundation courses for CTER OnLine, two one-unit courses were developed and offered, reducing the number of courses for a Master of Education degree from ten to eight. Two departments are affected by providing these online courses. In one department, the course is taught on-load, in the other department, the course is taught in the summer. As the program continues, we are seeing a greater demand for foundation courses taught in an online format. Staffing issues are being addressed. The home department of CTER OnLine also provides office space, computers and servers for the program, technical support graduate assistants and secretarial support to process new student applications.

Over the past two years, the College infrastructure has seen extensive growth in the area of technology. New computers were purchased for a student computer laboratory and for the Office of Educational Technology. The building's computer network is in the process of being upgraded, and the Faculty Development Center was established. Resources provided CTER OnLine at the College level through the Office of Educational Technology include server space, backup maintenance, classroom and computer lab space for face-to-face meetings, equipment checkout facilities, and overall coordination of technology used for both on-campus and online courses.

Based on data gathered from most of the stakeholders, CTER OnLine has been successful during its first two years. What lessons can we learn from the nature of the success, both to improve this online program and for online programs more generally? There are several major lessons that we can generalize from our experiences, which relate to broader theoretical frameworks developed in cognitive sciences.

One general aspect that stands out from our research of CTER OnLine is the importance given to the active learning community that was created and maintained over the first two year period. Sometimes a learning community will "spontaneously" emerge from a given course or program of courses, but many times such a community fails to grow. In CTER OnLine, explicit efforts at community building were taken from the very beginning, and community maintenance efforts continued throughout the first two years. All of the students and many of the initial faculty participated in a two-and-a-half day face-to-face orientation in June 1998. A wide variety of techniques were used to help establish a sense of community. During the orientation and other face-to-face meetings, nametags, icebreaker activities, group activities, social activities, and welcome greetings from the Dean and faculty members establish personal and professional interactions among the group. Individual and group digital pictures are taken for the CTER web page and personal web sites that are used for students' electronic portfolios. These techniques contributed to the idea that the students were part of something larger, CTER OnLine.

In CTER OnLine, attention is paid to communication between students and faculty members, students and technical support, and between students and other students. Active involvement by the CTER director and the CTER coordinator to provide alternative channels of communication rather than just through the course instructors have also helped keep the community together. These techniques reinforced the idea that the students were members of CTER, not just fellow students in a course or set of courses [10].

The importance of communities of practice for the professional development of teachers is becoming increasingly clear in the literature [5] [11] and the evaluation of CTER OnLine presented here certainly supports that concept.

Recent cognitive science research has clarified the nature of expertise in a given domain [12] [13] [14] Expertise consists of having multiple ways of thinking about a given domain, multiple conceptual representations that are coordinated so that the expert can move from one representation to another. In addition, the expert has the "meta-expertise" to know which conceptual representation to select at each point in the process of dealing with the domain and when to switch from one to another. This conception of expertise was used in CTER OnLine in two major ways, multiplicity and connections to contexts of practice.

In CTER OnLine, we explicitly built "multiplicity" into the program and courses [9]. We used multiple media tools for teaching and learning, multiple instructional formats, multiple student activities, multiple student assessment and program evaluation techniques, and multiple contexts for learning. The genesis of this use of multiplicity was the goal of helping students acquire multiple coordinated conceptual representations of the knowledge domains.

The courses in CTER OnLine were largely project-based in design, with student projects in most cases selected by the students to have strong connections to their own teaching contexts. In this way, students could learn and become expert with abstract conceptual representations in a number of different domains, but with a strong coordination with the concrete conceptual representations they have developed in their own teaching practice. Through this explicit "connection" process, the knowledge and skills that the CTER students acquired is much more likely to "transfer" to their ongoing practice. Their learning was also much more motivated, as they could see how and why they needed to learn the knowledge and skills in the courses of the program.

Current theories of learning have pointed to the importance of multiple contexts within which the learning occurs [15] [16]. Not only are the CTER students' contexts of work important, but also important are the other contexts as well (family, community, etc.). These students are busy adults and their identity as "student" is only one of many. We have found that identity formation, extension, and maintenance mechanisms play an important role in student learning [2] [3]. Recognizing these multiple identities and the constraints that they put on student involvement is crucial for a successful online program. This is why most of these people chose an online program, because of the flexibility it provides in time and place of learning.

Given the extensive evaluation of the first two years of CTER OnLine, we are now in a position to make a set of recommendations for future action. Some of these are somewhat limited to online programs with a similar set of goals and students as CTER OnLine, but many are relevant more generally.

Functional learning communities create powerful learning environments: 1) Employ systematic group formation techniques. 2) Employ systematic group maintenance techniques. 3) Provide for flexibility to enable community members to shape the program to fit their own needs. 4) Provide for adequate support for participants in the learning community, from multiple levels of the university. 5) Provide mediator roles between students and instructors.

The potential conflict between providing a cohort model and providing flexibility has been resolved for CTER OnLine by moving to a "semi-cohort" model. In this model, all students in a "cohort" go through a common orientation and take the first two core courses at the beginning of the program. For the next four courses, students get to select from a number of elective courses. The cohort then comes back together for the remaining two core courses at the end of the program. Other online efforts may well have other solutions to this issue.

Consider the value gained from selecting a particular instructional method or technology tool vs. the cost of implementing the new method or technology: 1) which instructional media will result in the best learning vs. the expense of purchasing the latest new products; 2) would asynchronous or synchronous interaction fulfill the communication need best vs. the expense of using the different communication methods; 3) what is the value of conducting a face-to-face meeting vs. the cost of travel for all the participants; 4) when and for what purpose should group vs. individual instruction take place; 5) what do the students gain by using multimedia vs. the cost of the purchasing hardware, development, and training everyone to use the multimedia; 6) when is it most appropriate to use simulations vs. authentic real-life activities vs. classroom exercises.

For any of these considerations, it is important to consider the value of simplicity and consistency in instructional design vs. the value of multiplicity in encouraging the development of multiple coordinated conceptual representations in the domain of expertise.

Conducting systematic program evaluation is valuable when implementing new online educational programs. This is especially true when utilizing new media. It is critical to build in mechanisms for providing guidance and corrective feedback to instructors and program administrators. There are three aspects to consider: 1) conduct both formative and summative evaluations; 2) use the formative evaluation to improve instructional methods and to help identify faculty training and support needs for teaching in an online environment; 3) summative evaluation provides a better understanding of the learning that takes place over the course of a longer period of time.

It was our goal to provide support for this final point through this paper, in that the systematic evaluation of CTER OnLine has provided a valuable set of formative and summative outcomes that have improved the program while providing guidance for future online efforts.

[1] Patton, M.Q. (1997). Utilization focused evaluation. Thousand Oaks: Sage Publications.

[2] Waddoups, G. L. (2000). Extending, adopting, and defending practitioner identities: K-12 teachers participating and learning in an online community of practice. Unpublished doctoral dissertation, University of Illinois, Champaign IL.

[3] Waddoups, G. L., Levin, S. R., & Levin, J. A. (April, 2000). Developing a community of practitioners via advanced technologies: The case studies of CTER OnLine. Paper presented at the American Educational Research Association Meetings. New Orleans, LA. Available online at http://lrs.ed.uiuc.edu/aera00/47.47/wll.aera.19apr00.html

[4] Schlager, M. S., Fusco, J., & Schank, P. (2000 in press). Evolution of an on-line education community of practice. To appear in K. A. Renninger & W. Shumar (Eds.), Building virtual communities: Learning and change in cyberspace. NY: Cambridge University Press.

[5] Schlager, M., Fusco, J., & Schank, P. (1998). Cornerstones for an on-line community of education professionals. IEEE Technology and Society, 17(4), 15-21,40.

[6] Levin, S. R., Buell, J. G., & Levin, J. A. (2000). The TEbase initiative: Research, development and evaluation for educational reform. Journal of Computing in Teacher Education, 16(3), 6-11. Available online at: http://faculty.ed.uiuc.edu/j-levin/award-paper.html

[7] Pincas, A. (1998). Successful online course design: virtual frameworks for Discourse Construction. Educational Technology & Society, 1(1). Available Online: http://ifets.gmd.de/periodical/vol_1_98/pincas.html

[8] Knox, E.L. (1997) The pedagogy of Web-site Design. ALN Magazine. Available Online: http://www.aln.org/alnweb/magazine/issue2/knox.htm

[9] Levin, J. A., Levin, S.R., & Waddoups, G. L. (1999). Multiplicity in learning and teaching: A framework for developing innovative online education. Journal of Research on Computing in Education, 32(2) 256-269.

[10] Levin, S. R., & Waddoups, G. (Feb 2000). CTER OnLine: Providing highly interactive and effective online learning environments. Proceedings of the SITE 2000 Conference. San Diego, CA.

[11] Riel, M., & Levin, J. (1990). Building electronic communities: Successes and failures in computer networking. Instructional Science 19, 145-169.

[12] Levin, J. A., Stuve, M. J., & Jacobson, M. J. (1999). Teachers' conceptions of the Internet and the World Wide Web: A representational toolkit as a model of expertise. Journal of Educational Computing Research, 21(1), 1-23.

[13] Chi, M. T. H., Feltovich, P. J., & Glaser, R. (1981). Categorization and representation of physics problems by experts and novices. Cognitive Science, 5, 121-152.

[14] Larkin, J. H., McDermott, J., Simon, D. P., & Simon, H. A. (1980). Expert and novice performance in solving physics problems. Science, 208, 1335-1342.

[15] Clift, R.T., Mullen, L., Levin, J., & Larson, A. (2000, in press). Technologies in contexts: Implications for teacher education. Teaching and Teacher Education.

[16] Clift, R. T., Thomas, L., Levin, J., & Larson, A. (1996). Learning in two contexts: Field and university influences on the role of telecommunications in teacher education. Paper presented at the American Education Research Association Annual Meeting, New York.

We would like to thank UI Online, Ed-Online, the College of Education, and the Department of Educational Psychology at the University of Illinois for their support of CTER OnLine and the evaluation reported in this paper. We also would like to thank the CTER OnLine students, faculty and support staff, without whom CTER OnLine would not be such a powerful context for learning and research.