EENOVATE – A New Agenda for Universities in Candidate Countries a Minerva project
Research Methodology (proposal)
Draft 23 June 2005
This document provides the methodological basis to be use by EENOVATE project team for an actionresearch approach in evaluating models for client-focused University elearning services. It also presents the elearning concept map developed by the RO1, RO2, UK1, ES1, BG1, IT1.
The methodological framework of the evaluation research on the quality of academic elearning services is based on: • the theories and practice of evaluation research, and especially those applicable to distance education; • EENOVATE project design and the development process; • the context in which the project is implemented and could be replicated. An empowerment evaluation approach was adopted, in order to allow an active implication of the project actors and stakeholders within the evaluation research design and implementation phases. The context reflects the political, social, economical, and technological media within which the research and the whole project are developed. Various contexts magnify or diminish the research findings and their value. In this perspective, we consider important to assign concepts as innovation, change and evolution a significance that is relative to local/ regional framework – what is considered innovative in a particular context could be a current practice in another one. Designing the present research evaluation plan, the following ideas were taken into account: − − The actors involved in the project share the same professional backgrounds (also working in academic elearning environments) and are used to working in transnational projects, though coming from different cultures. Still, because of the context the higher education institutions are functioning in, the evaluation research tools will be tailored to each partner’s peculiar needs and experiences. There were established two “areas” to be surveyed: two countries from the EU member states (Western Europe) and two countries from candidate countries (Eastern Europe). There will be developed a common analysis model of elearning programmes, but with two sets of research instruments. Common basis for data gathering would help in building an integrated up-todate model for client focused university elearning services. − − Evaluation research will refer to the academic elearning practices, focusing on a variety of activities, interactions, procedures, products, and performances. Thus, it will not only be a “summative” exercise at the end of an elearning programme of higher education institutions, but it will also involve “in-the-making” dynamics, work styles, and reactions (– partly due to the involvement of the researchers in academic elearning programmes). − − Evaluation is an on going process, made of “formal” procedures as well as of “informal” interactions, and unpredictable contingencies. It would be restrictive to consider the present research evaluation plan complete and non-changeable. On the contrary, it contains the main trends to be followed, leaving a margin of manoeuvre in order to have a hold on unmanageable complexity.
What is the “object” of the evaluation research? What should be Scope of the evaluation evaluated? research There will be evaluated the [quality of] elearning programmes in 10 universities from Candidate Countries (Romania and Bulgaria) and 6 universities from EU member states (UK and Spain. researchers. students. students. professionals in higher education institutions. researchers. Self-administered questionnaire. What kind of research is to be developed? Type of research Stakeholders Evaluation research.
.education via the Internet. correlation research. will be disseminated directly in universities through national education networks. professionals in higher education institutions. ex post facto research. What main stakeholders are involved or implicated in the project and its evaluation? Policy makers. elearning practitioners. techniques and instruments will be used? instruments Survey.experienced through a technology interface. interview guides. and is Internet-enabled. What is the target group? Target group Teachers/ tutors. What kind of research methods.Component
What is the main purpose of.
Agreement on working definition of key concepts BG:
E – learning is: . Semi-structured interview. Research methods. questionnaire. techniques. educational software developers. operator-led questionnaire. . . focus group. elearning practitioners. focus group guides. or standalone computer. Who will carry out the evaluation research and with what Work share resources? Project partners. elearning programme coordinators. How will the evaluation main findings be used and disseminated Utilisation of the evaluation and what activities will be carried out to facilitate use? results The final report will be geared to the information needs of policy makers. action research.component of distributed learning that includes digital content. educational software developers. and who are the main audiences for Purpose of the evaluation the evaluation research? research The overall purpose is the development and validation of an up-to-date model for clientfocused academic elearning services. The report. network. together with a synthesis.
. . was extended from the primary. audio or videotape. simulations.
< electronic learning (1) Largo senso.it encompasses all learning undertaken. elearning solutions etc. computer mediated communication. (Bates. . dictionaries. elearning (or e-learning) stand for any educational situation significantly using ICT means. With CBL the student works through pre-designed material. taken from Anglo-Saxon literature.it encompasses wireless and mobile learning as well /mobile phones.). learning through multimedia etc. didactic games. The concept of educational software covers a large area of electronic materials/ applications developed to simplify/ support educational process: maps. etymological sense of learning through electronic medium. elearning platforms. intranet/extranet. interactive TV and CD-ROM. e-books. digital/ mobile/ online learning/ education.it is often used in distance learning programs. Some sources define Knowledge Management as a form of elearning.
.it refers to using electronic applications and processes to learn. tests. .essentially the network-enabled transfer of skills and knowledge. didactic movies. evaluation or as a mean of communication (to deliver/ receive individual learning tasks etc. . interacting by answering questions embedded within the materials and choosing options or routes through the learning material. computer-based learning. Within these CBL programs different kinds of media can be used such as audio and video. . learning. Defined like this. tutorials. whether formal or informal.its applications and processes include Web-based learning. computer mediated learning. PDA devices. covering nowadays the entire domain of educational interventions using modern ICT. presentations in various formats.there are two quite different forms of teaching by means of a computer: pre-programmed computer-based learning (CBL) and computer-mediated communications (CMC). mobile computers/. The term.it focus on putting the student in control. 1995) Bearing this definition in mind we can determine CMC as a part of e – learning. . through electronic delivery. But it includes any technologically mediated learning using computers whether from a distance or in face to face classroom setting (computer assisted learning). more as eeducation.content is delivered via the Internet. satellite TV. smart phones. . virtual classrooms and digital collaboration. encyclopaedias. the semantic area of elearning concept is interfering and variably overlapping a multitude of terms intercepting the variety of didactic experiences which could benefit of technological support: computer assisted instruction.. The computer and electronic/ multimedia materials are used to support teaching.
4. through individualisation of training – non-linear structure of the content. which delivers learning materials in a sequential and systematised order to be achieved by students in their own rhythm. asynchronous seminars. 3. the online education systems imitate and adapt traditional (face-to-face) education components: planning. 1995) – continuous restructure of learning teams
Principles of quality in elearning (Chickering& Ehrmann. specific content and methodology. in spite of the fact that face-to-face meetings may never take place and the number of "contact hours" is reduced or eliminated. (b) distributed resources – by using/ integrating/ accessing electronic libraries and multimedia materials. and collaborating on projects or problems promote higher level thinking skills and communication. interaction. (c ) interchangeable roles (Symmetric Knowledge Advancement – Scardamalia. 5. Communication between students and instructor is important for enhancing learning as well as for student motivation and involvement. Sharing ideas. elearning represents a type of distance education. research. Good distance learning gives rich and rapid feedback to students. refer to (claim for): (a) a student centred approach. sharing resources. The delivery is done through the new communication and information technologies. Distance educators should promote collaborative learning. and effective teaching is more than talking and encouraging memorization of text. 2.(2) In a more restricted sense. student autonomy and independence. by involving top specialists in students discussions. Distance education students should be able to regularly assess their own learning as well as get feedback from others. insufficiently explored and used.
. Peers can be invaluable in the learning process. Feedback can be as simple as a webpage link to answers for a problem set or an e-mail message or as complex as using "hidden text" options on special software to give a critical analysis of written work. support and evaluation. especially through the Internet (online learning/ education) that represents both the distribution medium and the communication channel between the actors involved. Technology that increases access through asynchronous or synchronous means is a necessary component of effective distance learning. 1987) 1. Effective learning involves more than listening and reading. through personalised learning path – different ways to compose learning objects taking into account beneficiaries needs -. Active learning should involve application and problem solving. as a planed teaching/ learning experience organised by an institution. Distance learning does not minimize or obstruct contact between learners and teachers. Good distance learning incorporates active learning. Good distance learning develops relationships and promotes collaboration among students. and simulation. Good distance learning stresses time-on-task. Learning at its best is as much a collaborative activity as an independent one. Good distance learning encourages and maximizes contacts between learners and instructors. Being used now in higher education and for continuous training. The extensions brought by technological medium.
a questionnaire for students (STUD) and a focus-group guide (FOCUS). Quality in elearning programmes is supported by an integrated institutional approach comprised of concern for and continuous amelioration of general policies. student performance. Distance learning should be challenging -. Three general assumptions stand for milestones in the evaluation research grid: 1. Quality in elearning programmes is revealed by the extent to which staff commitment is reflected in instructional activities and complementing services. time wasted on locating materials. Busy work. management. implementation and data gathering Analysis of main findings Analysis of conditions for transfer Elaboration of project report and setting a new agenda for CC Platform creation and development Information and dissemination Validation of project results/conclusions
. Good distance learning sets high standards for student performance. Some specific issues the project team focuses on are to be employed in each research instrument (as a question also for teachers/ tutors/ administrative staff and students) – for a triangulation of data collected. activity. a proper operationalisation of the elearning concepts concealed by various theories and practices could lead to a model pivoting the institutional policy transfer. 6. Each of these assumptions is addressed by different items in the research instruments: an institutional evaluation grid (INST). evaluation. and applied activity can make distance learning more challenging than many of the conventional "chalkand-talk" classrooms. staff and students satisfaction. Quality in elearning programmes is measurable through institutional outputs. the opportunities for interaction. and performances should be clearly communicated from the onset. distance learning should be organized such that students find their own best ways for approaching problems. support and technical facilities. and time wasted figuring out the glitches in the system minimize learning and maximize frustration. completing tasks. a interview guide for teachers/ tutors (TUT). curriculum. Instead. 7. a interview guide for elearning programmes managers (MAN). The expectations in terms of projects.indeed. and using learning materials. products.The distance-learning environment should be rich with reading. 3. and interaction. Clearly established criteria set the standards for judging excellence. Distance education should not incorporate a lock-step.
EENOVATE project general assumptions The general philosophy of the EENOVATE project relies in the assumption that differences in elearning programmes can be identified and analysed. Therefore. "one size fits all" approach. 2.
PROJECT MAIN DIMENSIONS (as outlined in the project proposal): Design. staff development. Good distance learning respects individual differences and allows students opportunities for learning that acknowledge those differences. collaboration.
5 universities in BG) EU countries Institutional evaluation grid (… universities in UK.research plan .3. Development of an Elearning Research Matrix including: . 5 universities in BG) Interviews with managers. curriculum and instruction.agreed working definitions . Desk research in CC and EUC (outcome: national reports and synthesis report) 1.1. …universities in ES)
. 1. including categories such as: institutional policies.principles of quality elearning program . teachers/tutors and students(?) (… universities in UK. Field research in CC and EUC (outcome: report on CC representative universities needs and report on EUC practices on identified needs) STAGE 2: Data analysis. evaluation. identification of CC needs and EUC practices STAGE 3:Identification of transferable policies from EU countries and analysis of conditions of transfer STAGE 4: Validation of outcomes at institutional level (based on focus groups with stakeholders in elearning at national level in both CC and EU) STAGE 5: Elaboration of recommendations for further action in CC countries (policy level) DETAIL OF STAGE 1.4.4. teachers/tutors and students (5 universities in RO.overview of elearning concept map.Monitoring and evaluation
RESEARCH WORKING PLAN
STAGE 1: Data gathered on current situation in higher education elearning programs (CC and EUC) 1.fundamental assumptions of the project . Refining the research methodology and design of research instruments 1.2. staff development. technical facilities. CC Institutional evaluation grid (5 universities in RO. …universities in ES) Interviews with managers.
Distance education program completion takes the form of credits or credentials that are recognized by professional accreditation bodies and employers. Program plans and budget include financial and administrative commitment to the continuation of a program for a period sufficient to enable students to complete a degree/certificate.2005
Program plans and budget include integration of distance delivery with the institution’s overall policy framework. Program plans include written policies for all aspects of the distance education program.Elearning concept map
Legend: CC.06. in the questionnaire/ interview/ focus group guides CC EUC
1. note that only some of the items below will be taken into account and reflected in questionnaires or interviews to be developed!
A. Distance learning program is supported by an institutional commitment to instructional quality and effectiveness.European Union Countries INST– Institutional Evaluation Grid MAN – Interview Guide for Elearning Programmes Managers TUT – Interview Guide for Teachers/ Tutors STUD – Questionnaire for Students FOCUS – Focus-Group Guide
Please. 5. 3.
Draft 23. 4. 2.Candidate Countries EUC. Elearning-related institutional policies
University’ distance education policy
Corresponding item no.
Teaching / learning materials are prepared by qualified content experts working with qualified design experts
Draft 23. Appropriate and necessary personnel include customer-oriented management that helps with information. class and gender bias. 10.
B. Study programmes are promoted via Internet. Access to elearning services is facile.2005
. Enrolment services are free of cultural. evaluation. submitting proposals. 3. the percentage of theory. application. scheduling) are clearly stated. the course description etc. course/program advising. Learner involvement in decision-making is possible. 7. Curriculum and Instruction
Materials. Management of student records consists in documentation of student achievement in each course and at completion of a program. racial. INST 1. 9.
Marketing 12. Intended learning outcomes are shaped with input from learners. 8. approval procedures. 2. application and registration procedures. Study programmes are promoted via traditional means. programs and courses 1. 5. 15. The course design has to fit some institutional standards that regulate the length. 11. There are stated clear quality standards to be achieved and/or maintained. 4. Program plans include an adequate budget to achieve stated program goals. Product/service information for potential distance-learning students is comprehensive and complete.06. 14.1. Distance programme development and maintenance procedures (establishing responsibilities. 6. Distance learning programmes are presented on distinct leaflets. 13. Learning outcomes are matched to instructional and learning activities. 16.Management 6. Intended learning outcomes are clearly stated. Intended learning outcomes are appropriate to the rigor and breadth of the degree or certificate awarded. developing and completing checklists.
7. class and gender bias. 19. The course provides opportunities for students to engage in activities and tasks that enhance comprehension. 19. 10. understanding and knowledge. learning/ lecture notes and additional learning resources. Multimedia use is suitable to the topic and to the constraints of delivery.
INST 1. 11. TUT 19. 17. quizzes and examinations. Course maximizes the opportunities for regularized and ongoing interaction between teacher and students. and between students and learning environment. TUT 19. Curriculum content is culturally sensitive and free of bias. Courses accommodate multiple learning styles. 21. Course content is provided in segments that are technologically deliverable. 22. answers to questions/ quizzes. course schedule.1. 12. a portfolio of acquired learning. Teaching / learning materials are free of cultural. course activities and assignments. among students. STUD x.).1. 8.
The course content is approved/ validated by a (internal/ external) commission.06. Content 16. racial. 14. 18. Teaching / learning materials are readily available and learner friendly – able to be used by the average student. INST 1. 9.1.. STUD x.e. course/ project objectives. Plans are developed to have materials delivered to or available for students at remote sites in a timely manner. STUD x.
A complete learning package includes information about most of the following: course description. information about the instructor[s]. 20.2005
. 23. The courses’ content is updated regularly (at least once at every 3 years).
Curriculum content is updated consistently. consistent use of file format. and appropriateness. utility.
Distance learning methodology
Draft 23. Curriculum content is relevant. Course is consistent in delivery and information format (i. 24. 15.
13. INST.1. TUT navigation etc. Syllabi are reviewed regularly to ensure clarity. TUT 19.
Career counselling and/ or appropriate placement is ensured for the distance education programme graduated
Draft 23. Teaching staff/ tutors provide prompt feedback . 43. 27. Scheduling and timetabling is flexible and responsive to learners. 47. 39. If required. applying and constructing information. 37.2.
Support 36. The distance learning environment is rich with interaction. research. STUD 3 Specialised personnel offer advice and counselling within the distance education program. 44. Any required course-specific tutorial/ learning support resources and services are available online or at students’ locations. 34. 46. Specialised personnel deal with students’ technical difficulties. 28. 41. 31. Collaborative learning (student-student) is requested by specific tasks during the courses/ online seminars. appropriate interaction between students and faculty. 29. Distance students are given advance information about succeeding in a distance learning environment. Student-led/ instructor monitored forums are used. labs. Distance education students can follow personalised learning routes. Students are active learners in presenting. (contact data are available on the institution website) Faculty and student opportunities for interaction are frequent and planned. Specific ground-rules for discussion are established and communicated. 33. and among students. 42.2005
. An orientation program/service is available for the enrolling students. reviews etc. 45. Feedback to student assignments and questions is constructive. and simulation.
The distance learning environment is rich with activity. Courses and seminars involve application and problem solving. on every subject. 30.in less than 24 hours . Frequency and timing of participation expectations are stated and described. or synchronous session (live chat. (Online) meetings are scheduled to accommodate working students. Instructor-led discussion forums are used. 48. INST 1. 35. Contacts and communication between learners and the (teaching/ counselling/ technical) staff are encouraged. 26.06. Programs provide opportunities for timely and INST 1.to (individual) students. on campus extended campus. During the learning programme. Teaching staff/ counsellors/ tutors respect diverse ways of learning. Explanations of how students will receive their first orientation to the course are given.) are identifies and scheduled. one tutor is assigned to small groups (5-20) of students.25. 49. Alternate backup plans and responsibilities in the event of technical difficulties and technology failure are listed. organizing. 38. 40.2. 32. ideas and knowledge.
Online learning support resources are easily and accessible via distance delivery. and valid. Feedback procedures on assessments are described (posted grades in the gradebook. weighted scores. 2. 57. 59. Technical facilities
1. Student evaluation combines two or more forms of assessment (continuous evaluation. projectbased evaluation/ student portofolio). Distance education students get feedback from their colleagues. final exam.2005
. If other materials (books. they are listed with description of how they may be obtained. Teaching staff/ Tutors provide links to further resources. relevant. Technologies are appropriately used to increase proficiency at accessing. 5. reliable.students. Technologies are appropriately used to create opportunities for students to do meaningful work. Library and learning resources 50. 51. letter. 52.) Evaluation of learning is against criteria that are transparent. 58. Teaching / learning materials are relatively easy to use and free from technical hitches. software. realistic. 6. 56. The distance learning environment is rich with readings. 53. and electronic services/ resources) are required. or how critiques of assignments are returned).
C. Methods and plans for student assessment are described. grades etc. 3. 61. Technologies are appropriately used to make students feel comfortable. evaluating and communicating
Draft 23. Grading criteria are listed (percentages. points. Learning assessment is frequent. 60. Assignment submission instruction or guidelines (acceptable formats/ file/ software formats are provided and described). Distance education students have the opportunity to regularly assess their own learning. Technologies are appropriately used to accommodate and promote individualization.
Assessment 54.06. 62. 4. 55. The elearning system provides appropriate interface design. journals.
Human resources management practices include availability of technical skills development and support. Technologies are appropriately used to drill students on basic concepts to reach mastery.06. Human resources management practices include a requirement for ongoing professional development in content areas. 6. Communications facilities. 12. Staff
1. 10. Communications facilities. ensuring both the rigor of programs and the quality of instruction Evidence of program success is support through routine review and evaluation of course content and
Draft 23. 14. 2.2005
. Technologies are appropriately used to enable active engagement in the construction of knowledge. processes and practices are able to encourage contact between students and faculty. 7.
information. Staff is assisted in the transition from classroom teaching to online instruction. processes and practices are able to develop reciprocity and cooperation among students.
E. 2.7. including peer mentoring. Communications facilities. The institution's faculty assumes responsibility for and exercises oversight over distance education. 9. Technologies are appropriately used to improve students’ abilities to solve complex problems. Communications facilities.
TUT 22. 5. 8. 4. Human resources management practices include regular evaluation of competence. Appropriate and necessary personnel include instructors / teachers / professors with qualifications in the subject area. Distance education personnel are qualified for teaching in elearning environments. Technologies are appropriately used to nurture artistic expression. processes and practices are able to provide the opportunity to “hear” other students’ questions. 13. Evaluation/ monitoring
1. 3. continues through the progression of the online course. processes and practices are able to provide flexible opportunities for interactions and problem-solving.
D. 11. Instructor training and assistance.
The system provides a link to evaluation form/process for assessing student satisfaction with services provided by the distance learning programme.
Draft 23. Evidence of program success is support through routine review and evaluation of teaching staff. 6. student performance. Evidence of program success is support through routine review and evaluation of learning and student achievement.
objectives. teacher interaction with students.
5.3. Distance learning program is evaluated periodically and includes: course content. student participation. students’ interactions. evaluation tools etc. course design (academic rigor. 7. course presentation. learning materials.2005
.06. pedagogical and technical suitability). Evidence of program success is support through routine review and evaluation of students’ satisfaction. Evidence of program success is support through routine review and evaluation of policies and management practices. 4. instructional design. curriculum alignment.