You are on page 1of 12

Examination System

Introduction to Online Examinations

Online Examinations are a software solution, which allows a particular company or institute to arrange, conduct and manage examinations via an online environment. This can be done through the Internet, Intranet and/or Local Area Network environments. Candidates answer his/her examination paper on the computer and submit their answers. The Examination Software usually evaluates the submitted answers and the results are then immediately available upon completion of the examination. Online examinations are ideal for open-book type examinations in an uncontrolled environment, although closed book examinations can be conducted in a controlled environment. An Online Examination System usually consists of several modules to enable it to function within an examination environment. Examples of such modules are: Registration Module Booking Module Candidate Information Module Subject Management Module Questionnaire Generation Module Student Verification Module Statistical Module Remote Marking Module

There are advantages and disadvantages in online examinations. The main advantage is that candidates can write their respective examinations remotely and the marking process can be fully automated. Another advantage is that candidates can write their examinations at any time at a location of their choice. Disadvantages are that the type of question might be limited, depending on the marking process thats used by the system as well as possible security risks. However methodologies do exist that addresses these disadvantages. By making full use of an Online Examination System organizations can uplift industry standards. A lot of information is available in an electronic examination system, which enables one to see industry trends, shortcomings and valuable feedback can be provided to training institutes, instructors and students.

1 / 12

Examination System

Various types of examinations exist. Below are some examples: - To determine deficiency gaps for training purposes - To determine the basic skill/knowledge level of a candidate - As preparation for major exams (Practice/Mock Exams) - As part of a qualification process to determine whether a candidate has the necessary knowledge/skills as stipulated by curriculum - As aptitude tests where no answer is deemed right or wrong but rather creates a profile of a candidate

Why are the distinction made between various examination types? If one is conducting an examination for qualification purposes the Examination System must provide the necessary tools to ensure that environment security is not compromised. Is it wise to conduct qualification examinations in an uncontrolled environment and are generated questionnaires generated in a balanced and fair way ensuring all candidates are tested fairly and balanced. If not the results can and will probably be challenged in a court of law at some point. It is therefore very important to ensure that the right kind of examination system is used for a particular purpose as few systems cater for all type of environments/examinations. Unfortunately every system that allows users to write questionnaires online call themselves an examinations system while very few are actually structured and developed for a serious examination environment.

XAM Overview
The XAM examination system composes of four integrated main areas. These areas can be roughly broken down into the following:

Base Station Module

Subject Structure

The subject structure consists of a basic framework/summary of all the subjects. Questions can then be assigned at a later stage against this subject structure.

Question Area
This is the area where all questions are entered into the system. Each question is assigned

2 / 12

Examination System

against a subject framework. The physical question and answer is also entered into the system. Each question is then assigned a difficulty level from 1 to 5, where 5 is the most difficult. Each question also has a differential value, which indicates to the system administrator how efficient this question has been in past examinations. This is a calculated value that tells us if the question actually targets what it is supposed to target. A taxonomy can also be entered against each question.

Exam Structure Area

The exam structure consists of a basic framework/summary of all the examinations. This forms the basis of all examinations. This tells the system of all the types of examinations we are going to have and also which subjects each examination consists of. We also have to specify to the system what the time limit and pass mark for subject should be. The intervals can also be set to prohibit individuals to enter for another examination (or re-write) before a certain time period has elapsed. Criteria for credits can also be specified to enable the system to know when student should carry credits or not after a sitting.

Recipe Area
This is where we tell the system how a questionnaire should be generated by the system. This will ensure that a questionnaire will be well balanced, the way it should be, when a questionnaire is generated by the system.

Questionnaire Area
This is the area where the actual question will be generated by using a particular recipe. A pass mark and time limit will also be automatically set for each questionnaire by calculating each individual questions mark and time limit. The system will also ensure that questions is rotated within each questionnaire by going through the past history and statistical area of the system. This ensures that the newest and unused questions will be given the highest priority. However this feature functions within the recipe criteria.

Student Area
This is the area where all student information and history is kept. The system is able to indicate to the administrator exactly what questionnaire each student, as well as the answers the student used for each question. Students are also entered for any particular examination from this area. Certificates and Results can also be displayed or printed from within this area. Additional information can be assigned to students telling the system from which flight school this student attended. By assigning this value the system will be able to tell the administrator exactly where each flight schools problem areas are. This can then be used to report back to flight schools

3 / 12

Examination System

enabling them to lift their standard in certain problem areas.

Statistical Area
This is the area where all kinds of statistics can be performed. Examples of statistical reports: - How many students entered/passed/failed any particular examination - How many students entered/passed/failed any particular subject - List all badly structured questions within a particular subject (Differential value) - Adjust difficulty levels of questions accordingly (After questions had been used a number of times in questionnaires the system are able to indicate and adjust the difficulty level) - Flight school reports indicating to them exactly how their student fared in any particular subject and area. - Reports indicating shortcoming within aviation by performing certain formulas and calculations. This will once again enable RAASA to uplift the standard of training within the private sector.

Online Server Module

The online server is used to control the student stations within a controlled examination environment. The online server integrates tightly with the base station. By doing this the online server knows exactly how each examination is structured and what it is allowed to do and not. The online server also updates the main database automatically. The server consists roughly out of the following modules:

This allows the users to book students for certain examinations. The system will only allow users to book students if they are entered for a particular examination within the base station. This prevents the user from accidentally entering student into incorrect examinations etc.

This area allows examination sessions to be loaded onto student stations. This is where the actual loading of each student station occurs. Only booking sessions that is scheduled will be allowed to be loaded. This prevents the user from accidentally loading the wrong session onto the student stations within the examination room.

4 / 12

Examination System

This tells the user exactly what is happening within the examination room without the invigilator having to walk around. It even tells the invigilator what time is left on each student station and what the progress is that the student is making by indicating number of questions answered and remaining etc. When a student completes an examination the system allows the user to automatically upload all data to the main database. Printing of certificates and result lists is also done automatically as each students results is uploaded to the main database.

Student Station Module

This is where the student actually does his examination on. The student station allows student to answer and move between questions. The student is also able to tag questions, which he/she would like to review at a later stage. The station will automatically close the examination if the students time expires. Each questionnaire is marked on the student station and the results displayed to the student. The student station communicates the whole time with the online server informing the online server of what is happening on the station itself. A student is also able to alert the invigilator by the press of a button if they require assistance. This causes the online server to display an alert message to the invigilator. Student stations can run on both a controlled or uncontrolled examination environment.

Written Examinations Module

This module is used for examinations where students write examinations via a pre-printed answer sheet, which is then marked back into the system via a scanner or manual marking facilities to mark the answer sheets of students. The system is also able to indicate certain problem questions. If there were a problem with any particular question the system will be able to credit each student with that particular question if so required. Check lists and additional reports is also available to ensure that any potential problem will be indicated to the user. Results are then exported into PDF files which can be published on the CAA website.

XAM runs on the Windows operating system and makes use of a SQL database to store data. TCP and UDP are used where needed in data transfer and all data transmissions between server and clients are encrypted. The system makes use of a Local Area network, Wide Area network as well as the Internet Infrastructure. The system supports both uncontrolled as well as controlled examination environments. For student stations a normal lightweight PC can be used.

5 / 12

Examination System

Implementing an Online Examination System

When implementing an Online Examination system, requirements may vary between various organizations. However the basic principles should still apply no matter what kind of examination environment are used for online examinations.

Planning is Important
Sadly when most people implement an examination system the most important steps are skipped, which forms the foundation of any examination system. Therefore planning is very important to ensure that one is doing the right things the right way. This will ensure that the system is implemented upon solid foundations and that one can maximize the systems potential.

Initial Steps before conducting examinations

The following is the steps that should take place before any system is ready for examinations. Defining Subjects Defining Subject Structures Defining a Taxonomy Creating Questions Linking Questions to Subject Structures Define Examinations Define Questionnaires Define Recipes for Questionnaire Generation

Usually the subjects can easily be defined. The subject structure is probably one of the most

6 / 12

Examination System

important foundations of any examination system. With a solid Subject Structure one can identify deficiency gaps in the industry as well as give more thorough deficiency reports to candidates after completion of an examination. The best to illustrate a Subject Structure is through an example. Lets assume we have a subject called Aircraft Technical. We now need a structure for Aircraft technical and define the following basic example structure: Principles of Flight Derivation of Lift Drag Stability Stalling Lift Augmentation Flying Controls Performance Aircraft Elements Airframe and Systems Flight Controls etc

That gives us a very basic structure, however it limits the system as well as possible feedback. Lets break down the structure further. For this example we will focus on Airframe and Systems: Principles of Flight Derivation of Lift Drag Stability Stalling Lift Augmentation Flying Controls Performance Aircraft Elements Airframe and Systems Fuselage Cockpit and Cabin Windows Aerofoils Control Surfaces Landing Gear Types Construction Locking Devices

7 / 12

Examination System

- Emergency Extention Systems - Accidental Retraction prevention - Position, Movement lights and indicators Nose Wheel Stearing Wheels and Tyres Braking System etc

- Flight Controls - etc

In the above example Airframe and Systems were broken down further. Landing gear in turn was broken down even further in the subject structure to give us an example. The depth of a Subject structure should not be limited but care must be taken to ensure that a good balanced structure is defined. What is the advantage of a good, solid subject structure? Imagine the feedback that can be given to flight schools and instructors. If students that received training from a specific instructor are grouped together the system will be able to generate a deficiency report for that Instructor, indicating the problem areas his/her student had. For example one could give feedback and say We have found that your students tend to struggle with Locking Devices under Landing Gear (Airframe and Systems). That would be helpful for the instructor, flight school etc and aid as one area where the industry standard of training can be uplifted where needed. Now, if the subject structure was poorly defined proper useful feedback cant be given back to the industry. It is important to define a Taxonomy that will be used by the System before one start to create questions. Different taxonomies exist and a very basic taxonomy can be: - Recall - Application - Insight

Recall is when a student must rely on recalling his knowledge in order to answer a particular question. Application goes a bit further, not only does the student need to recall his/her knowledge but also they must apply that knowledge in order to solve a particular question. Insight is where the candidate must use his/her insight into a subject area in order to answer a particular question. Keep in mind that although making use of a Taxonomy is optional its highly advisable to make use of it.

8 / 12

Examination System

By making use of taxonomy in combination with a solid subject structure one can generate balanced questionnaires where one is assured a subject will be not only covered properly in the various topics but also balanced on the different type of questions that are asked within a given questionnaire. This also means another level of feedback. Take the example we gave earlier on the feedback. One can now say We have found that your students tend to struggle applying their knowledge with Locking Devices under Landing Gear (Airframe and Systems) This might mean the problem arent that the topic was insufficiently covered as candidates Recalled the applicable information but struggled to apply it. This is valuable information for any instructor and another level towards uplifting industry standards. When all those are in place one can create questions for the various subjects and set the taxonomy as well as link it to the necessary subject structure. A default difficulty level can be assigned to questions and it can be automatically adjusted by the system as time passes to ensure that a questions difficulty is always maintained. Various tools should be available for subject matter experts to assist them in identifying problem questions as well as ensure that the questions are targeting what they should. Each question should be allocated a mark as well as an estimated time it should take a candidate to complete the specific question. Defining Examinations and Questionnaires are very important, as this will determine the business rules within a system regarding pass/fail and what subjects needs passing before a candidate passes all examinations and a certificate should be issued. Defining a recipe is where most of the above items get together. The subject matter expert should indicate from which subjects it should generate questionnaires; how it should be balanced content wise as well ensure that it will be a balanced questionnaire regarding content, taxonomy, difficulty, time etc. Once all these are set up properly the system is ready to be implemented for online examinations usage.

XAM Features
SQL Server database are supported Interbase database are supported Integration into other systems ie. Aircrew Licencing Full user security for system features is available Management reporting capabilities Capability to use and link data from various databases together

Capable of handling multiple subjects Can be structured into multiple levels of detail Subject structures can be viewed in tree form (Tree View) Subject structures can be locked once completed Subject structures can be moved to different locations

9 / 12

Examination System

- Default structure levels can be attached to the lowest levels - Subjects can be allocated to applicable groups (I.E Countries) - Owners/Users can be allocated to own a subject

- Questions can be entered only once for multiple type of exams - Multiple weighting and times can be set against a single question - Difficulty level can be allocated towards each question - Difficulty level can be automatically adjusted by System - Taxonomy can be allocated towards each question - Discrimination values can be automatically calculated for questions - Discrimination values can be automatically calculated for each distractor - Indicate percentage of students who chose a particular response on a question - Indicate which question response was chosen by students who scored in the top 20% for that given subject. - Indicate the number of times the question was not answered. - Indicate the average time students took to answer a question during online examinations. - Questions can contain Graphics/Images for written and online exams - Video, Sound based questions are supported for online exams - Answer hotspots can be defined on questions containing images - Reference to subject material can be added to questions for part of feedback - Information can be entered against each question that can be used for feedback in examinations where students didnt answer that question correctly. - Questions can be allocated for only specific type exams (I.E Online and/or Paper-based) - User defined question layouts are supported - Questions can be disabled for use until further notice

- Multiple exams can be defined - Recipes can be attached for automatic question generation - Balanced questionnaires can be generated automatically - Negative marking can be implemented - Exams can consist of multiple questionnaires - Exam papers can be generated with the click of a button - Questionnaire times can be automatically calculated - Examiners, if needed, can adjust questionnaire times - System supports written and/or online examinations - Questions can be marked via scanner and/or digitizer - System can differentiate between students from different groups and use correct questionnaires for those groups (I.E Use applicable Airlaw for students from different countries - System keeps tracks of student attempts/credits etc

10 / 12

Examination System

- Exam statistics can be generated - Result lists can be printed - Student Certificates can be printed upon completion of examinations

Personal & contact details can be captured User defined fields can be added Complete student history is available Complete examination history is available for each student Notes & reminders can be e-mailed to students via the system Statistics/Feedback can be generated for groups of students Results are automatically updated against a students history Problem areas against groups of students can be identified Students can register remotely via Internet Students can pay remotely for their exams via credit card Student are not allowed to write an examination unless payment are received Student can submit scanned documentation upon registration Examinars can verify and approve submitted documentation upon student registration

Online Environment
Student bookings can be made and scheduled via sessions Fingerprints, Photos of students can be captured upon registration System can verify fingerprints during exam sittings automatically System can verify fingerprints during exam sittings by request System can verify student station environment during exams System can verify student station environment before exams Abovementioned video and fingerprint verification can be done remotely The online server can generate exam papers automatically Student stations are automatically loaded once a session is loaded Student stations can be monitored via the online server Online server is capable of standing on a remote location Results and certificates can be printed once a student finishes System can pinpoint problem areas for each student Student can resume where they left off after a power failure Student can continue even if the Internet connection goes down Multiple student stations are supported

11 / 12

Examination System

12 / 12