By default, a test-taker receives one point per completely correct interaction. However, there are instances where a test author may need to manually configure individual items for alternative scoring methods (i.e manual scoring to be completed by a scorer or partial credit).
To ensure that test-takers’ responses are processed correctly, the relevant test items need to be configured for manual scoring when the item is authored. Follow the steps below to do this.
1. Prerequisite - Define the response processing
Go to the Response Properties panel within the item. Select an option from the Response processing list (a good practice is to use match correct).
2. Label the item as one which requires manual scoring
Locate the Outcome Declarations in the Response Properties panel to the right of the item. Additional outcomes added by the item author determine what the scorer will see when grading the item.
Important: By default, SCORE and MAXSCORE are automatically included, as they are part of the QTI standard. The default configuration of these outcome variables should generally not be modified - especially the External Scored field. Changing this field from its default value (None) may negatively affect the overall score calculation for both the item and the test. Improvements to enhance the stability and consistency of outcome variable editing are currently in progress and will be available in a future release.
-
Select Add an outcome.
-
Click on the edit icon to the right of the new outcome.
-
Add a label for the outcome. This should be one word. For example, add UNDERSTANDING or WORDCHOICE.
-
The second value is Interpretation – the name of the outcome. This is the name which will be displayed to the person carrying out the manual scoring. In the example below, this is Word Choice.
-
In the External Scored drop-down menu, select Human. This value indicates the type of scoring which is needed for this particular outcome (in this case, scoring by a human). For more information on the other menu options, see Additional scoring options.
If an item has more than one outcome (i.e. if different aspects of a response are awarded separate scores, meaning that multiple scores are needed for one item), the scoring method must be configured for each one.
3. Add a description of the scoring criteria (optional)
Sometimes a manual scorer needs to know what scoring system to use for the item (or outcome) in question. This is shown in the form of a rubric which contains the scoring criteria you wish to be employed, and is accessed via a link to a hosted PDF file.
In the box labeled Long Interpretation, add a link to the hosted PDF file that contains the scoring rubric. This will be accessed later by the scorer.
If you are using a PDF from Google Drive, set the sharing permissions to “Anyone with the link can view”. Once you have copied and pasted the link to the PDF in the Long Interpretation box, remove the URL’s appendix after the last forward slash. Append the word preview in all lowercase letters after the slash. See the following example:
https://drive.google.com/file/d/[your_file]/preview
For a demonstration of this feature, view the following video:
4. Define the score range
Enter the range (e.g. 1-4) in the box labeled Value.
The following rules must be respected:
-
both the minimum and maximum should be defined, using only integers
-
the maximum should not be negative and should be greater than the minimum.
In order for an item’s responses to be flagged for manual scoring, there must be a valid score range under Value and human must be selected in the externalScored box.
5. View and score items configured for manual scoring.
You can validate the scoring rules for an item by using item preview. For more information, see Previewing an item.
Once test takers complete an assessment, scorers will be able to view and score items for manual scoring from the session overview page. For more information, see Scoring assessments manually (TAO Ignite).