Creating items with manual scoring (TAO Ignite)
By default, a test-taker receives one point per completely correct interaction. However, there are instances where a test author may need to manually configure individual items for alternative scoring methods (i.e manual scoring to be completed by a scorer or partial credit).
Scorers will be able to view which assessments need scoring from a session’s monitoring page. As soon as a test taker has submitted a test and the automatic scoring of any relevant items has been carried out, the Score button for an individual test taker will trigger the process. For more information, see Scoring assessments manually (TAO Ignite).
To ensure that test-takers’ responses are processed correctly, the relevant test items need to be configured for manual scoring when the item is authored. Follow the steps below to do this.
Prerequisite - Define the response processing
Go to the Response Properties panel within the item. Select anything except “none” from the Response processing list (a good practice is to use match correct).
1. Label the item as one which requires manual scoring.
Click on Outcome Declarations in the Response Properties panel to the right of the item. The outcome(s) for your item will appear, as in the image below. The outcomes are defined by the user. By default, SCORE and MAXSCORE are automatically added, as they are part of the QTI standard.
Clicking on the edit icon to the right of an outcome will open the values associated with it. In the example below, the first custom outcome (created by the user) listed for this item of the test (i.e. the first aspect of the response which needs to be scored) is WORDCHOICE.
The first value is Interpretation – the name of the outcome. This is the name which will be displayed to the person carrying out the manual scoring. In the example below, this is Word Choice.
Adding Outcome Declarations
In the drop-down menu under External Scored, select Human.
This value indicates the type of scoring which is needed for this particular outcome (in this case, scoring by a human).
2. Add a description of the scoring criteria.
The manual scorer needs to know what scoring system to use for the item (or outcome) in question. This is shown in the form of a rubric which contains the scoring criteria you wish to be employed, and is accessed via a link to a hosted PDF file.
In the box labeled Long Interpretation, add a link to the hosted PDF file that contains the scoring rubric. This will be accessed later by the scorer.
3. Define the score range.
Enter the range (e.g. 1-4) in the box labeled Value.
It is possible to define different ranges, as long as the following rules are respected: both the minimum and maximum should be defined, using only integers. The maximum should not be negative and should be greater than the minimum.
In order for an item’s responses to be flagged for manual scoring, there must be a valid score range under Value and human must be selected in the externalScored box.
If an item has more than one outcome (i.e. if different aspects of a response are awarded separate scores, meaning that multiple scores are needed for one item), the scoring method must be configured for each one. This is described as ‘multi-trait scoring’ on the manual scoring platform. In the example on the image above, two other aspects of this test item are visible: Understanding and Mechanics.
4. View and score items configured for manual scoring.
Once test takers complete an assessment, scorers will be able to view and score items for manual scoring from the session overview page. For more information, see Scoring assessments manually (TAO Ignite).