How does the review by criteria work?

Even3 has two evaluation methods:

1. Standard review
2. Review by criteria

Standard review

To learn more about the standard assessment, check out this article: How does the standard review work?

Review by criteria

For scientific and academic events, evaluation by criteria has become a fundamental part of scientific work submissions. In this article, you'll learn how to use this type of evaluation for your event.

First of all, you need to know that the review through Even3 is double-blind. In other words, the authors don't know who the reviewers are and the reviewers don't know who the authors are.

Learn more! For a better understanding, please know the importance of double-blind review.

How evaluation by criteria works

In a criteria-based evaluation, you can define criteria for the evaluators to base their evaluation on. 

For each criterion, you must define the weight it will have in the rating, and you will also define the minimum and maximum ratings that the reviewer can give for each of the criteria.

For example, you could define that the criteria for evaluating works at your event are:

• Cohesion - Weight 3
• Methodology - Weight 3
• Orthography - Weight 2

And the ratings are:

• Minimum rating: 1
• Maximum rating: 5

In other words, the reviewer will have to evaluate the work by giving it a score from 1 to 5 for each of the criteria established. And the final score will be the weighted average of these scores, based on the weights of each criterion.

Let's take an example, a reviewer gave the following scores:

Reviewer 1:

• Cohesion: 4
• Methodology: 3
• Orthography: 5

The system will calculate the weighted average automatically.

1. Each grade is multiplied by its weight (4x3=12, 3x3=9, 5x2=10).
2. Then the results are added up (12+9+10=31)
3. It is then divided by the sum of the weights (sum of the weights: 3+3+2=8).

Final result: 31/8= 3,87

Works are approved by the organizer, who will set a cut-off score as explained at the end of this article.

How to set up the evaluation by criteria

Activate

Access the Submissions menu. Click on Review. Go to the Settings tab. In Type of Review, define the Review by criteria option. Set minimum and maximum scores.  Click on Save review type.

Set up

After activating the review by criteria, on the Submissions > Review menu, there will be the Criteria tab. On this tab you will be able to add the criteria and set the weights for each of them. 

Go to the Criteria tab. Click on the Add Criterion button.

A window will open where you can define each reviewing criterion, you can add as many criteria as you like.

Fill out: Title of the criterion  Weight Instruction (will be available for the reviewers at the time of evaluation) Click on Save Criterion.

Done! The review by criteria is validated in your event!

Approval and rejection of works

The approval of the works on the review by criteria depends on the event organization. 

It works as such:

Accessing the Review menu, you will click on the finish reviews button.

In order to all works being evaluated only once, all you need to do is set the modality and subject area as all, on the selection menu.

Then, you will set the type of ruling, if approved or rejected.

As for the ruling and value, those will be where you will set the minimum grade for approval, being greater or less than the score entered on the value field.

For example, if the ruling is set as approved if the score is greater than 5, all works with a 5 or greater score will be approved in the event. Those with 4,9 and below will show as rejected on the Results menu.

Now what? Now that you already know how the review by criteria works, learn how do automatic and manual distributions work.