ReLiS

Screening

The screening step follows the protocol you have specified in the configuration file. You may have multiple screening phases, a validation phase, and a quality assurance phase. The video on how to create a project illustrates how to configure these steps.

The video above demonstrates how screening is performed in ReLiS.

Adding a screening process

If your systematic review includes a screening process (which is in most cases), the format is:

PROJECT short_name "Long name"

SCREENING
Reviews n
Conflict on conflict_type resolved_by conlfict_resolution
Criteria = ["EC1", "EC2"] // optional
Sources = ["Source 1", "Source 2"] // optional
Strategies = ["Database search", "Manual entry", "Snowballing"] // optional
Validation x% validation_type // optional
Phases "Title" "Description"  Fields(field1, field2) // can be repeated multiple times

CLASSIFICATION
...

Reviews

n specifies the number of reviewers to assign to a single article to decide on its inclusion/exclusion. For example, 2 means every article will be screened by two reviewers.

conflict_type

In case multiple reviewers are assigned to each article (so n > 1 on the previous line), there may be conflicts in their decision when screening the same article. conflict_type can be one of the following values. Decision means that if one reviewer included the article, but another one excluded it. Criteria is like Decision, but also if both excluded it but not form the same exclusion criteria.

resolved_by

This indicates how conflicts between reviews (see above) shall be resolved. resolved_by can be one of the following values. Unanimity means that the article will be marked as in conflict until all reviewers assigned to the article agree on the decision of inclusion/exclusion or exclusion criteria of the article. Majority means that the decision of the article is based on the decision of the majority of the reviewers.

Criteria

Enumerates the exclusion criteria to choose from when excluding an article during the screening phase. This is optional and can be modified later through the menu. In the example, "EC1" is one of the exclusion criteria. You can write any text and enclose it between quotation marks.

Sources

Enumerates the data sources of where the articles are coming from. This is set when importing articles. For example, the name of a digital library. This is optional and can be modified later through the menu. In the example, "Source 1" is one of the data source. You can write any data source name (e.g., "Web of Science") and enclose it between quotation marks.

Strategies

Enumerates the search strategies of how the articles were collected. This is set when importing articles. This is optional and can be modified later through the menu. In the example, "Database search" is one of the search strategies. You can write any text and enclose it between quotation marks.

Validation

The validation phase is optional. You can decide to have a sanity check phase where x percent of the excluded articles will be assigned to a validator. The selection of the articles is decided at random following a uniform distribution among all excluded articles. Note that only articles resolved and marked as excluded are considered in the validation. Those still marked in conflict are not considered.

validation_type

In case the validator decides to include an excluded article, we need to decide how this will be handled. validation_type can be one of the following values. Info means it will only be flagged in the statistics but does not change the status (inclusion/exclusion) of the article. Normal means his decision counts like any other reviewers. In this case, the resolved_by option in Conflict applies. For example, suppose Unanimity was selected and the two reviewers had decided to exclude the article. If the validator decides to include the article, then the article is marked in conflict. Veto means that the decision of the validator overrides those of the reviewers.

Phases

There can be as many screening phases as desired, but at least one is mandatory. Giving a title and short description is useful when there is more than one phase. Otherwise, you may want to simply call it "Screening". To add more phases, you can add another line starting with Phases. In case of multiple screening phases, note that only articles marked as included in a previous phase are available for screening in the following phase. The Field lists the meta-information that will displayed when screening each article. The values can be any combination of:

Title to show the title of the article
Abstract to show the abstract of the article (if available)
Preview to show all the meta-information of the article, including authors, year of publication, venue of publication, etc.
Bibtex to show all the meta-information in BibTeX format
Link to show the link to the actual article online and have access to the full text (if available)

Once the articles have been screened, they become available for classification and data extraction.

Adding a quality assurance step

If desired, you may add a quality assurance (QA) step after all the screening phases and before the the data extraction step. Note that only articles marked as included in the final screening phase are considered for quality assurance. In this step, reviewers can fill in a checklist for each article. This section of the configuration file describes the questions and answers of the checklist. During the QA step, reviewers are expected to have access to the full text of the articles. The format of the QA configuration is:

PROJECT
...
SCREENING
...
QA // optional section
Questions = ["Quality question 1", "Quality question 2"]
Response = ["Answer1": w1, "Answer2": w2]
Min_score threshold

Questions: Lists all the questions in the order as they appear
Response: The possible answers for all questions. They are the same for all questions. Typically, you would have Yes, No, Partially. Each answer is assigned a weight like: ["Yes":3, "Partially":1, "No":0].
threshold: Once a checklist is filled in for an article, ReLiS calculates the score of this article by summing the weights of the answers selected. The threshold defines the minimum score that each article should have to be considered for data extraction. Otherwise, the article will be excluded.

Note that you can change the configuration of the screening and QA at any point in time through the menu. However, if you regenerate the configuration file, those changes would be lost unless you have manually updated the configuration file.