The screening step follows the protocol you have specified in the configuration file. You may have multiple screening phases, a validation phase, and a quality assurance phase. The video on how to create a project illustrates how to configure these steps.
The video above demonstrates how screening is performed in ReLiS.
Adding a screening process
If your systematic review includes a screening process (which is in most cases), the format is:
PROJECT short_name "Long name" SCREENING Reviews n Conflict on conflict_type resolved_by conlfict_resolution Criteria = ["EC1", "EC2"] // optional Sources = ["Source 1", "Source 2"] // optional Strategies = ["Database search", "Manual entry", "Snowballing"] // optional Validation x% validation_type // optional Phases "Title" "Description" Fields(field1, field2) // can be repeated multiple times CLASSIFICATION ...
Reviewsnspecifies the number of reviewers to assign to a single article to decide on its inclusion/exclusion. For example, 2 means every article will be screened by two reviewers.conflict_type- In case multiple reviewers are assigned to each article (so
n > 1on the previous line), there may be conflicts in their decision when screening the same article.conflict_typecan be one of the following values.Decisionmeans that if one reviewer included the article, but another one excluded it.Criteriais likeDecision, but also if both excluded it but not form the same exclusion criteria. resolved_by- This indicates how conflicts between reviews (see above) shall be resolved.
resolved_bycan be one of the following values.Unanimitymeans that the article will be marked as in conflict until all reviewers assigned to the article agree on the decision of inclusion/exclusion or exclusion criteria of the article.Majoritymeans that the decision of the article is based on the decision of the majority of the reviewers. Criteria- Enumerates the exclusion criteria to choose from when excluding an article during the screening phase. This is optional and can be modified later through the menu. In the example, "EC1" is one of the exclusion criteria. You can write any text and enclose it between quotation marks.
Sources- Enumerates the data sources of where the articles are coming from. This is set when importing articles. For example, the name of a digital library. This is optional and can be modified later through the menu. In the example, "Source 1" is one of the data source. You can write any data source name (e.g., "Web of Science") and enclose it between quotation marks.
Strategies- Enumerates the search strategies of how the articles were collected. This is set when importing articles. This is optional and can be modified later through the menu. In the example, "Database search" is one of the search strategies. You can write any text and enclose it between quotation marks.
Validation- The validation phase is optional. You can decide to have a sanity check phase where
xpercent of the excluded articles will be assigned to a validator. The selection of the articles is decided at random following a uniform distribution among all excluded articles. Note that only articles resolved and marked as excluded are considered in the validation. Those still marked in conflict are not considered. validation_type- In case the validator decides to include an excluded article, we need to decide how this will be handled.
validation_typecan be one of the following values.Infomeans it will only be flagged in the statistics but does not change the status (inclusion/exclusion) of the article.Normalmeans his decision counts like any other reviewers. In this case, theresolved_byoption inConflictapplies. For example, supposeUnanimitywas selected and the two reviewers had decided to exclude the article. If the validator decides to include the article, then the article is marked in conflict.Vetomeans that the decision of the validator overrides those of the reviewers. Phases- There can be as many screening phases as desired, but at least one is mandatory. Giving a title and short description is useful when there is more than one phase. Otherwise, you may want to simply call it "Screening". To add
more phases, you can add another line starting with
Phases. In case of multiple screening phases, note that only articles marked as included in a previous phase are available for screening in the following phase. TheFieldlists the meta-information that will displayed when screening each article. The values can be any combination of:Titleto show the title of the articleAbstractto show the abstract of the article (if available)Previewto show all the meta-information of the article, including authors, year of publication, venue of publication, etc.Bibtexto show all the meta-information in BibTeX formatLinkto show the link to the actual article online and have access to the full text (if available)
Once the articles have been screened, they become available for classification and data extraction.
Adding a quality assurance step
If desired, you may add a quality assurance (QA) step after all the screening phases and before the the data extraction step. Note that only articles marked as included in the final screening phase are considered for quality assurance. In this step, reviewers can fill in a checklist for each article. This section of the configuration file describes the questions and answers of the checklist. During the QA step, reviewers are expected to have access to the full text of the articles. The format of the QA configuration is:
PROJECT
...
SCREENING
...
QA // optional section
Questions = ["Quality question 1", "Quality question 2"]
Response = ["Answer1": w1, "Answer2": w2]
Min_score threshold
Questions- Lists all the questions in the order as they appear
Response- The possible answers for all questions. They are the same for all questions. Typically, you would have
Yes, No, Partially. Each answer is assigned a weight like:["Yes":3, "Partially":1, "No":0]. threshold- Once a checklist is filled in for an article, ReLiS calculates the score of this article by summing the weights of the answers selected. The
thresholddefines the minimum score that each article should have to be considered for data extraction. Otherwise, the article will be excluded.
Note that you can change the configuration of the screening and QA at any point in time through the menu. However, if you regenerate the configuration file, those changes would be lost unless you have manually updated the configuration file.