Reporting

ReLiS reports basic statistics for each step of the systematic review process. The data can be exported for further analysis with dedicated statistical tools, such as Excel, SPSS, or R. All reports are accessible at any point in time, even when the project is not yet complete.

Reporting the screening

It is straightforward to track the progress of each step with progress bars and gauges showing the percentage of completion for each screening phase, validation, QA, and data extraction.

Furthermore, you can consult various statistics of each screening phase reporting the articles included, excluded, and in conflict. ReLiS also displays the inter-rater agreement Kappa. Kappa is relevant only if there are more than one reviewer assigned per article. It is a value between 0 and 1, where:

  • 0 means there is no agreement
  • 0.1 - 0.2 means there is a slight agreement
  • 0.21 - 0.4 means there is a fair agreement
  • 0.43 - 0.6 means there is a moderate agreement
  • 0.61 - 0.8 means there is a substantial agreement
  • 0.8 - 0.99 means there is a near perfect agreement
  • 1 means there is perfect agreement

Kappa is update continuously as articles are screened during the phase. Therefore, after all conflicts are resolved Kappa will be 1. If your protocol requires to have intermediate checks on the degree of agreement, we recommend you take snapshots of the value at the appropriate times. For example, this can happen if you are starting with a sample of articles to make sure all reviewers are on the same page and then you continue with the remaining articles.

Reporting the QA

When articles are processed for quality assurance, the results display the score of each article. You can then decide to exclude the articles below the predefined threshold or exclude articles manually.

Reporting the classification

While the classification/data extraction is ongoing, you can visualize the results in the form of a table, charts or export to a file. You can export all the classification data in a CSV file. You can export all articles, only those included, or those excluded at different steps to BibTeX or CSV.

In the configuration file, you can specify the charts to report in ReLiS. You can visualize data along one or two dimensions. Here is the sample code you can write at the end of the configuration file: 

REPORT
Simple chart_name "Name of chart" on category_name charts(chart_type1, chart_type2)
Compare chart_name "Name of chart" on category1_name with category2_name charts(chart_type1, chart_type2)

One-dimension chart

Simple indicates that we want to visualize along one category of the extraction form. The chart_name is a unique variable to identify the chart. You may optionally give it a more user-friendly name between quotation marks. The category_name is the category along which we will report the frequencies. The chart_type can be line (for scattered plots with a continuous line), bar (for a bar chart), or pie (for a pie chart). You can specify one or more chart type. They will all be displayed.

For example, lets assume we created the Chocoholics project (see the Data extraction help page). Now we add the following lines in the report section: 

Simple year "Year published" on year charts(line)
Simple country on country charts(pie, bar)
Here we will display three charts. The first one is a graph of the number of articles per year. The second one is a pie chart showing the percentage of articles per country. The third one is showing the same information in the form of a bar chart.

Two-dimension chart

One-dimension charts assume that the y-axis is the number of papers for each category value. Two-dimension charts allow you to customize the x and y-axis to any dimension collected in a category. Compare indicates that we want to visualize along two categories of the extraction form. The category1_name is the category that will be used for the y-axis. The category2_name is the category that will be used for the x-axis.

For example, lets go back to the Chocoholics project. The following line will display a graph showing the number countries per year. 

Compare country_year "Country per year" on country with year charts(line)

Querying the database

The automated analytics in ReLiS are limited. Nevertheless, if you wish to perform more analytics on all aspects of your systematic review (process, data extracted), you can directly query the database of your project. Note that you need to have the appropriate user role to have access to this option.

To query the database, go to the home page of your project and click on the Query Database menu. You can write any arbitrary query using MySQL syntax. Note that you should avoid modifying the data directly in the tables (INSERT, UPDATE, DELETE). You should certainly not modify the database or the tables (DROP, ALTER) as this may corrupt your project. Read-only queries using SELECT statements are what you should be considering.

You can find the list of all the tables and views in the database of your project by executing the query:

SHOW TABLES

Examples of queries

You can get the number of papers per year that have been included after screening as follows:

SELECT year, COUNT(*)
FROM paper
WHERE screening_status='Included'
GROUP BY year

You can get the list of papers that were excluded for a specific exclusion criteria (here EC1 is an example of the name of an exclusion criterion) as follows:

SELECT paper_id, bibtexKey
FROM screening_paper
INNER JOIN ref_exclusioncrieria ON ref_exclusioncrieria.ref_id = screening_paper.exclusion_criteria
INNER JOIN paper ON paper.id = screening_paper.paper_id
WHERE ref_value='EC1' AND ref_active=1