How to do a Systematic Review of Literature in Computer Science area
This article presents the steps to how a systematic literature review (a formal study that is used to map a specific area of knowledge) should be done in Computer Science area.
What is SRL and why is it important to know about it?
A Systematic Review of Literature (SRL) is a methodology study that aims to synthesize formally through well-defined steps, the main studies available in a specific area [3].
According to [1] the systematics reviews must be comprehensive and unbiased in its preparation. So, the researcher can repeat the steps taken and achieve the same results.
In order to do a good SRL is necessary a well-defined question beyond criteria for identifying, selecting, evaluating and synthesizing the study results.
For a good basement for future works this kind of research is very important to know some information like, the state of art in the researched area, how authors have been creating this kind of application, what system and what tools are used.
How to do a Systematic Review
The first step to be defined is the question that will guide the entire literature search, which usually is established similarity to something like this: “What are the main articles involved in the context of (some specific topic)?”. Table 1 is intended to summarize how the main elements were defined for a review, including the main issue, among other information such as objective search criteria, inclusion, and exclusion criteria. The next step in SRL is to define the bases for literature research. In Table 2 there are a example of selected bases for a study as suggested by [2].
Table 1. Systematic literature review protocol
| Main Question | “What are the main articles involved in the context of (some specific topic)?” |
| Goal | Search the literature for studies that are in the context of some specific topic during a determined range of time. |
| Inclusion Criteria | Articles that contain the determined key terms according to the topic chosen. |
| Exclusion Criteria | Criterias that should eliminate articles. |
Table 2. Example of Databases selected for a SRL
| Name | URL |
| ACM | https://dl.acm.org/ |
| IEEE Xplore | https://ieeexplore.ieee.org/Xplore/home.jsp |
| Science Direct | https://www.sciencedirect.com/ |
| Springer Link | https://link.springer.com/ |
Since the protocols of the research and the databases have already been established, the next step is necessary to define the key terms that will be used in the searches.
According to [2] the main point is to form a string that represents the key terms searched. For instance, if it is wanted to search for the term multiagent, the string should not contain this only variation of the term but should also include other variations that appear in the literature, such as multi-agent, multiagent system, among others. The same method must be used for the other searched terms, so the terms are included and their synonyms. The search result is desired to find articles that contain any of the terms of that synonym group and equally for the two other term groups. Boolean logical operators like OR and AND are used to achieve this. Finally, it is possible to observe an example of the compilation of themes such as multiagent systems, machine learning, and recommendation system in Table 3.
Table 3. Key terms used to define the search string
| (“multiagent system” OR “multiagent systems” OR “multiagents system” OR “multiagents systems” OR “multi-agent system” OR “multi-agent systems” OR “multi-agents system” OR “multi-agents systems” OR “agent-based system” OR “agent-based systems” OR “agent system” OR “agent systems” OR “agents system” OR “agents systems” OR “agent based model” OR “agent based models” OR “agent-based model” OR “agent-based models”) | |
| AND | |
| (“machine learning” OR “development of thinking computer systems” OR “expert system” OR “expert systems” OR “automatic learning” OR “machine intelligence”) | |
| AND | |
| (“decision support system” OR “decision support systems” OR “recommendation system” OR “recommendation systems”) | |
After collecting articles found in the four databases cited in Table 2, it is suggested the use of auxiliary softwares to organize, administ and edit the articles found. JabRef is a good software that can be used to remove duplicate articles, considering that sometimes an article is found in more than one database [5]. Mendeley is recommended for reading and annotating in the articles, allowing the author to work synchronously with others and integrate for several devices [4].
Evaluation Steps
The evaluation steps of the articles found should be applied as indicated in the study of [2]. The method consists of three steps: title and abstract evaluation, introduction evaluation and full-text evaluation, applied in exactly that order. In title and abstract evaluation, the authors read the title and abstract to evaluate if each study is in the context of SRL’s main question, considering the inclusion and exclusion criteria defined in Table 1. So, studies outside the theme are eliminated from the research.
In the introduction evaluation, the authors read introductions from every article approved in the stage before. The evaluation occurs by analyzing if the introduction is related to SRL’s main question, besides considering the inclusion and exclusion criteria. The studies approved in this stage still need to pass by a final evaluation. The last evaluation stage is the reading of the full text of the articles that remain under evaluation. In full-text evaluation, the methodology suggested to use is very specific for computing area, where for each article read are analyzed and answered ten questions, whose answers can be yes (1 point), partially (1/2 point) or not (0 point). Composing a final grade for each article evaluated. The cutoff note is defined by the researchers. Below are listed the 10 specified questions defined by [2]:
1. “Is there is a clear statement of the aim of the research?”
2. “Is the study is put into context of other studies and research?”
3. “Are system or algorithmic design decisions justified?”
4. “Is the test data set reproducible?”
5. “Is the study algorithm reproducible?”
6. “Is the experimental procedure thoroughly explained and reproducible?”
7. “Is it clearly stated in the study which other algorithms the study’s algo-
rithm(s) have been compared with?”
8. “Are the performance metrics used in the study explained and justified?”
9. “Are the test results thoroughly analyzed?”
10. “Does the test evidence support the findings presented?”
When performing the searching probably are collected a large number of articles referring to the review in question. In articles extracted from the databases it is suggested to use JabRef to removing the duplicates, the articles that remains should be finally analyzed by the study.
In the first evaluation step, title and summary led to the removal a part of the articles, leaving a much smaller quantity of articles to be analyzed in the next step. Second step is to evaluate the introduction of the articles already approved in the previous step. The last step evaluates the remaining articles from step two. To summarize the results of this step the articles can be organized in a Table. For instance as demonstrated in Table 4, where the objective is to evaluate the articles according to the questions mentioned before. Numbers 1 through 10 correspond to the questions set out before, respectively.
Table 4. Results of last stage (example table)
| Article Title | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | Points |
| Article Title 1 | 1 | 1 | 0.5 | 0 | 0.5 | 0.5 | 0.5 | 0.5 | 1 | 1 | 6.5 |
| Article Title 2 | 1 | 1 | 0.5 | 0 | 0.5 | 0.5 | 1 | 1 | 1 | 1 | 7.5 |
| Article Title 3 | 1 | 1 | 1 | 0 | 0 | 0.5 | 1 | 1 | 1 | 1 | 7.5 |
| Article Title 4 | 1 | 1 | 1 | 0.5 | 0.5 | 1 | 1 | 0.5 | 0.5 | 0.5 | 7.5 |
| Article Title 5 | 1 | 0.5 | 0.5 | 0 | 0 | 0.5 | 0 | 0 | 0 | 0 | 2.5 |
| Article Title 6 | 1 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2 |
| Article Title 7 | 1 | 1 | 1 | 0.5 | 0.5 | 0.5 | 1 | 0 | 0 | 0 | 5.5 |
| Article Title 8 | 1 | 1 | 1 | 0.5 | 0.5 | 0.5 | 1 | 0.5 | 0 | 0 | 6 |
| Article Title 9 | 0.5 | 0.5 | 1 | 0 | 0.5 | 0.5 | 0.5 | 0 | 0 | 0 | 3.5 |
For the selection of the final articles, they should be ranked according to their final score from Table 4. In case of the same score, the rank order could be presented in alphabetical order. As a cutoff note only for example in this article, it was decided to use at least 6 points, so the study needs to address at least 60% of punctuated criteria. However as mentioned before, this cutoff note is decided by the authors. So, according to the example only 5 articles should be within the scope of SRL and meet minimally the quality requirements that a scientific paper must contain. These articles are listed above:
1. Article 2
2. Article 3
3. Article 4
4. Article 1
5. Article 8
Any questions? Send me a message and we can talk about it! Contact: karine.ramos@poatek.com
References
1. Galvão, T. F.; Pereira, M. G. Revisões sistemáticas da literatura: passos para sua elaboração. Epidemiologia e Serviços de Saúde, 183-–184 (2014)
2. Kofod-Petersen, A. How to do a Structured Literature Review in computer science (2012)
3. Okoli, C.; Schabram, K. A Guide to Conducting a Systematic Literature Review of Information Systems Research. SSRN Electronic Journal (2010)
4. Singh, J. et al. Mendeley: A free research management tool for desktop and web. Journal of Pharmacology and Pharmacotherapeutics 1(1), (2010)
5. TEAM, J. D. et al. JabRef [Software].
