Literature Review in Three Steps

Literature review is a must-have for most scholarly publications, from journal and conference articles to theses and dissertations. Having reviewed and handled many papers for many prestigious journals, I find a very high correlation between the level of understanding of the literature and quality of the manuscripts. The ones with outstanding literature review are often getting accepted at the end, and vice versa. The ultimate goal of literature review is to convince the reviewers and editors that you have advanced the state-of-the-art by identifying a new problem, developing a new solution, or both.

The figure below illustrates the four types of situations when a researcher starts doing literature review, where I use a hammer to represent the core technologies, techniques and methodologies, and a nail to represent a research problem.

Four Quadrants of Literature Review

• First quadrant - When you have a hammer and a nail, you are doing the literature review to see if you have got the right hammer for the right nail, in other words, whether your work is novel and significant enough for a potential publication.
• Second quadrant - When you have a hammer but have not found a nail, you are doing the literature review to see where this hammer has been used, and trying to figure out if there are other places that you can use your hammer.
• Fourth quadrant - When you have a nail but have not got a hammer. You are doing the literature review to see whether and how other people have been dealing with the nail.
• Third quadrant - When you have neither a hammer nor a nail, you are doing the literature review to get started with your research by looking for a field of interest. 

The nature of research is quite dynamic. Very often a researcher has to go through each quadrant multiple times before writing a publishable paper. The best position is obviously on the top right quadrant, which is a place everyone has to be before concluding a research project. This blog post is written for the ones in the first quadrant. 

Prior to the end of the project, you never know how many papers you are going to read. I read 1200+ load forecasting papers during my PhD study. 94 of them finally went to the reference list of my dissertation. It's impossible to manage these papers manually. A good reference manager software can save you a lot of time. 

In the old days, I used EndNote as the primary reference manager. Since not all my computers have EndNote installed, I used MS Excel as a supplement. Sometimes I also printed out some papers and/or abstracts to read. In today's world, I would recommend Mendeley + Evernote. Both of them offer cloud-based services, so make sure you also install their apps on your mobile devices in addition to the computers.

Step 1. Search

Energy forecasting is an interdisciplinary area. When searching papers, I would recommend Google Scholar, IEEE Xplore and Web of Science. There are also many valuable industry reports and white papers, which can be covered by Google. 

Energy forecasting is also a very noisy field, which means many papers are being published every year. For example, when searching load forecasting on Web of Science, XXX papers show up. I doubt anyone has the bandwidth to read all of them. The objectives are to read as many high quality and relevant papers and to minimize the time on low quality or irrelevant papers.

How to tell whether a paper's quality before reading it?

There are three short-cuts I can offer. Each one has its own drawbacks. A recommended approach is to combining them in practice. 

A. Read experts' work

If you are in load forecasting, you can start with my recommended reading list. Then trace those authors' work. In other areas, you may ask your advisor for a few good papers to start with. 

Note that many good papers are being published every year. Sticking to the well-established experts may result in missing significant work from recent publications, such as the ones in these two lists: GEFCom2012 Papers and Analytics for Energy Forecasting with Applications to Smart Grid.

B. Read highly cited papers

All three academic search engines I recommended earlier offer the citation counts. Usually good papers are being cited frequently. Read the highly cited ones may save you a lot of time. However, high citation count does not equal to high quality. Some times a paper is highly cited just because of a few key words in its title. On the other hand, the newly published papers are in an disadvantage position comparing to the older ones. Therefore, you should consider the publication year when looking at the citation count.

C. Read top journals

The research community today is flooded with too many journals with varying quality. Although you may encounter some so-so papers in some top journals, the top journals tend to publish high quality papers. 

How to find top journals? 

You may look at the impact factor, but it is not quite reliable considering many journals are playing the game of self citation or citing sister journals. The impact factor does not include citations from the conference papers either. Some new journals are prestigious but do not have an impact factor yet. Google Scholar offers an H-index, which seems to be more meaningful than the impact factor. 

Of course the easiest way is to ask your supervisor for the journals to read. In general, the journals I did reviews for are all good journals. At least I tried my best to turn down some low quality papers. Take load forecasting for example. Both power and energy community and forecasting community are interested in the topic. Each community has a large professional society, i.e., IEEE Power & Energy Society (PES) and International Institute of Forecasters (IIF). Both societies publish prestigious journals. 

The following journals from IEEE PES often publish good forecasting papers among other power engineering papers: 

• IEEE Transactions on Smart Grid
• IEEE Transactions on Power Systems
• IEEE Transactions on Sustainable Energy.

IIF has two journals:

• International Journal of Forecasting (IJF)
• Foresight: The International Journal of Applied Forecasting.

IJF collects original research papers in forecasting methodologies and many forecasting topics in various industries including the energy industry. Foresight focuses more on practical issues in business forecasting. Both journals publish very high quality work and can be inspiring to energy forecasting research and practice.

Step 2. Read

Let's say now you have identified dozens of papers to read. Another question pops up:

How to read the scholarly papers? 

First you should be clear about the objective is to answer the following questions:

• What are the authors doing? Short term or long term forecast? Hourly or monthly load? 
• How did they do it? What techniques were applied there, regression analysis, time series analysis or neural networks? Any specific methodology, such as combining forecasts?
• Why did they do it? How did they advance the state of the art at that time? What gap did they fill? How and why was the proposed approach better than its counterparts?
• What can be further improved? Is there any significant technical issue with the proposed approach? What can be further enhanced?
• How is this paper relevant to your own research? What are the differences and similarities? Can your work use some of their methodology, findings or conclusions? Is it a good example for comparison purpose?

Then you have to read them and take notes. There are a few steps I take when reading a paper: 

• Title, abstract and references; 
• Introduction and conclusion; 
• Case studies; 
• Formulation and solution. 

Of course, the more relevant a paper is to your research, the more details you should get into, and the more notes you should take.

Step 3. Compile

After reading these papers, you may find that your work is nothing new nor significant. Then you may have to go back to one of the other three quadrants. Under a different scenario, you may have even more confidence about your work. Then it's time to compile the notes into the so called literature review, which can be a chapter of your thesis/dissertation, or a paragraph of your paper. 

Do not simply do a roster call when you write the literature review. Everything you write should serve your research topic. For instance, if the research topic is to apply fuzzy regression to short term load forecasting, then you can group the literature into:

• Theoretical aspects of fuzzy regression; 
• Applications of fuzzy regression in forecasting; 
• Short term load forecasting techniques; 
• Other similar fuzzy regression based short term load forecasting if there is any. 

When discussing "similar" approaches, make sure you want to highlight the differences and illustrate why and how your proposal is a superior solution. 

The actual literature review process may not sound as easy as the above three steps. We often have to go through these steps back and forth in practice. Hopefully this blog post is helpful to many folks who are struggling with the literature review.