A general framework and guidelines for benchmarking computational intelligence algorithms applied to forecasting problems derived from an application domain-oriented survey

Benchmarking computational intelligence algorithms provides valuable knowledge for selecting the best or, at least, the proper algorithm for a certain problem. The experimental results of the computational intelligence techniques applications in various domains, as well as the comparative studies that were reported in the literature can be analyzed and synthesized as development strategies for new successful applications of CI algorithms. Starting from an application domain-oriented survey of selected recently reported research work, the paper presents a general benchmarking framework applicable to computational intelligence algorithms and a set of guidelines for the selection of the best or more suitable CI algorithm for solving forecasting problems. Our approach proposes the integration of software and knowledge engineering best practice towards CI benchmarking, being a computational intelligence engineering methodology. The framework uses two knowledge bases, one for the application domain and one for the CI algorithms, providing heuristic knowledge for a more informed and efficient benchmarking, a case base in which solved problems are recorded with their solution and lessons that were learned, and a knowledge-based problem instance features selection. Some examples of how to apply the framework for problems of forecasting in seismology, environmental protection, hydrology and energy are also discussed. We point out that the framework might be implemented as a software tool (e.g. a decision support system) or as a tool suite. The main conclusion of our research work is that the integration of the derived knowledge from an application domain-oriented survey into the general benchmarking framework along with the set of guidelines for best or proper CI algorithms selection can improve significantly the forecasting accuracy and the response time, in case of real time forecasters.