ROC evaluation for MLP ANN drought forecasting model

Min-Su Jeong; Jong-Suk Kim; Ho-Won Jang; Joo-Heon Lee

doi:10.3741/JKWRA.2016.49.10.877

All Issue

2016 Vol.49, Issue 10 Preview Page Next Page

ROC evaluation for MLP ANN drought forecasting model MLP ANN 가뭄 예측 모형에 대한 ROC 평가

31 October 2016. pp. 877-885

PDF XML

Abstract

In this study, the Standard Precipitation Index(SPI), meteorological drought index, was used to evaluate the temporal and spatial assessment of drought forecasting results for all cross Korea. For the drought forecasting, the Multi Layer Perceptron-Artificial Neural Network (MLP-ANN) was selected and the drought forecasting was performed according to different forecasting lead time for SPI (3) and SPI (6). The precipitation data observed in 59 gaging stations of Korea Meteorological Adminstration (KMA) from 1976~2015. For the performance evaluation of the drought forecasting, the binary classification confusion matrix, such as evaluating the status of drought occurrence based on threshold, was constituted. Then Receiver Operating Characteristics (ROC) score and F score according to conditional probability are computed. As a result of ROC analysis on forecasting performance, drought forecasting performance, of applying the MLP-ANN model, shows satisfactory forecasting results. Consequently, two-month and five-month leading forecasts were possible for SPI (3) and SPI (6), respectively.

Keywords

Meteorological Drought

Standardized Precipitation Index (SPI)

MLP ANN

ROC

F score

본 연구에서는 기상학적 가뭄지수인 표준강수지수(Standardized Precipitation Index, SPI)를 이용하여 우리나라 전역에 대한 가뭄예측의 시공간적인 평가를 수행하였다. 또한 다층 퍼셉트론 인공신경망(Multi Layer Perceptron-Artificial Neural Network, MLP-ANN) 예측 기법을 이용하여 SPI(3), (6)에 대한 선행예보시간별 가뭄 예측을 실시하였다. 입력 자료는 기상청 산하의 59개 관측소에서 관측된 기상자료를 활용하였고, 관측자료 기간은 1976~2015년이다. 예측 모델의 성능평가는 기준점(Threshold)에 따른 가뭄 발생유무와 같은 이진분류 혼동행렬을 구성하여 Receiver Operating Characteristics (ROC) score와 조건부 확률에 따른 F score를 산정하여 예측 성능평가를 수행하였다. 예측성능에 대한 ROC 분석결과 다층 퍼셉트론 인공신경망(MLP-ANN) 모형을 적용한 가뭄예측성능이 매우 우수한 것으로 나타났으며, SPI (3)은 2개월, SPI (6)는 5개월 정도의 선행예측이 충분히 가능한 것으로 나타났다.

키워드

기상학적 가뭄

표준강수지수(SPI)

MLP ANN

ROC

F score

References

Bae, D. H., Son, K. H., Ahn, J. B., Hong, J. Y., Kim, G. S., Chung, J. S., Jung, U. S., and Kim, J. K. (2012). “Development of real-time drought monitoring and prediction system on korea & east asia region.” Atmosphere Korean Meteorological Society, Vol. 22, No. 2, pp. 267-277.

Bae, D. H., Son, K. H., and Kim, H. A. (2013). “Derivation & evaluation of drought threshold level considering hydro- meteorological data on south korea.” J. Korea Water Resources Association, Vol. 46, No. 3, pp. 287-299.

Egan, J. P. (1975). “Signal detection theory and ROC analysis, series in cognition and perception.” Academic Press, New York.

Heckerling, P. S. (2002). “Parametric receiver operating characteristic curve analysis using mathematica.” Computer Methods and Programs in Biomedicine, Vol. 69, No. 1, pp. 65-73.

IPCC (2014). Fifth Assessment Report (AR5) ｢Climate change 2014 : impacts, adaptation, and vulnerability｣

Kang, B. S., and Lee, B. K. (2011). “Application of artificial neural network to improve auntitative precipitation forecasts of meso-scale numerical weather prediction.” Journal of Korea Water Resources Association, Vol. 44, No. 2, pp. 97-107.

Kim, G. S., and Lee, J. W. (2011). “Evaluation on drought indices using the drought Records.” J. Korea Water Resources Association, Vol. 44, No. 8, pp. 639-652.

Kwon, H. H., Moon, J. W., Song, H. S., and Moon, Y. I. (2009). “Climate information and GCMs seasonal forecasts based short-term forecasts for drought.” Conference of Korea Water Resources Association, pp. 1186-1190.

Lee, J. H., Kim, J. S., Jang, H. W., and Lee, J. C. (2013) “Drought forecasting using the multi layer perceptron(MLP) artificial neural network model.” J. Korea Water Resources Association, Vol. 46, No. 12, pp. 1249-1263.

Mckee, T. B., Doesken, N. J., and Kleist, J. (1995). “Drought monitering with multiple time scales preprints.” 9th Conference on Applied Climatology, 15-20 Janiary, Dallas, TX, pp. 233-236.

Murphy, A. H. (1993). “What is a good forecast? an essay on the nature of goodness in weather forecasting.” American Meteorlogical Society, Vol. 8, pp. 281-293.

Provost, F., and Fawcett, T. (1998). “Robust classification systems for imprecise environments.” In Proceedings of AAAI-98. AAAI Press. To appear. Available: http://www.croftj.net/~ fawcett/papers/aaai98-dist.ps.gz.

Provost, F., Fawcett, T., and Kohavi, R. (1998). The case against accuracy estimation for comparing induction algorithms. In: Shavlik, J. (Ed.), Proc. ICML-98. Morgan Kaufmann, San Francisco, CA, pp. 445-453.

Son, K. H., Bae, D. H., and Cheong, H. S. (2015). “Construction & evaluation of glosea5-based hydrological drought outlook system.” Atmosphere, Vol. 25, No. 2, pp. 271-281.

Spackman, K. A. (1989). “Signal detection theory: valuable tools for evaluating inductive learning. In: Proc. Sixth Internat”. Workshop on Machine Learning. Morgan Kaufman, San Mateo, CA, pp. 160-163.

Swets, J. A., Dawes, R. M., and Monahan, J. (2000). “Better decisions through science.” Scientific American 283, pp. 82-87.

Yoo, J. Y., Song, H., Kim, T. W., and Ahn, J. H. (2013). “Evaluation of short-term drought using daily standardized precipitation index and ROC analysis.” Journal of The Korean Society of Civil Engineers, Vol. 33, No. 5, pp. 1851-1860.

Information

Publisher :KOREA WATER RESOURECES ASSOCIATION
Publisher(Ko) :한국수자원학회
Journal Title :Journal of Korea Water Resources Association
Journal Title(Ko) :한국수자원학회 논문집
Volume : 49
No :10
Pages :877-885
Received Date : 2016-06-08
Revised Date : 2016-08-17
Accepted Date : 2016-09-07
DOI :https://doi.org/10.3741/JKWRA.2016.49.10.877

Journal of Korea Water Resources Association ISSN:2799-8746(Print) 2799-8754(Online) 한국수자원학회 논문집

All Issue