Improvement of multi layer perceptron performance using combination of adaptive moments and improved harmony search for prediction of Daecheong Dam inflow

Won Jin Lee; Eui Hoon Lee

doi:10.3741/JKWRA.2023.56.1.63

All Issue

2023 Vol.56, Issue 1 Preview Page

Research Article

Improvement of multi layer perceptron performance using combination of adaptive moments and improved harmony search for prediction of Daecheong Dam inflow 대청댐 유입량 예측을 위한 Adaptive Moments와 Improved Harmony Search의 결합을 이용한 다층퍼셉트론 성능향상

31 January 2023. pp. 63-74

PDF XML

Abstract

High-reliability prediction of dam inflow is necessary for efficient dam operation. Recently, studies were conducted to predict the inflow of dams using Multi Layer Perceptron (MLP). Existing studies used the Gradient Descent (GD)-based optimizer as the optimizer among MLP operators to find the optimal correlation between data. However, the GD-based optimizers have disadvantages in that the prediction performance is deteriorated due to the possibility of convergence to the local optimal value and the absence of storage space. This study improved the shortcomings of the GD-based optimizer by developing Adaptive moments combined with Improved Harmony Search (AdamIHS), which combines Adaptive moments among GD-based optimizers and Improved Harmony Search (IHS). In order to evaluate the learning and prediction performance of MLP using AdamIHS, Daecheong Dam inflow was learned and predicted and compared with the learning and prediction performance of MLP using GD-based optimizer. Comparing the learning results, the Mean Squared Error (MSE) of MLP, which is 5 hidden layers using AdamIHS, was the lowest at 11,577. Comparing the prediction results, the average MSE of MLP, which is one hidden layer using AdamIHS, was the lowest at 413,262. Using AdamIHS developed in this study, it will be possible to show improved prediction performance in various fields.

Keywords

Dam inflow

Optimizer

Adaptive moments

Improved harmony search

Adaptive moments combined with improved harmony search

높은 신뢰도의 댐 유입량 예측은 효율적인 댐 운영을 위해 필요하다. 최근 다층퍼셉트론(Multi Layer Perceptron, MLP)을 활용하여 댐의 유입량을 예측하는 연구들이 진행되었다. 기존 연구들은 MLP의 연산자 중 자료 간의 최적 상관관계를 찾는 optimizer로 경사하강법(Gradient Descent, GD) 기반의 optimizer를 사용하였다. 하지만, GD 기반의 optimizer들은 지역 최적값으로의 수렴 가능성과 저장공간 부재로 인해 예측성능이 저하된다는 단점이 있다. 본 연구는 GD 기반 optimizer 중 Adaptive moments와 Improved Harmony Search (IHS)를 결합한 Adaptive moments combined with Improved Harmony Search (AdamIHS)를 개발하여 GD 기반 optimizer의 단점을 개선하였다. AdamIHS를 사용한 MLP의 학습 및 예측성능을 평가하기 위해 대청댐 유입량을 학습 및 예측하였으며, GD 기반 optimizer를 사용한 MLP의 학습 및 예측성능과 비교하였다. 학습결과를 비교하면, AdamIHS를 사용한 은닉층 5개인 MLP의 Mean Squared Error (MSE) 평균값이 11,577로 가장 낮았다. 예측결과를 비교하면, AdamIHS를 사용한 은닉층 1개인 MLP의 MSE 평균값이 413,262로 가장 낮았다. 본 연구에서 개발된 AdamIHS를 활용하면 다양한 분야에서 향상된 예측성능을 보여줄 수 있을 것이다.

키워드

댐 유입량

Optimizer

Adaptive moments

Improved harmony search

Adaptive moments combined with improved harmony search

References

Agatonovic-Kustrin, S., and Beresford, R. (2000). "Basic concepts of artificial neural network (ANN) modeling and its application in pharmaceutical research." Journal of Pharmaceutical and Biomedical Analysis, Vol. 22, No. 5, pp. 717-727. 10.1016/S0731-7085(99)00272-110815714

Amjady, N., and Keynia, F. (2011). "A new neural network approach to short term load forecasting of electrical power systems." Energies, Vol. 4, No. 3, pp. 488-503. 10.3390/en4030488

Aqil, M., Kita, I., Yano, A., and Nishiyama, S. (2007). "A comparative study of artificial neural networks and neuro-fuzzy in continuous modeling of the daily and hourly behaviour of runoff." Journal of Hydrology, Vol. 337, No. 1-2, pp. 22-34. 10.1016/j.jhydrol.2007.01.013

Babaei, M., Moeini, R., and Ehsanzadeh, E. (2019). "Artificial neural network and support vector machine models for inflow prediction of dam reservoir (case study: Zayandehroud dam reservoir)." Water Resources Management, Vol. 33, No. 6, pp. 2203-2218. 10.1007/s11269-019-02252-5

Cho, Y.T., Lee, G.H., Hong, J.H., and Geem, G.W. (2022). "Prediction of photovoltaic generation using machine learning models with various weight optimization techniques." Journal of Korean Institute of Intelligent Systems, Vol. 32, No. 1, pp. 1-6. 10.5391/JKIIS.2022.32.1.1

Choi, K. (2016). "Real-time artificial neural network for high-dimensional medical image." The Korean Society of Radiology, Vol. 10, No. 8, pp. 637-643. 10.7742/jksr.2016.10.8.637

Dorigo, M., Birattari, M., and Stutzle, T. (2006). "Ant colony optimization." IEEE Computational Intelligence Magazine, Vol. 1, No. 4, pp. 28-39. 10.1109/MCI.2006.329691

Eom, J., and Jung, K. (2019). "Estimation of hourly dam inflow using time series data." Journal of the Korean Society of Hazard Mitigation, Vol. 19, No. 2, pp. 163-168. 10.9798/KOSHAM.2019.19.2.163

Geem, Z.W., Kim, J.H., and Loganathan, G.V. (2001). "A new heuristic optimization algorithm: harmony search." Simulation, Vol. 76, No. 2, pp. 60-68. 10.1177/003754970107600201

Geem, Z.W., Kim, J.H., and Loganathan, G.V. (2002). "Application of harmony search algorithm to water resources problems." Conference of the Environmental and Water Resources Institute of ASCE, ASCE, Roanoke, VA, U.S.

Göçken, M., Özçalıcı, M., Boru, A., and Dosdoğru, A.T. (2016). "Integrating metaheuristics and artificial neural networks for improved stock price prediction." Expert Systems with Applications, Vol. 44, pp. 320-331. 10.1016/j.eswa.2015.09.029

Goldberg, D.E., and Holland, J.H. (1988). "Genetic algorithms and machine learning." Machine Learning, Vol. 3, pp. 95-99. 10.1023/A:1022602019183

Granata, F., Gargano, R., and De Marinis, G. (2016). "Support vector regression for rainfall-runoff modeling in urban drainage: A comparison with the EPA's storm water management model." Water, Vol. 8, No. 3, 69. 10.3390/w8030069

Han, H., Choi, C., Jung, J., and Kim, H.S. (2021). "Application of sequence to sequence learning based LSTM model (LSTM-s2s) for forecasting dam inflow." Journal of Korea Water Resources Association, Vol. 54, No. 3, pp. 157-166.

Jeon, G., Park, J., Jung, J., and Yoon, H. (2021). "Structural response estimation using gated recurrent unit." Korean Society of Hazard Mitigation, Vol. 21, pp. 171-179. 10.9798/KOSHAM.2021.21.3.171

Joo, G., Park, C., and Im, H. (2020). "Performance evaluation of machine learning optimizers." Journal of IKEEE, Vol. 24, No. 3, pp. 766-776.

Jung, S., Lee, D., and Lee, K. (2018). "Prediction of river water level using deep-learning open library." Journal of Korean Society of Hazard Mitigation, Vol. 18, No. 1, pp. 1-11. 10.9798/KOSHAM.2018.18.1.1

Kennedy, J., and Eberhart, R. (1995). "Particle swarm optimization." Proceedings of the IEEE International Conference on Neural Networks, Vol. 4, pp. 1942-1948. 10.1109/ICNN.1995.488968

Kim, D., Kim, Y., Jang, D., and Baek, C. (2019). "A study on the estimation of CSVR for flood damage analysis." Journal of the Korean Society Hazard Mitigation, Vol. 19, No. 6, pp. 303-311. 10.9798/KOSHAM.2019.19.6.303

Kim, L.S. (2004). "Firm bankruptcy prediction using artificial neural network models." The East-West Economics Association of Korea, No. 16, pp. 65-80.

Lee, A., Geem, Z.W., and Suh, K.D. (2016a). "Determination of optimal initial weights of an artificial neural network by using the harmony search algorithm: application to breakwater armor stones." Applied Sciences, Vol. 6, No. 6, 164. 10.3390/app6060164

Lee, E.H., Yoo, D.G., Choi, Y.H., and Kim, J.H. (2016b). "Development of the new meta-heuristic optimization algorithm inspired by a vision correction procedure: Vision correction algorithm," Journal of the Korea Academia-Industrial, Vol. 17, No. 3, pp. 117-126. 10.5762/KAIS.2016.17.3.117

Lee, J., Sun, Y.G., Lee, S.M., Kim, S.H., Kim, Y., Lee, W., Sim, I., and Kim, J.Y. (2019). "Implementation of smart meter applying power consumption prediction based on GRU model." The Journal of The Institute of Internet, Broadcasting and Communication, Vol. 19, No. 5, pp. 93-99.

Lee, W.Y., Ko, K.E., Geem, Z.W., and Sim, K.B. (2017). "Method that determining the Hyperparameter of CNN using HS algorithm." Journal of the Korean Institute of Intelligent Systems, Vol. 27, No. 1, pp. 22-28. 10.5391/JKIIS.2017.27.1.022

Mahdavi, M., Fesanghary, M., and Damangir, E. (2007). "An improved harmony search algorithm for solving optimization problems." Applied Mathematics and Computation, Vol. 188, No. 2, pp. 1567-1579. 10.1016/j.amc.2006.11.033

Mahsa, M., and Lee, T. (2018). "Comparison of optimization algorithms in deep learning-based neural networks for hydrological forecasting: Case study of Nam River daily runoff." Journal of the Korean Society of Hazard Mitigation, Vol. 18, No. 6, pp. 377-384. 10.9798/KOSHAM.2018.18.6.377

McCulloch, W.S., and Pitts, W. (1943). "A logical calculus of the ideas immanent in nervous activity." The Bulletin of Mathematical Biophysics, Vol. 5, No. 4, pp. 115-133. 10.1007/BF02478259

Mok, J.Y., Choi, J.H., and Moon, Y.I. (2020). "Prediction of multipurpose dam inflow using deep learning." Journal of Korea Water Resources Association, Vol. 53, No. 2, pp. 97-105. 10.3741/JKWRA.2020.53.2.97

Moon, B.S., and Lee, K.J. (2011). "Prediction of the inflow in Seum-jin Dam according to rainfall." Journal of the Korea Entertainment Industry Association, Vol. 5, No. 2, pp. 104-111. 10.21184/jkeia.2011.06.5.2.104

Park, C., and Chung, I.M. (2020). "Evaluating the groundwater prediction using LSTM model." Journal of Korea Water Resources Association, Vol. 53, No. 4, pp. 273-283. 10.3741/JKWRA.2020.53.4.273

Park, M.K., Yoon, Y.S., Lee, H.H., and Kim, J.H. (2018). "Application of recurrent neural network for inflow prediction into multi-purpose dam basin." Journal of Korea Water Resources Association, Vol. 51, No. 12, pp. 1217-1227. 10.3741/JKWRA.2018.51.12.1217

Riad, S., Mania, J., Bouchaou, L., and Najjar, Y. (2004). "Predicting catchment flow in a semi-arid region via an artificial neural network technique." Hydrological Processes, Vol. 18, No. 13, pp. 2387-2393. 10.1002/hyp.1469

Rosenblatt, F. (1958). "The perceptron: a probabilistic model for information storage and organization in the brain." Psychological Review, Vol. 65, No. 6, 386. 10.1037/h004251913602029

Rumelhart, D.E., Hinton, G.E., and Williams, R.J. (1986). "Learning representations by back-propagating errors." Nature, Vol. 323, No. 6088, pp. 533-536. 10.1038/323533a0

Ryu, Y.M., and Lee, E.H. (2022). "Application of neural networks to predict Daecheong Dam water levels." Journal of the Korean Society of Hazard Mitigation, Vol. 22, No. 1, pp. 67-78. 10.9798/KOSHAM.2022.22.1.67

Sedki, A., Ouazar, D., and El Mazoudi, E. (2009). "Evolving neural network using real coded genetic algorithm for daily rainfall - runoff forecasting." Expert Systems with Applications, Vol. 36, No. 3, pp. 4523-4527. 10.1016/j.eswa.2008.05.024

Seo, Y., Kim, S., Kisi, O., and Singh, V.P. (2015). "Daily water level forecasting using wavelet decomposition and artificial intelligence techniques." Journal of Hydrology, Vol. 520, pp. 224-243. 10.1016/j.jhydrol.2014.11.050

Valipour, M., Banihabib, M.E., and Behbahani, S.M.R. (2012). "Parameters estimate of autogressive moving average and autogressive integrated moving average models and compare their ability for inflow forecasting." Journal of Mathematics and Statistics, Vol. 8, No. 3, pp. 330-338. 10.3844/jmssp.2012.330.338

Valipour, M., Banihabib, M.E., and Behbahani, S.M.R. (2013). "Comparison of the ARMA, ARIMA, and the autogressive artificial neural network models in forecasting the monthly inflow of Dez dam resorvoir." Journal of Hydrology, Vol. 479, pp. 433-441.

Yoo, Y., Kim, D., and Lee, J. (2020). "Performance analysis of various activation functions in super resolution model." Proceedings of the Korea Information Processing Society Conference, Vol. 27, No. 1, pp. 504-507.

Zhang, D., Lindholm, G., and Ratnaweera, H. (2018). "Use long short-term memory to enhance Internet of Things for combined sewer overflow monitoring." Journal of hydrology, Vol. 556, pp. 409-418. 10.1016/j.jhydrol.2017.11.018

Zhou, Z., Li, Z., Snowling, S., Baetz, B.W., and Na, D. (2019). "A random forest model for inflow prediction at wastewater treatment plants." Stochastic Environmental Research and Risk Assessment, Vol. 33, No. 10, pp. 1781-1792. 10.1007/s00477-019-01732-9

Information

Publisher :KOREA WATER RESOURECES ASSOCIATION
Publisher(Ko) :한국수자원학회
Journal Title :Journal of Korea Water Resources Association
Journal Title(Ko) :한국수자원학회 논문집
Volume : 56
No :1
Pages :63-74
Received Date : 2022-10-31
Revised Date : 2022-11-26
Accepted Date : 2022-11-29
DOI :https://doi.org/10.3741/JKWRA.2023.56.1.63

Journal of Korea Water Resources Association ISSN:2799-8746(Print) 2799-8754(Online) 한국수자원학회 논문집

All Issue