Comparative analysis of activation functions of artificial neural network for prediction of optimal groundwater level in the middle mountainous area of Pyoseon watershed in Jeju Island

Mun-Ju Shin; Jin-Woo Kim; Duk-Chul Moon; Jeong-Han Lee; Kyung Goo Kang

doi:10.3741/JKWRA.2021.54.S-1.1143

All Issue

2021 Vol.54, Issue 12S Preview Page Next Page

Research Article

Comparative analysis of activation functions of artificial neural network for prediction of optimal groundwater level in the middle mountainous area of Pyoseon watershed in Jeju Island 제주도 표선유역 중산간지역의 최적 지하수위 예측을 위한 인공신경망의 활성화함수 비교분석

31 December 2021. pp. 1143-1154

PDF XML

Abstract

The selection of activation function has a great influence on the groundwater level prediction performance of artificial neural network (ANN) model. In this study, five activation functions were applied to ANN model for two groundwater level observation wells in the middle mountainous area of the Pyoseon watershed in Jeju Island. The results of the prediction of the groundwater level were compared and analyzed, and the optimal activation function was derived. In addition, the results of LSTM model, which is a widely used recurrent neural network model, were compared and analyzed with the results of the ANN models with each activation function. As a result, ELU and Leaky ReLU functions were derived as the optimal activation functions for the prediction of the groundwater level for observation well with relatively large fluctuations in groundwater level and for observation well with relatively small fluctuations, respectively. On the other hand, sigmoid function had the lowest predictive performance among the five activation functions for training period, and produced inappropriate results in peak and lowest groundwater level prediction. The ANN-ELU and ANN-Leaky ReLU models showed groundwater level prediction performance comparable to that of the LSTM model, and thus had sufficient potential for application. The methods and results of this study can be usefully used in other studies.

Keywords

Activation function

ANN

LSTM

Groundwater level prediction

Pyoseon watershed in Jeju Island

활성화함수의 선택은 인공신경망(Artificial Neural Network, ANN) 모델의 지하수위 예측성능에 큰 영향을 미친다. 특히 제주도의 중산간 지역과 같이 지하수위의 변동폭이 크고 변동양상이 복잡한 경우 적절한 지하수위 예측을 위해서는 다양한 활성화함수의 비교분석을 통한 최적의 활성화함수 선택이 반드시 필요하다. 본 연구에서는 지하수위의 변동폭이 크고 변동양상이 복잡한 제주도 표선유역 중산간지역 2개 지하수위 관측정을 대상으로 5개의 활성화함수(sigmoid, hyperbolic tangent (tanh), Rectified Linear Unit (ReLU), Leaky Rectified Linear Unit (Leaky ReLU), Exponential Linear Unit (ELU))를 ANN 모델에 적용하여 지하수위 예측결과를 비교 및 분석하고 최적 활성화함수를 도출하였다. 그리고 최근 널리 사용되고 있는 순환신경망 모델인 Long Short-Term Memory (LSTM) 모델의 결과와 비교분석하였다. 분석결과 지하수위 변동폭이 상대적으로 큰 관측정과 상대적으로 작은 관측정에 대한 지하수위 예측에 대해서는 각각 ELU와 Leaky ReLU 함수가 최적의 활성화함수로 도출되었다. 반면 sigmoid 함수는 학습기간에 대해 5개 활성화함수 중 예측성능이 가장 낮았으며 첨두 및 최저 지하수위 예측에서 적절하지 못한 결과를 도출하였다. 따라서 ANN-sigmoid 모델은 가뭄기간의 지하수위 예측을 통한 지하수자원 관리목적으로 사용할 경우 주의가 필요하다. ANN-ELU와 ANN-Leaky ReLU 모델은 LSTM 모델과 대등한 지하수위 예측성능을 보여 활용가능성이 충분히 있으며 LSTM 모델은 ANN 모델들 보다 예측성능이 높아 인공지능 모델의 예측성능 비교분석 시 참고 모델로 활용될 수 있다. 마지막으로 학습기간의 정보량에 따라 학습기간의 지하수위 예측성능이 검증 및 테스트 기간의 예측성능보다 낮을 수 있다는 것을 확인하였으며, 관측지하수위의 변동폭이 크고 변동양상이 복잡할수록 인공지능 모델별 지하수위 예측능력의 차이는 커졌다. 본 연구에서 제시한 5개의 활성화함수를 적용한 연구방법 및 비교분석 결과는 지하수위 예측뿐만 아니라 일단위 하천유출량 및 시간단위 홍수량 등 지표수 예측을 포함한 다양한 연구에 유용하게 사용될 수 있다.

키워드

활성화함수

ANN

LSTM

지하수위 예측

제주도 표선유역

References

Afzaal, H., Farooque, A.A., Abbas, F., Acharya, B., and Esau, T. (2020). "Groundwater estimation from major physical hydrology components using artificial neural networks and deep learning." Water, Vol. 12, No. 1, p. 5. 10.3390/w12010005

Barthel, R., and Banzhaf, S. (2016). "Groundwater and surface water interaction at the regional-scale - a review with focus on regional integrated models." Water Resources Management, Vol. 30, No. 1, pp. 1-32. 10.1007/s11269-015-1163-z

Bengio, Y., Simard, P., and Frasconi, P. (1994). "Learning long-term dependencies with gradient descent is difficult." IEEE Transactions on Neural Networks, Vol. 5, No. 2, pp. 157-166. 10.1109/72.27918118267787

Chang, J., Wang, G., and Mao, T. (2015). "Simulation and prediction of suprapermafrost groundwater level variation in response to climate change using a neural network model." Journal of Hydrology, Vol. 529, pp. 1211-1220. 10.1016/j.jhydrol.2015.09.038

Chollet, F. Allaire, J.J. (2018). Deep learning with R, Manning Publications, Shelter Island, NY, U.S., p. 360.

Clevert, D.A., Unterthiner, T., and Hochreiter, S. (2016). "Fast and accurate deep network learning by exponential linear units (ELUs)." arXiv preprint arXiv:1511.07289.

Coulibaly, P., Anctil, F., Aravena, R., and Bobée, B. (2001). "Artificial neural network modeling of water table depth fluctuations." Water Resources Research, Vol. 37, No. 4, pp. 885-896. 10.1029/2000WR900368

Emamgholizadeh, S., Moslemi, K., and Karami, G. (2014). "Prediction the groundwater level of bastam plain (Iran) by artificial neural network (ANN) and adaptive neuro-fuzzy inference system (ANFIS)." Water Resources Management, Vol. 28, No. 15, pp. 5433-5446. 10.1007/s11269-014-0810-0

Falbel, D., Allaire, J.J., Chollet, F., Tang, Y., Van Der Bijl, W., Studer, M., Keydana, S. (2019). R interface to 'Keras'. R package version 2.2.4.1, accessed on 5 April 2019, <https://CRAN.R-project.org/package=keras>.

Hahnloser, R.H., Sarpeshkar, R., Mahowald, M.A., Douglas, R.J., and Seung, H.S. (2000). "Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit." Nature, Vol. 405, No. 6789, pp. 947-951. 10.1038/3501607210879535

Haykin, S. (2009). Neural networks and learning machines, Pearson Prentice Hall, Upper Saddle River, NJ, U.S.

Hochreiter, S., and Schmidhuber, J. (1997). "Long short-term memory." Neural Computation, Vol. 9, No. 8, pp. 1735-1780. 10.1162/neco.1997.9.8.17359377276

Hosseini, Z., Gharechelou, S., Nakhaei, M., and Gharechelou, S. (2016). "Optimal design of BP algorithm by ACO_R model for groundwater-level forecasting: A case study on Shabestar plain, Iran." Arabian Journal of Geosciences, Vol. 9, No. 6, p. 436. 10.1007/s12517-016-2454-2

Jeju Special Self-Governing Province (JSSGP) (2018). Comprehensive water resources management plan in Jeju Island. pp. 1-328.

Jeong, J., and Park, E. (2019). "Comparative applications of data-driven models representing water table fluctuations." Journal of Hydrology, Vol. 572, pp. 261-273. 10.1016/j.jhydrol.2019.02.051

Jha, M.K., and Sahoo, S. (2014). "Efficacy of neural network and genetic algorithm techniques in simulating spatio‐temporal fluctuations of groundwater." Hydrological Processes, Vol. 29, No. 5, pp. 671-691. 10.1002/hyp.10166

Kim, G.B., and Oh, D.H. (2018). "Determination of the groundwater yield of horizontal wells using an artificial neural network model incorporating riverside groundwater level data." The Journal of Engineering Geology, Vol. 28, No. 4, pp. 583-592.

Kim, J., Jun, S.M., Hwang, S., Kim, H.K., Heo, J., and Kang, M.S. (2021a). "Impact of activation functions on flood forecasting model based on artificial neural networks." Journal of The Korean Society of Agricultural Engineers, Vol. 63, No. 1, pp. 11-25.

Kim, M., Choi, J.Y., Bang, J., Yoon, P., and Kim, K. (2021b). "Comparison of artificial neural network model capability for runoff estimation about activation functions." Journal of The Korean Society of Agricultural Engineers, Vol. 63, No. 1, pp. 103-116.

Kim, T.W., and Valdés, J.B. (2003). "Nonlinear model for drought forecasting based on a conjunction of wavelet transforms and neural networks." Journal of Hydrologic Engineering, Vol. 8, No. 6, pp. 319-328. 10.1061/(ASCE)1084-0699(2003)8:6(319)

Kingma, D.P., and Ba, J. (2014). "Adam: A method for stochastic optimization." arXiv preprint arXiv:1412.6980.

Klemeš, V. (1986). "Operational testing of hydrological simulation models." Hydrological Sciences Journal, Vol. 31, No. 1, pp. 13-24. 10.1080/02626668609491024

Krishna, B., Satyaji Rao, Y.R., and Vijaya, T. (2008). "Modelling groundwater levels in an urban coastal aquifer using artificial neural networks." Hydrological Processes, Vol. 22, No. 8, pp. 1180-1188. 10.1002/hyp.6686

Le, X.H., Ho, H.V., Lee, G., and Jung, S. (2019). "Application of long short-term memory (LSTM) neural network for flood forecasting." Water, Vol. 11, No. 7, p. 1387. 10.3390/w11071387

Lee, S., Lee, K.K., and Yoon, H. (2019). "Using artificial neural network models for groundwater level forecasting and assessment of the relative impacts of influencing factors." Hydrogeology Journal, Vol. 27, No. 2, pp. 567-579. 10.1007/s10040-018-1866-3

Maas, A.L., Hannun, A.Y., and Ng, A.Y. (2013). "Rectifier nonlinearities improve neural network acoustic models." Proceedings of the 30 th International Conference on Machine Learning, Atlanta, GA, U.S., Vol. 30, No. 1, p. 3.

Maier, H.R., and Dandy, G.C. (2000). "Neural networks for the prediction and forecasting of water resources variables: A review of modelling issues and applications." Environmental Modelling & Software, Vol. 15, No. 1, pp. 101-124. 10.1016/S1364-8152(99)00007-9

Maxwell, R.M., Condon, L.E., and Kollet, S.J. (2015). "A high-resolution simulation of groundwater and surface water over most of the continental US with the integrated hydrologic model ParFlow v3." Geoscientific Model Development, Vol. 8, No. 3, pp. 923-937. 10.5194/gmd-8-923-2015

McDonald, M.G., and Harbaugh, A.W. (1988). A modular three-dimensional finite-difference ground-water flow model. Vol. 6. US Geological Survey, Reston, VA, U.S.

Mohanty, S., Jha, M.K., Kumar, A., and Panda, D.K. (2013). "Comparative evaluation of numerical model and artificial neural network for simulating groundwater flow in Kathajodi - Surua Inter-basin of Odisha, India." Journal of Hydrology, Vol. 495, pp. 38-51. 10.1016/j.jhydrol.2013.04.041

Moriasi, D.N., Arnold, J.G., Van Liew, M.W., Bingner, R.L., Harmel, R.D., and Veith, T.L. (2007). "Model evaluation guidelines for systematic quantification of accuracy in watershed simulations." Transactions of the ASABE, Vol. 50, No. 3, pp. 885-900. 10.13031/2013.23153

Müller, J., Park, J., Sahu, R., Varadharajan, C., Arora, B., Faybishenko, B., and Agarwal, D. (2021). "Surrogate optimization of deep neural networks for groundwater predictions." Journal of Global Optimization, Vol. 81, No. 1, pp. 203-231. 10.1007/s10898-020-00912-0

Nash, J.E., and Sutcliffe, J.V. (1970). "River flow forecasting through conceptual models part I - A discussion of principles." Journal of Hydrology, Vol. 10, No. 3, pp. 282-290. 10.1016/0022-1694(70)90255-6

Nayak, P.C., Rao, Y.S., and Sudheer, K.P. (2006). "Groundwater level forecasting in a shallow aquifer using artificial neural network approach." Water Resources Management, Vol. 20, No. 1, pp. 77-90. 10.1007/s11269-006-4007-z

Prechelt, L. (2012) "Early stopping - But when?." Neural networks: Tricks of the trade, Edited by Montavon G., Orr G.B., and Müller KR., Springer, Berlin Heidelberg, pp. 53-67. 10.1007/978-3-642-35289-8_5

Rajaee, T., Ebrahimi, H., and Nourani, V. (2019). "A review of the artificial intelligence methods in groundwater level modeling." Journal of Hydrology, Vol. 572, pp. 336-351. 10.1016/j.jhydrol.2018.12.037

Rakhshandehroo, G.R., Vaghefi, M., and Aghbolaghi, M.A. (2012). "Forecasting groundwater level in Shiraz plain using artificial neural networks." Arabian Journal for Science and Engineering, Vol. 37, No. 7, pp. 1871-1883. 10.1007/s13369-012-0291-5

Sahoo, S., and Jha, M.K. (2013). "Groundwater-level prediction using multiple linear regression and artificial neural network techniques: a comparative assessment." Hydrogeology Journal, Vol. 21, No. 8, pp. 1865-1887. 10.1007/s10040-013-1029-5

Sahoo, S., Russo, T.A., Elliott, J., and Foster, I. (2017). "Machine learning algorithms for modeling groundwater level changes in agricultural regions of the US." Water Resources Research, Vol. 53, No. 5, pp. 3878-3895. 10.1002/2016WR019933

Shin, M.J., Moon, S.H., Kang, K.G., Moon, D.C., and Koh, H.J. (2020). "Analysis of groundwater level variations caused by the changes in groundwater withdrawals using long short-term memory network." Hydrology, Vol. 7, No. 3, p. 64. 10.3390/hydrology7030064

Sit, M., Demiray, B.Z., Xiang, Z., Ewing, G.J., Sermet, Y., and Demir, I. (2020). "A comprehensive review of deep learning applications in hydrology and water resources." Water Science and Technology, Vol. 82, No. 12, pp. 2635-2670. 10.2166/wst.2020.36933341760

Sun, Y., Wendi, D., Kim, D.E., and Liong, S.Y. (2016). "Application of artificial neural networks in groundwater table forecasting-a case study in a Singapore swamp forest." Hydrology and Earth System Sciences, Vol. 20, No. 4. pp. 1405-1412. 10.5194/hess-20-1405-2016

Taormina, R., Chau, K.W., and Sethi, R. (2012). "Artificial neural network simulation of hourly groundwater levels in a coastal aquifer system of the Venice lagoon." Engineering Applications of Artificial Intelligence, Vol. 25, No. 8, pp. 1670-1676. 10.1016/j.engappai.2012.02.009

Todd, D.K. and Larry, W.M. (2004). Groundwater hydrology, Third edition. John Wiley & Sons Inc., Hoboken, NJ, U.S., pp. 1-656.

Ukkonen, P., and Mäkelä, A. (2019). "Evaluation of machine learning classifiers for predicting deep convection." Journal of Advances in Modeling Earth Systems, Vol. 11, No. 6, pp. 1784-1802. 10.1029/2018MS001561

Vu, M.T., Jardani, A., Massei, N., and Fournier, M. (2021). "Reconstruction of missing groundwater level data by using Long Short-Term Memory (LSTM) deep neural network." Journal of Hydrology, Vol. 597, p. 125776. 10.1016/j.jhydrol.2020.125776

Wen, X., Feng, Q., Deo, R.C., Wu, M., and Si, J. (2017). "Wavelet analysis - artificial neural network conjunction models for multi-scale monthly groundwater level predicting in an arid inland river basin, northwestern China." Hydrology Research, Vol. 48, No. 6, pp. 1710-1729. 10.2166/nh.2016.396

White, J.T., Doherty, J.E., and Hughes, J.D. (2014). "Quantifying the predictive consequences of model error with linear subspace analysis." Water Resources Research, Vol. 50, No. 2, pp. 1152-1173. 10.1002/2013WR014767

White, J.T., Knowling, M.J., and Moore, C.R. (2020). "Consequences of groundwater-Model vertical discretization in risk-Based decision-Making." Groundwater, Vol. 58, No. 5, pp. 695-709. 10.1111/gwat.1295731667821

Worland, S.C., Steinschneider, S., Asquith, W., Knight, R., and Wieczorek, M. (2019). "Prediction and inference of flow duration curves using multioutput neural networks." Water Resources Research, Vol. 55, No. 8, pp. 6850-6868. 10.1029/2018WR024463

Xu, B., Wang, N., Chen, T., and Li, M. (2015). "Empirical evaluation of rectified activations in convolutional network." arXiv preprint arXiv:1505.00853.

Yoon, H., Hyun, Y., Ha, K., Lee, K.K., and Kim, G.B. (2016). "A method to improve the stability and accuracy of ANN- and SVM-based time series models for long-term groundwater level predictions." Computers & Geosciences, Vol. 90, pp. 144-155. 10.1016/j.cageo.2016.03.002

Yoon, H., Jun, S.C., Hyun, Y., Bae, G.O., and Lee, K.K. (2011). "A comparative study of artificial neural networks and support vector machines for predicting groundwater levels in a coastal aquifer." Journal of Hydrology, Vol. 396, No. 1-2, pp. 128-138. 10.1016/j.jhydrol.2010.11.002

Yu, H., Wen, X., Feng, Q., Deo, R.C., Si, J., and Wu, M. (2018). "Comparative study of hybrid-wavelet artificial intelligence models for monthly groundwater depth forecasting in extreme arid regions, Northwest China." Water Resources Management, Vol. 32, No. 1, pp. 301-323. 10.1007/s11269-017-1811-6

Information

Publisher :KOREA WATER RESOURECES ASSOCIATION
Publisher(Ko) :한국수자원학회
Journal Title :Journal of Korea Water Resources Association
Journal Title(Ko) :한국수자원학회 논문집
Volume : 54
No :12
Pages :1143-1154
Received Date : 2021-09-27
Revised Date : 2021-11-09
Accepted Date : 2021-11-18
DOI :https://doi.org/10.3741/JKWRA.2021.54.S-1.1143

Journal of Korea Water Resources Association ISSN:2799-8746(Print) 2799-8754(Online) 한국수자원학회 논문집

All Issue