All Issue

2024 Vol.57, Issue 7

Research Article

Prediction of multipurpose dam inflow utilizing catchment attributes with LSTM and transformer models

유역정보 기반 Transformer및 LSTM을 활용한 다목적댐 일 단위 유입량 예측

Hyung Ju Kima
,
Young Hoon Songb*
,
Eun Sung Chungc
김 형주a
,
송 영훈b*
,
정 은성c
aMaster Student, Department Civil Engineering, Seoul National University of Science and Technology, Seoul, Korea
bSenior Researcher, Institute of Construction Future Talent, Seoul National University of Science and Technology, Seoul, Korea
cProfessor, Department Civil Engineering, Seoul National University of Science and Technology, Seoul, Korea
a서울과학기술대학교 건설시스템공학과 석사과정
b서울과학기술대학교 건설미래인재연구소 선임연구원
c서울과학기술대학교 건설시스템공학과 교수
*Corresponding Author
31 July 2024. pp. 437-449
PDF XML Citation
Abstract

Rainfall-runoff prediction studies using deep learning while considering catchment attributes have been gaining attention. In this study, we selected two models: the Transformer model, which is suitable for large-scale data training through the self-attention mechanism, and the LSTM-based multi-state-vector sequence-to-sequence (LSTM-MSV-S2S) model with an encoder-decoder structure. These models were constructed to incorporate catchment attributes and predict the inflow of 10 multi-purpose dam watersheds in South Korea. The experimental design consisted of three training methods: Single-basin Training (ST), Pretraining (PT), and Pretraining-Finetuning (PT-FT). The input data for the models included 10 selected watershed attributes along with meteorological data. The inflow prediction performance was compared based on the training methods. The results showed that the Transformer model outperformed the LSTM-MSV-S2S model when using the PT and PT-FT methods, with the PT-FT method yielding the highest performance. The LSTM-MSV-S2S model showed better performance than the Transformer when using the ST method; however, it showed lower performance when using the PT and PT-FT methods. Additionally, the embedding layer activation vectors and raw catchment attributes were used to cluster watersheds and analyze whether the models learned the similarities between them. The Transformer model demonstrated improved performance among watersheds with similar activation vectors, proving that utilizing information from other pre-trained watersheds enhances the prediction performance. This study compared the suitable models and training methods for each multi-purpose dam and highlighted the necessity of constructing deep learning models using PT and PT-FT methods for domestic watersheds. Furthermore, the results confirmed that the Transformer model outperforms the LSTM-MSV-S2S model when applying PT and PT-FT methods.

Keywords
Multi-purepose dam
Inflow
Deep learning
Transformer
Fine-tuning

딥러닝을 활용하여 유역 특성을 반영한 유량 예측 및 비교 연구가 주목받고 있다. 본 연구는 셀프 어텐션 메커니즘을 통해 대용량 데이터 훈련에 적합한 Transformer와 인코더-디코더(Encoder-Decoder) 구조를 가지는 LSTM-based multi-state-vector sequence-to-sequence(LSTM-MSV-S2S) 모형을 선정하여 유역정보(catchment attributes)를 고려할 수 있는 모형을 구축하였고 이를 토대로 국내 10개 다목적댐 유역의 유입량을 예측하였다. 본 연구에서 설계한 실험 구성은 단일유역-단일훈련(Single-basin Training, ST), 다수유역-단일훈련(Pretraining, PT), 사전학습-파인튜닝(Pretraining-Finetuning, PT-FT)의 세 가지 훈련 방법을 사용하였다. 모형의 입력 자료는 선정된 10가지 유역정보와 함께 기상 자료를 사용하였으며, 훈련 방법에 따른 유입량 예측 성능을 비교하였다. 그 결과, Transformer 모형은 PT와 PT-FT 방법에서 LSTM-MSV-S2S보다 우수한 성능을 보였으며, 특히 PT-FT 기법 적용 시 가장 높은 성능을 나타냈다. LSTM-MSV-S2S는 ST 방법에서는 Transformer보다 높은 성능을 보였으나, PT 및 PT-FT 방법에서는 낮은 성능을 보였다. 또한, 임베딩 레이어 활성화 값과 원본 유역정보를 군집화하여 모형의 유역 간 유사성 학습 여부를 분석하였다. Transformer는 활성화 벡터가 유사한 유역들에서 성능이 향상되었으며, 이는 사전에 학습된 다른 유역의 정보를 활용해 성능이 개선됨을 입증하였다. 본 연구는 다목적댐별 적합한 모형 및 훈련 방법을 비교하고, 국내 유역에 PT 및 PT-FT 방법을 적용한 딥러닝 모형 구축의 필요성을 제시하였다. 또한, PT 및 PT-FT 방법 적용 시 Transformer가 LSTM-MSV-S2S보다 성능이 더 우수하였다.

키워드
다목적댐
유입량
딥러닝
트랜스포머
파인튜닝
References
1

Addor, N., Newman, A.J., Mizukami, N., and Clark, M.P. (2017). "The CAMELS data set: Catchment attributes and meteorology for large-sample studies." Hydrology and Earth System Sciences, Vol. 21, No. 10, pp. 5293-5313.

10.5194/hess-21-5293-2017
2

Beck, H.E., Vergopolan, N., Pan, M., Levizzani, V., Van Dijk, A.I., Weedon, G.P., Brocca, L., Pappenberger, F., Huffman, G.J., and Wood, E.F. (2017). "Global-scale evaluation of 22 precipitation datasets using gauge observations and hydrological modeling." Hydrology and Earth System Sciences, Vol. 21, No. 12, pp. 6201-6217.

10.5194/hess-21-6201-2017
3

Blöschl, G., and Sivapalan, M. (1995). "Scale issues in hydrological modelling: A review." Hydrological Processes, Vol. 9, No. 3-4, pp. 251-290.

10.1002/hyp.3360090305
4

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., and Askell, A. (2020). "Language models are few-shot learners." Advances in Neural Information Processing Systems, Vol. 33, pp. 1877-1901.

5

Brunner, M.I., Slater, L., Tallaksen, L.M., and Clark, M. (2021). "Challenges in modeling and predicting floods and droughts: A review" Wiley Interdisciplinary Reviews: Water, Vol. 8, No. 3, e1520.

10.1002/wat2.1520
6

Chen, C., Hui, Q., Xie, W., Wan, S., Zhou, Y., and Pei, Q. (2021). "Convolutional Neural Networks for forecasting flood process in Internet-of-Things enabled smart city." Computer Networks, Vol. 186, 107744.

10.1016/j.comnet.2020.107744
7

Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). "Bert: Pre-training of deep bidirectional transformers for language understanding." arXiv preprint, arXiv:1810.04805.

8

Ding, Y., Zhu, Y., Feng, J., Zhang, P., and Cheng, Z. (2020). "Interpretable spatio-temporal attention LSTM model for flood forecasting." Neurocomputing, Vol. 403, pp. 348-359.

10.1016/j.neucom.2020.04.110
9

Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). "An image is worth 16x16 words: Transformers for image recognition at scale" arXiv preprint, arXiv:2010.11929.

10

Fang, K., Kifer, D., Lawson, K., Feng, D., and Shen, C. (2022). "The data synergy effects of time‐series deep learning models in hydrology." Water Resources Research, Vol. 58, No. 4, e2021WR029583.

10.1029/2021WR029583
11

Gao, S., Zhang, S., Huang, Y., Han, J., Luo, H., Zhang, Y., and Wang, G. (2022). "A new seq2seq architecture for hourly runoff prediction using historical rainfall and runoff as input." Journal of Hydrology, Vol. 612, 128099.

10.1016/j.jhydrol.2022.128099
12

Gupta, H., Perrin, C., Blöschl, G., Montanari, A., Kumar, R., Clark, M., and Andréassian, V. (2014). "Large-sample hydrology: A need to balance depth with breadth." Hydrology and Earth System Sciences, Vol. 18, No. 2, pp. 463-477.

10.5194/hess-18-463-2014
13

Gupta, H.V., Kling, H., Yilmaz, K.K., and Martinez, G.F. (2009). "Decomposition of the mean squared error and NSE performance criteria: Implications for improving hydrological modelling." Journal of Hydrology, Vol. 377, No. 1-2, pp. 80-91.

10.1016/j.jhydrol.2009.08.003
14

Hochreiter, S., and Schmidhuber, J. (1997). "Long short-term memory." Neural Computation, Vol. 9, No. 8, pp. 1735-1780.

10.1162/neco.1997.9.8.17359377276
15

Jeong, J., and Park, E. (2019). "Comparative applications of data-driven models representing water table fluctuations." Journal of Hydrology, Vol. 572, pp. 261-273.

10.1016/j.jhydrol.2019.02.051
16

Jun, H., and Lee, J. (2013). "A methodology for flood forecasting and warning based on the characteristic of observed water levels between upstream and downstream." Journal of the Korean Society of Hazard Mitigation, Vol. 13, No. 6, pp. 367-374.

10.9798/KOSHAM.2013.13.6.367
17

Jung, J., Mo, H., Lee, J., Yoo, Y., and Kim, H.S. (2021). "Flood stage forecasting at the Gurye-Gyo station in Sumjin River Using LSTM-based deep learning models." Journal of the Korean Society of Hazard Mitigation, Vol. 21, No. 3, pp. 193-201.

10.9798/KOSHAM.2021.21.3.193
18

Jung, S., Lee, D., and Lee, K. (2018). "Prediction of river water level using deep-learning open library." Journal of the Korean Society of Hazard Mitigation, Vol. 18, No. 1, pp. 1-11.

10.9798/KOSHAM.2018.18.1.1
19

Kao, I.-F., Zhou, Y., Chang, L.-C., and Chang, F.-J. (2020). "Exploring a Long Short-Term Memory based Encoder-Decoder framework for multi-step-ahead flood forecasting." Journal of Hydrology, Vol. 583, 124631.

10.1016/j.jhydrol.2020.124631
20

Kratzert, F., Klotz, D., Brenner, C., Schulz, K., and Herrnegger, M. (2018). "Rainfall-runoff modelling using Long Short-Term Memory (LSTM) networks." Hydrology and Earth System Sciences, Vol. 22, No. 11, pp. 6005-6022.

10.5194/hess-22-6005-2018
21

Kratzert, F., Klotz, D., Shalev, G., Klambauer, G., Hochreiter, S., and Nearing, G. (2019). "Towards learning universal, regional, and local hydrological behaviors via machine learning applied to large-sample datasets." Hydrology and Earth System Sciences, Vol. 23, No. 12, pp. 5089-5110.

10.5194/hess-23-5089-2019
22

Kratzert, F., Nearing, G., Addor, N., Erickson, T., Gauch, M., Gilon, O., Gudmundsson, L., Hassidim, A., Klotz, D., and Nevo, S. (2023). "Caravan-A global community dataset for large-sample hydrology." Scientific Data, Vol. 10, No. 1, 61.

10.1038/s41597-023-01975-w36717577PMC9887008
23

LeCun, Y., Bengio, Y., and Hinton, G. (2015). "Deep learning." Nature, Vol. 521, No. 7553, pp. 436-444.

10.1038/nature1453926017442
24

Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021). "Swin transformer: Hierarchical vision transformer using shifted windows." Proceedings of the IEEE/CVF International Conference on Computer Vision, Microsoft Research Asia, pp. 10012-10022.

10.1109/ICCV48922.2021.00986
25

Mok, J.-Y., Choi, J.-H., and Moon, Y.-I. (2020). "Prediction of multipurpose dam inflow using deep learning." Journal of Korea Water Resources Association, Vol. 53, No. 2, pp. 97-105.

26

Nash, J.E., and Sutcliffe, J.V. (1970). "River flow forecasting through conceptual models part I - A discussion of principles." Journal of Hydrology, Vol. 10, No. 3, pp. 282-290.

10.1016/0022-1694(70)90255-6
27

Nearing, G.S., Kratzert, F., Sampson, A.K., Pelissier, C.S., Klotz, D., Frame, J.M., Prieto, C., and Gupta, H.V. (2021). "What role does hydrological science play in the age of machine learning?" Water Resources Research, Vol. 57, No. 3, e2020WR028091.

10.1029/2020WR028091
28

Newman, A.J., Clark, M.P., Sampson, K., Wood, A., Hay, L.E., Bock, A., Viger, R.J., Blodgett, D., Brekke, L., and Arnold, J. (2015). "Development of a large-sample watershed-scale hydrometeorological data set for the contiguous USA: Data set characteristics and assessment of regional variability in hydrologic model performance" Hydrology and Earth System Sciences, Vol. 19, No. 1, pp. 209-223.

10.5194/hess-19-209-2015
29

Oudin, L., Andréassian, V., Perrin, C., Michel, C., and Le Moine, N. (2008). "Spatial proximity, physical similarity, regression and ungaged catchments: A comparison of regionalization approaches based on 913 French catchments." Water Resources Research, Vol. 44, No. 3, W03413.

10.1029/2007WR006240
30

Razavi, T., and Coulibaly, P. (2013). "Streamflow prediction in ungauged basins: review of regionalization methods" Journal of Hydrologic Engineering, Vol. 18, No. 8, pp. 958-975.

10.1061/(ASCE)HE.1943-5584.0000690
31

Schmidhuber, J. (2015). "Deep learning in neural networks: An overview" Neural Networks, Vol. 61, pp. 85-117.

10.1016/j.neunet.2014.09.00325462637
32

Shen, C., Laloy, E., Elshorbagy, A., Albert, A., Bales, J., Chang, F.-J., Ganguly, S., Hsu, K.-L., Kifer, D., and Fang, Z. (2018). "HESS Opinions: Incubating deep-learning-powered hydrologic science advances as a community." Hydrology and Earth System Sciences, Vol. 22, No. 11, pp. 5639-5656.

10.5194/hess-22-5639-2018
33

Sivapalan, M., Takeuchi, K., Franks, S.W., Gupta, V.K., Karambiri, H., Lakshmi, V., Liang, X., McDonnell, J.J., Mendiondo, E.M., O'Connell, P.E., Oki, T., Pomeroy, J.W., Schertzer, D., Uhlenbrook, S., and Zehe, E. (2003). "IAHS Decade on Predictions in Ungauged Basins (PUB), 2003-2012: Shaping an exciting future for the hydrological sciences." Hydrological Sciences Journal, Vol. 48, No. 6, pp. 857-880.

10.1623/hysj.48.6.857.51421
34

Tuli, S., Casale, G., and Jennings, N.R. (2022). "Tranad: Deep transformer networks for anomaly detection in multivariate time series data." arXiv preprint, arXiv:2201.07284.

10.14778/3514061.3514067
35

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017). "Attention is all you need." Advances in Neural Information Processing Systems, Vol. 30, Long Beach, CA, U.S.

36

Wen, Q., He, K., Sun, L., Zhang, Y., Ke, M., and Xu, H. (2021). "RobustPeriod: Robust time-frequency mining for multiple periodicity detection." Proceedings of the 2021 International Conference on Management of Data, China, pp. 2328-2337.

10.1145/3448016.3452779
37

Wu, H., Xu, J., Wang, J., and Long, M. (2021). "Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting." Advances in Neural Information Processing Systems, Vol. 34, pp. 22419-22430.

38

Xu, J., Wu, H., Wang, J., and Long, M. (2021). "Anomaly transformer: Time series anomaly detection with association discrepancy" arXiv preprint, arXiv:2110.02642.

39

Xu, Y., Lin, K., Hu, C., Wang, S., Wu, Q., Zhang, L., and Ran, G. (2023). "Deep transfer learning based on transformer for flood forecasting in data-sparse basins." Journal of Hydrology, Vol. 625, 129956.

10.1016/j.jhydrol.2023.129956
40

Yang, C.-H.H., Tsai, Y.-Y., and Chen, P.-Y. (2021). "Voice2series: Reprogramming acoustic models for time series classification." International Conference on Machine Learning, PMLR, pp. 11808-11819.

41

Yin, H., Guo, Z., Zhang, X., Chen, J., and Zhang, Y. (2022). "RR-Former: Rainfall-runoff modeling based on Transformer." Journal of Hydrology, Vol. 609, 127781.

10.1016/j.jhydrol.2022.127781
42

Yin, H., Zhang, X., Wang, F., Zhang, Y., Xia, R., and Jin, J. (2021). "Rainfall-runoff modeling using LSTM-based multi-state-vector sequence-to-sequence model." Journal of Hydrology, Vol. 598, 126378.

10.1016/j.jhydrol.2021.126378
43

Yin, H., Zhu, W., Zhang, X., Xing, Y., Xia, R., Liu, J., and Zhang, Y. (2023). "Runoff predictions in new-gauged basins using two transformer-based models" Journal of Hydrology, Vol. 622, 129684.

10.1016/j.jhydrol.2023.129684
44

Zhang, Y., Chiew, F. H., Li, M., and Post, D. (2018). "Predicting runoff signatures using regression and hydrological modeling approaches." Water Resources Research, Vol. 54, No. 10, pp. 7859-7878.

10.1029/2018WR023325
45

Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., and Zhang, W. (2021). "Informer: Beyond efficient transformer for long sequence time-series forecasting." Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, Canada, Vol. 35, No. 12, pp. 11106-11115.

10.1609/aaai.v35i12.17325
Information
  • Publisher :KOREA WATER RESOURECES ASSOCIATION
  • Publisher(Ko) :한국수자원학회
  • Journal Title :Journal of Korea Water Resources Association
  • Journal Title(Ko) :한국수자원학회 논문집
  • Volume : 57
  • No :7
  • Pages :437-449
  • Received Date : 2024-04-17
  • Revised Date : 2024-06-11
  • Accepted Date : 2024-06-12
  • DOI :https://doi.org/10.3741/JKWRA.2024.57.7.437
Journal Informaiton Journal of Korea Water Resources Association Journal of Korea Water Resources Association
Online Submission
Archives
  • scopus
  • KCI-수자원
  • NRF
  • KOFST
  • KISTI Current Status
  • KISTI Cited-by
  • crosscheck
  • orcid
  • open access
  • ccl
Journal Informaiton Journal Informaiton - close