All Issue

2024 Vol.57, Issue 3 Preview Page

Research Article

Comparative analysis of wavelet transform and machine learning approaches for noise reduction in water level data

웨이블릿 변환과 기계 학습 접근법을 이용한 수위 데이터의 노이즈 제거 비교 분석

Yukwan Hwanga
,
Kyoung Jae Limb
,
Jonggun Kimc
,
Minhwan Shind
,
Youn Shik Parke
,
Yongchul Shinf
,
Bongjun Jig*
황 유관a
,
임 경재b
,
김 종건c
,
신 민환d
,
박 윤식e
,
신 용철f
,
지 봉준g*
aGraduate Student, Regional Infrastructure Engineering, Kangwon National University, Chuncheon, Korea
bProfessor, Department of Regional Infrastructure Engineering, Kangwon National University, Chuncheon, Korea
cAssistant Professor, Department of Regional Infrastructure Engineering, Kangwon National University, Chuncheon, Korea
dCEO, EM Research Institute, Chuncheon, Korea
eProfessor, Department of Regional Construction Engineering, Kongju National University, Yesan, Korea
fAssociate Professor, Department of Agricultural Civil Engineering, Kyungpook National University, Daegu, Korea
gAssistant Professor, Department of Regional Infrastructure Engineering, Kangwon National University, Chuncheon, Korea
a강원대학교 지역건설공학과 석사과정
b강원대학교 지역건설공학과 교수
c강원대학교 지역건설공학과 조교수
d이엠연구소 대표
e공주대학교 지역건설공학과 교수
f경북대학교 농업토목공학전공 부교수
g강원대학교 지역건설공학과 조교수
*Corresponding Author
31 March 2024. pp. 209-223
PDF XML Citation
Abstract

In the context of the fourth industrial revolution, data-driven decision-making has increasingly become pivotal. However, the integrity of data analysis is compromised if data quality is not adequately ensured, potentially leading to biased interpretations. This is particularly critical for water level data, essential for water resource management, which often encounters quality issues such as missing values, spikes, and noise. This study addresses the challenge of noise-induced data quality deterioration, which complicates trend analysis and may produce anomalous outliers. To mitigate this issue, we propose a noise removal strategy employing Wavelet Transform, a technique renowned for its efficacy in signal processing and noise elimination. The advantage of Wavelet Transform lies in its operational efficiency - it reduces both time and costs as it obviates the need for acquiring the true values of collected data. This study conducted a comparative performance evaluation between our Wavelet Transform-based approach and the Denoising Autoencoder, a prominent machine learning method for noise reduction.. The findings demonstrate that the Coiflets wavelet function outperforms the Denoising Autoencoder across various metrics, including Mean Absolute Error (MAE), Mean Absolute Percentage Error (MAPE), and Mean Squared Error (MSE). The superiority of the Coiflets function suggests that selecting an appropriate wavelet function tailored to the specific application environment can effectively address data quality issues caused by noise. This study underscores the potential of Wavelet Transform as a robust tool for enhancing the quality of water level data, thereby contributing to the reliability of water resource management decisions.

Keywords
Water level data
Denoising Autoencoder
Wavelet Transform
Coiflets function

4차 산업혁명 시대에 접어들어 데이터 기반의 의사결정이 보편화되고 있다. 하지만 데이터 품질이 확보되지 않은 채 수행되는 데이터 분석은 왜곡된 결과를 낳을 가능성이 존재한다. 수자원 관리의 기초가 되는 수위 데이터도 마찬가지로 결측, 스파이크, 잡음 등 다양한 품질 문제를 가진다. 본 연구에서는 잡음으로 인해 발생하는 데이터 품질 문제를 해결하고자 하였다. 잡음은 데이터의 트렌드 분석을 어렵게 하고 비정상적인 이상치를 생성할 가능성이 있다. 본 연구는 이러한 문제를 해결하기 위해 Wavelet Transform을 이용한 잡음 제거 접근 방안을 제안한다. Wavelet Transform은 신호처리에 주로 사용되는 방법으로 잡음 제거에 효과적인 것으로 알려져 있으며 수집된 데이터의 정답 데이터(True value) 수집을 요구하지 않으므로 시간과 비용을 줄일 수 있다는 점에서 적용이 용이한 편이다. 본 연구는 Wavelet Transform의 성능 평가를 위해 대표적인 머신러닝 기반 잡음 제거 방법인 Denoising Autoencoder와 성능 비교를 수행하였다. 그 결과 Wavelet Transform 중 Coiflets 함수는, Denoising Autoencoder에 비해 Mean Absolute Error, Mean Absolute Percentage Error, Mean Squared Error 등 모든 측면에서 우수한 성능을 보이는 것으로 나타났다. 이러한 결과는 환경에 맞는 적절한 웨이블릿 함수의 선택을 통한 잡음 문제를 효과적으로 해결할 수 있음을 시사한다. 본 연구는 수위 데이터의 품질을 향상시켜 수자원 관리 결정의 신뢰성에 기여하는 강력한 도구로서 Wavelet Transform의 잠재력을 확인한 의의가 있다.

키워드
수위 데이터
잡음제거 오토인코더
웨이블릿 변환
코이플릿 함수
References
1
Alan, T.M. (2010). "Advances in location modeling: GIS linkages and contributions." Journal of Geographical Systems, Vol. 12, pp. 335-354. 10.1007/s10109-009-0105-9
2
Anthony, J.M., Robert. N.F., Yang. L., Nathaniel. A.W., and Steven. D.B. (2004). "An introduction to decision tree modeling." Journal of Chemomertics, Wiley, Vol. 1, No.18, pp. 275-285. 10.1002/cem.873
3
Baldi, P. (2012). "Autoencoders, unsupervised learning, and deep architectures." In Proceedings of ICML Workshop on Unsupervised and Transfer Learning, PMLR, Scotland, pp. 37-49.
4
Baliunas, S., Frick, P., Sokoloff, D., and Soon, W. (1997). "Time scales and trends in the central England temperature data (1659- 1990): A wavelet analysis." Geophysical Research Letters, Vol. 24, No. 11, pp. 1351-1354. 10.1029/97GL01184
5
Chan, T.F., and Shen, J. (2005). Image processing and analysis: variational, PDE, wavelet, and stochastic methods. Society for Industrial and Applied Mathematics, SIAM, Philadelphia, PA, U.S.
6
Chen, Z., Yeo, C.K., Lee, B.S., and Lau, C.T. (2018). "Autoencoder- based network anomaly detection." Wireless Telecommunications Symposium, IEEE, Phoenix, AZ, U.S., pp. 1-5. 10.1109/WTS.2018.8363930
7
Cho, J.H., Choi, J.H, Kim, Y.S., Yeon. J.H., Lee. S.J., and Ji, B.J. (2022). "A study on the prediction of slope collapse based on LSTM Autoencoder." Gangwon Agricultural Life and Environment Research, Vol. 34, No. 3, pp. 345-353.
8
Choi, J., Lim, K.J., and Ji, B. (2023). "Robust imputation method with context-aware voting ensemble model for management of water-quality data." Water Research, Vol. 243, 120369. 10.1016/j.watres.2023.12036937499538
9
Chun-Lin, L. (2010). A tutorial of the wavelet transform. NTUEE, Taiwan.
10
Creswell, A., and Bharath, A.A. (2018). "Denoising adversarial autoencoders." IEEE Transactions on Neural Networks and Learning Systems, Vol. 30, No. 4, pp. 968-984. 10.1109/TNNLS.2018.285273830130236
11
Davis, P.J. (1975). Interpolation and approximation. Courier Corporation, MA, U.S.
12
De Moortel, I., Munday, S.A., and Hood, A.W. (2004). "Wavelet analysis: The effect of varying basic wavelet parameters." Solar Physics, Vol. 222, No. 2, pp. 203-228. 10.1023/B:SOLA.0000043578.01201.2d
13
Escobar, F., Hunter, G., Bishop, I., and Zerger, A. (2008). Introduction to GIS, accessed 13 December 2023, <http://www.sliunimelb.edu.au/gisweb/unimelb.edu.au/gisweb/.
14
Eva, O., and Oskar, O. (2012). "Forecasting using simple exponential smoothing method." Acta Electrotechnica et Informatica, ISSN, Vol. 12, No. 3, pp. 62-66. 10.2478/v10198-012-0034-2
15
Fourier, J.B.J. (1822). Théorie analytique de la chaleur. Firmin Didot, Paris.
16
Głowacz, A. (2014). "Diagnostics of direct current machine based on analysis of acoustic signals with the use of symlet wavelet transform and modified classifier based on words." Eksploatacja i Niezawodność, Vol. 16, No. 4, pp. 554-558.
17
Graham, J.W. (2009). "Missing data analysis: Making it work in the real world." Annual Review of Psychology, Vol. 60, pp. 549- 576. 10.1146/annurev.psych.58.110405.08553018652544
18
Heil, C.E., and Walnut, D.F. (1989). "Continuous and discrete wavelet transforms." SIAM Review, Vol. 31, No. 4, pp. 628-666. 10.1137/1031129
19
Hinton, G.E., and Salakhutdinov, R.R. (2006). "Reducing the dimensionality of data with neural networks." Science, Vol. 313, No. 5786, pp. 504-507. 10.1126/science.112764716873662
20
Jo, H.Y. (2009). "A study of constructiong index fund using wavelet analysis." The Journal of Information Systems, Vol. 18, No. 3, pp. 351-373.
21
Jonathan, M.L., and Sofia. C.O. (2010). "On the analytic wavelet transform." IEEE Transactions on Information Theory, IEEE, Vol. 56, No. 8, pp. 4135-4156. 10.1109/TIT.2010.2050935
22
Kang, S.M., Park, M.J., Kim, S.H., and Kim, S.J. (2007). "A study on the mitigation of inundation damage using flood inundation analysis model FLUMEN." Academic Presentation of the Korean Water Resources Association, pp. 815-819.
23
Kim, D.K. (2019). "Introduction to machine learning." Korean Society Of Facilities Engineering. Vol. 48, pp. 80-82.
24
Kim, H.L., Han. K.Y., and Lee. J.Y. (2020). "Predicition of urban flood extent by lstm model and logistic regression." Journal of the Korean Society of Civil Engineers, ISSN, Vol. 40, No. 3, pp. 273-283.
25
Ko, J.P. (2005). "Solving multi-class problem using support vector machines." Journal of Information Science Society, Vol. 32, No. 12, pp. 1260-1270.
26
Kuo, H.H. (2018). White noise distribution theory. CRC press, Boca Raton, FL, U.S. 10.1201/9780203733813-13
27
Kwon, H.S., and Lee, M.H. (2023). "Stock index forecasting using combined model of wavelet transform lstm and multi-head attention." Vol. 40, No. 2, pp. 97-112. 10.7737/KMSR.2023.40.2.097
28
Lee, D.T., and Yamamoto, A. (1994). "Wavelet analysis: theory and applications." Hewlett Packard journal, Vol. 45, p. 44. 10.1177/1742271X9400200123
29
Lee, M.H., Park. J.S., and Won. Y.S. (2022). "Future of flood forecasting considering flood damage phenomenon." Journal of the Korean Meteorological Society's Academic Conference, p. 226-226.
30
Lewis, C.D. (1982). Industrial and business forecasting methods. Butterworths, London.
31
Morlet, J., and Grossmann, A. (1984). "Decomposition of hardy function into square integrable wavelets of constant shape." SIAM Journal on Mathemactical Analysis, Vol. 15, No. 4, pp. 723-736. 10.1137/0515056
32
Öberg, G., Metson, G. S., Kuwayama, Y., and A. Conrad, S. (2020). "Conventional sewer systems are too time-consuming, costly and inflexible to meet the challenges of the 21st century." Sustainability, Vol. 12, No. 16, 6518. 10.3390/su12166518
33
Oh, J.h., and Seo, K.D. (2011). "Analysis of the intensity and duration of typhoons on the Korean Peninsula using wavelet analysis." Collections of Academic Conferences of the Korean Chosun Association, pp. 2040-2043.
34
Reilly, J., and David, S. (2000). "Irreversibility, uncertainty, and learning: Portraits of adaptation to long-term climate change." Climatic Change, Vol. 45, No. 1, pp. 253-278. 10.1023/A:1005669807945
35
Sifuzzaman, M., Islam, M.R., and Ali, M.Z. (2009). "Application of wavelet transform and its advantages compared to fourier transform." Journal of Physical Sciences, ISSN, Vol. 13, pp. 121-134.
36
Sridhar, S., Kumar, P.R., and Ramanaiah, K.V. (2013). "Daubechies, coiflet, symlet and biorthogonal wavelet transforms for effectual image compression and analysis of fidelity measures." I-Manager's Journal on Digital Signal Processing, Vol. 1, No. 2, pp. 1-9. 10.26634/jdp.1.2.2326
37
Umberto, M. (2022). "An introduction to autoencoder." arXiv Preprint, arXiv:2201.03898.
38
Yu, Y., Si, X., Hu, C., and Zhang, J. (2019). "A review of recurrent neural networks: LSTM cells and network architectures." Neural Computation, Vol. 31, No. 7, pp. 1235-1270. 10.1162/neco_a_0119931113301
Information
  • Publisher :KOREA WATER RESOURECES ASSOCIATION
  • Publisher(Ko) :한국수자원학회
  • Journal Title :Journal of Korea Water Resources Association
  • Journal Title(Ko) :한국수자원학회 논문집
  • Volume : 57
  • No :3
  • Pages :209-223
  • Received Date : 2024-02-08
  • Revised Date : 2024-03-06
  • Accepted Date : 2024-03-07
  • DOI :https://doi.org/10.3741/JKWRA.2024.57.3.209
Journal Informaiton Journal of Korea Water Resources Association Journal of Korea Water Resources Association
Online Submission
Archives
  • scopus
  • KCI-수자원
  • NRF
  • KOFST
  • KISTI Current Status
  • KISTI Cited-by
  • crosscheck
  • orcid
  • open access
  • ccl
Journal Informaiton Journal Informaiton - close