Rainfall image DB construction for rainfall intensity estimation from CCTV videos: focusing on experimental data in a climatic environment chamber

Jongyun Byun; Changhyun Jun; Hyeon-Joon Kim; Jae Joon Lee; Hunil Park; Jinwook Lee

doi:10.3741/JKWRA.2023.56.6.403

All Issue

2023 Vol.56, Issue 6 Preview Page Next Page

Research Article

Rainfall image DB construction for rainfall intensity estimation from CCTV videos: focusing on experimental data in a climatic environment chamber CCTV 영상 기반 강우강도 산정을 위한 실환경 실험 자료 중심 적정 강우 이미지 DB 구축 방법론 개발

30 June 2023. pp. 403-417

PDF XML

Abstract

In this research, a methodology was developed for constructing an appropriate rainfall image database for estimating rainfall intensity based on CCTV video. The database was constructed in the Large-Scale Climate Environment Chamber of the Korea Conformity Laboratories, which can control variables with high irregularity and variability in real environments. 1,728 scenarios were designed under five different experimental conditions. 36 scenarios and a total of 97,200 frames were selected. Rain streaks were extracted using the k-nearest neighbor algorithm by calculating the difference between each image and the background. To prevent overfitting, data with pixel values greater than set threshold, compared to the average pixel value for each image, were selected. The area with maximum pixel variability was determined by shifting with every 10 pixels and set as a representative area (180×180) for the original image. After re-transforming to 120×120 size as an input data for convolutional neural networks model, image augmentation was progressed under unified shooting conditions. 92% of the data showed within the 10% absolute range of PBIAS. It is clear that the final results in this study have the potential to enhance the accuracy and efficacy of existing real-world CCTV systems with transfer learning.

Keywords

CCTV

Rainfall image

Rainfall estimation

Environment chamber

DB construction

CNN

본 연구에서는 CCTV 영상 기반 강우강도 산정 시 필수적으로 요구되는 적정 강우 이미지 DB를 구축하기 위한 방법론을 개발하였다. 먼저, 실환경에서 불규칙적이고 높은 변동성을 보일 수 있는 변수들(바람으로 인한 빗줄기의 변동성, 녹화 환경에서 포함되는 움직이는 객체, 렌즈 위의 흐림 현상 등)에 대한 통제가 가능한 한국건설생활환경시험연구원 내 기후환경시험실에서 CCTV 영상 DB를 구축하였다. 서로 다른 5개의 실험 조건을 고려하여 이상적 환경에서 총 1,728개의 시나리오를 구성하였다. 본 연구에서는 1,920×1,080 사이즈의 30 fps (frame per second) 영상 36개에 대하여 프레임 분할을 진행하였으며, 총 97,200개의 이미지를 사용하였다. 이후, k-최근접 이웃 알고리즘을 기반으로 산정된 최종 배경과 각 이미지와의 차이를 계산하여 빗줄기 이미지를 분리하였다. 과적합 방지를 위해 각 이미지에 대한 평균 픽셀 값을 계산하고, 설정한 픽셀 임계치보다 큰 자료를 선별하였다. 180×180 사이즈로의 재구성을 위해서 관심영역을 설정하고 10 Pixel 단위로 이동을 진행하여 픽셀 변동성이 최대가 되는 영역을 산정하였다. 합성곱 신경망 모델의 훈련을 위해서 120×120 사이즈로 재변환하고 과적합 방지를 위해 이미지 증강 과정을 거쳤다. 그 결과, 이미지 기반 강우강도 합성곱 신경망 모델을 통해 산정된 결과값과 우량계에서 취득된 강우자료가 전반적으로 유사한 양상을 보였으며, 모든 강우강도 실험 조건에 대해서 약 92%의 데이터의 PBIAS (percent bias)가 절댓값 범위 10% 이내에 해당하였다. 본 연구의 결과물과 전이학습 등의 방법을 연계하여 기존 실환경 CCTV의 한계점을 개선할 수 있을 것으로 기대된다.

키워드

CCTV

강우 이미지

강우강도 산정

실환경 실험

DB 구축

CNN

References

Abdelraouf, A., Abdel-Aty, M., and Wu, Y. (2022). "Using vision transformers for spatial-context-aware rain and road surface condition detection on freeways." IEEE Transactions on Intelligent Transportation Systems, Vol. 23, No. 10, pp. 18546-18556. 10.1109/TITS.2022.3150715

Abraham, M.T., Satyam, N., Pradhan, B., and Alamri, A.M. (2020). "IoT-based geotechnical monitoring of unstable slopes for landslide early warning in the Darjeeling Himalayas." Sensors, Vol. 20, No. 9, 2611. 10.3390/s2009261132375265PMC7273224

Abudi, I., Carmi, G., and Berliner, P. (2012). "Rainfall simulator for field runoff studies." Journal of Hydrology, Vol. 454, pp. 76-81. 10.1016/j.jhydrol.2012.05.056

Ajith, J.B., Manimegalai, R., and Ilayaraja, V. (2020). "An IoT based smart water quality monitoring system using cloud." In 2020 International Conference on Emerging Trends in Information Technology and Engineering (ic-ETITE), IEEE, Vellore, India, pp. 1-7. 10.1109/ic-ETITE47903.2020.450

Allamano, P., Croci, A., and Laio, F. (2015). "Toward the camera rain gauge." Water Resources Research, Vol. 51, No. 3, 1744-1757. 10.1002/2014WR016298

Amale, O., and Patil, R. (2019). "IOT Based rainfall monitoring system using WSN enabled architecture." In 2019 3rd International Conference on Computing Methodologies and Communication (ICCMC), Erode, India, IEEE, pp. 789-791. 10.1109/ICCMC.2019.8819721

Barbosa, J.H.S., Fernandes, A., Lima, A., and Assis, L. (2019). "The influence of spatial discretization on HEC-HMS modelling: A case study." International Journal of Hydrology, Vol. 3, pp. 442-449. 10.15406/ijh.2019.03.00209

Byun, J., Jun, C., Kim, J., Cha, J., and Narimani, R. (2023). "Deep Learning-based Rainfall Prediction using Cloud Image Analysis." IEEE Transactions on Geoscience and Remote Sensing, Vol. 61, pp. 1-11. 10.1109/TGRS.2023.3263872

Cardoso de Salis, H.H., Monteiro da Costa, A., Moreira Vianna, J. H., Azeneth Schuler, M., Künne, A., Sanches Fernandes, L.F., and Leal Pacheco, F.A. (2019). "Hydrologic modeling for sustainable water resources management in urbanized karst areas." International Journal of Environmental Research and Public Health, Vol. 16, No. 14, 2542. 10.3390/ijerph1614254231315302PMC6678514

Dawson, C.W., Abrahart, R.J., and See, L.M. (2007). "HydroTest: A web-based toolbox of evaluation metrics for the standardised assessment of hydrological forecasts." Environmental Modelling and Software, Vol. 22, No. 7, pp. 1034-1052. 10.1016/j.envsoft.2006.06.008

Dong, R., Liao, J., Li, B., Zhou, H., and Crookes, D. (2017). "Measurements of rainfall rates from videos." 2017 10th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI), IEEE, Shanghai, China, pp. 1-9. 10.1109/CISP-BMEI.2017.8302066

Fu, X., Huang, J., Zeng, D., Huang, Y., Ding, X., and Paisley, J. (2017). "Removing rain from single images via a deep detail network." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, U.S., pp. 3855-3863. 10.1109/CVPR.2017.186

Garg, K., and Nayar, S.K. (2004). "Detection and removal of rain from videos." Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, IEEE, Washington, DC, U.S., Vol. 1, pp. I-I. 10.1109/CVPR.2004.1315077

Guo, B., Han, Q., Chen, H., Shangguan, L., Zhou, Z., and Yu, Z. (2017). "The emergence of visual crowdsensing: Challenges and opportunities." IEEE Communications Surveys and Tutorials, Vol. 19, No. 4, pp. 2526-2543. 10.1109/COMST.2017.2726686

Haidar, A., and Verma, B. (2018). "Monthly rainfall forecasting using one-dimensional deep convolutional neural network." IEEE Access, Vol. 6, pp. 69053-69063. 10.1109/ACCESS.2018.2880044

Hakim, A.L., and Dewi, R. (2021). "Automatic rain detection system based on digital images of CCTV cameras using convolutional neural network method." IOP Conference Series: Earth and Environmental Science, Vol. 893, No. 1, 012048. 10.1088/1755-1315/893/1/012048

Han, K., and Xiang, X. (2020). "Decomposed cyclegan for single image deraining with unpaired data." ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, Barcelona, Spain, pp. 1828-1832. 10.1109/ICASSP40776.2020.9053123

Humphry, J.B., Daniel, T.C., Edwards, D.R., and Sharpley, A.N. (2002). "A portable rainfall simulator for plot-scale runoff studies." Applied Engineering in Agriculture, Vol. 18, No. 2, 199.

Jiang, S., Babovic, V., Zheng, Y., and Xiong, J. (2019). "Advancing opportunistic sensing in hydrology: A novel approach to measuring rainfall with ordinary surveillance cameras." Water Resources Research, Vol. 55, No. 4, pp. 3004-3027. 10.1029/2018WR024480

Lee, J., Byun, J., Baik, J., Jun, C., and Kim, H.-J. (2023). "Estimation of raindrop size distribution and rain rate with infrared surveillance camera in dark conditions." Atmospheric Measurement Techniques, Vol. 16, No. 3, pp. 707-725. 10.5194/amt-16-707-2023

Lee, S.H., Jeon, J.H., Choi, D.Y., Park, J.M., and Song, B.C. (2022). "Image based rainfall amount estimation for auto-wiping of vehicles." Neural Computing and Applications, Vol. 34, pp. 15543-15554. 10.1007/s00521-022-07269-3

Li, M., Xie, Q., Zhao, Q., Wei, W., Gu, S., Tao, J., and Meng, D. (2018a). "Video rain streak removal by multiscale convolutional sparse coding." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, U.S., pp. 6644-6653. 10.1109/CVPR.2018.0069530398056

Li, Z., Hyeon, J., and Choi, H. J. (2018b). "Rainfall recognition from road surveillance videos using TSN." Journal of Korean Society for Atmospheric Environment, Vol. 34, No. 5, pp. 735-747. 10.5572/KOSAE.2018.34.5.735

Liu, T., Xu, M., and Wang, Z. (2019). "Removing rain in videos: A large-scale database and a two-stream ConvLSTM approach." In 2019 IEEE International Conference on Multimedia and Expo (ICME), IEEE, Shanghai, China, pp. 664-669. 10.1109/ICME.2019.00120

Notarangelo, N.M., Hirano, K., Albano, R., and Sole, A. (2021). "Transfer learning with convolutional neural networks for rainfall detection in single images." Water, Vol. 13, No. 5, 588. 10.3390/w13050588

Rani, D.S., Jayalakshmi, G.N., and Baligar, V.P. (2020). "Low cost IoT based flood monitoring system using machine learning and neural networks: flood alerting and rainfall prediction." In 2020 2nd International Conference on Innovative Mechanisms for Industry Applications (ICIMIA), IEEE, Bangalore, India, pp. 261-267. 10.1109/ICIMIA48430.2020.9074928

Regmi, T.P., and Thompson, A.L. (2000). "Rainfall simulator for laboratory studies." Applied Engineering in Agriculture, Vol. 16, No. 6, 641.

Sadeghi, M., Asanjan, A.A., Faridzad, M., Nguyen, P., Hsu, K., Sorooshian, S., and Braithwaite, D. (2019). "PERSIANN-CNN: Precipitation estimation from remotely sensed information using artificial neural networks-convolutional neural networks." Journal of Hydrometeorology, Vol. 20, No. 12, pp. 2273-2289. 10.1175/JHM-D-19-0110.1

Santos, S.C., Firmino, R.M., Mattos, D.M., and Medeiros, D.S. (2020). "An IoT rainfall monitoring application based on wireless communication technologies." In 2020 4th Conference on Cloud and Internet of Things (CIoT), IEEE, Niterói, Brazil, pp. 53-56. 10.1109/CIoT50422.2020.9244293

Singh, V.K., Kumar, M.M., Yuvaraj, J., Rubeshkumar, T., Kumar, S., and Padmapriya, S. (2022). "IoT based rainfall surveillance system with sensor integrated infrastructure." 2022 7th International Conference on Communication and Electronics Systems (ICCES), IEEE, Coimbatore, India, pp. 468-472.

Thompson, A.L., and James, L.G. (1985). "Water droplet impact and its effect on infiltration." Transactions of the ASAE, Vol. 28, No. 5, pp. 1506-1510. 10.13031/2013.32468

Tu, Z., Talebi, H., Zhang, H., Yang, F., Milanfar, P., Bovik, A., and Li, Y. (2022). "Maxim: Multi-axis mlp for image processing." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, U.S., pp. 5769-5780. 10.1109/CVPR52688.2022.00568

Wang, T., Yang, X., Xu, K., Chen, S., Zhang, Q., and Lau, R.W. (2019). "Spatial attentive single-image deraining with a high quality real rain dataset." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, U.S., pp. 12270-12279. 10.1109/CVPR.2019.01255

Wei, Y., Zhang, Z., Wang, Y., Xu, M., Yang, Y., Yan, S., and Wang, M. (2021). "Deraincyclegan: Rain attentive cyclegan for single image deraining and rainmaking." IEEE Transactions on Image Processing, Vol. 30, pp. 4788-4801. 10.1109/TIP.2021.307480433929960

Xue, M., Hang, R., Liu, Q., Yuan, X.T., and Lu, X. (2021). "CNN-based near-real-time precipitation estimation from Fengyun-2 satellite over Xinjiang, China." Atmospheric Research, Vol. 250, 105337. 10.1016/j.atmosres.2020.105337

Yang, W., Tan, R.T., Feng, J., Liu, J., Guo, Z., and Yan, S. (2017). "Deep joint rain detection and removal from a single image." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, U.S., pp. 1357-1366. 10.1109/CVPR.2017.183

Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., and Yang, M.H. (2022). "Restormer: Efficient transformer for high-resolution image restoration." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, U.S., pp. 5728-5739. 10.1109/CVPR52688.2022.00564

Zen, R., Arsa, D.M.S., Zhang, R., Er, N.A.S., and Bressan, S. (2019). "Rainfall estimation from traffic cameras." Database and Expert Systems Applications: 30th International Conference, Linz, Austria, pp. 18-32. 10.1007/978-3-030-27615-7_2

Zhang, H., and Patel, V.M. (2018). "Density-aware single image de-raining using a multi-stream dense network." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, U.S., pp. 695-704. 10.1109/CVPR.2018.00079

Zhu, H., Peng, X., Zhou, J.T., Yang, S., Chanderasekh, V., Li, L., and Lim, J.H. (2019). "Singe image rain removal with unpaired information: A differentiable programming perspective." Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33, No. 01, pp. 9332-9339. 10.1609/aaai.v33i01.33019332

Information

Publisher :KOREA WATER RESOURECES ASSOCIATION
Publisher(Ko) :한국수자원학회
Journal Title :Journal of Korea Water Resources Association
Journal Title(Ko) :한국수자원학회 논문집
Volume : 56
No :6
Pages :403-417
Received Date : 2023-04-12
Revised Date : 2023-06-14
Accepted Date : 2023-06-14
DOI :https://doi.org/10.3741/JKWRA.2023.56.6.403

Journal of Korea Water Resources Association ISSN:2799-8746(Print) 2799-8754(Online) 한국수자원학회 논문집

All Issue