A study on the development of quality control algorithm for internet of things (IoT) urban weather observed data based on machine learning

Seung Woon Lee; Seung Kwon Jung

doi:10.3741/JKWRA.2021.54.S-1.1071

All Issue

2021 Vol.54, Issue 12S Preview Page Next Page

Research Article

A study on the development of quality control algorithm for internet of things (IoT) urban weather observed data based on machine learning 머신러닝기반의 사물인터넷 도시기상 관측자료 품질검사 알고리즘 개발에 관한 연구

31 December 2021. pp. 1071-1081

PDF XML

Abstract

In addition to the current quality control procedures for the weather observation performed by the Korea Meteorological Administration (KMA), this study proposes quality inspection standards for Internet of Things (IoT) urban weather observed data based on machine learning that can be used in smart cities of the future. To this end, in order to confirm whether the standards currently set based on ASOS (Automated Synoptic Observing System) and AWS (Automatic Weather System) are suitable for urban weather, usability was verified based on SKT AWS data installed in Seoul, and a machine learning-based quality control algorithm was finally proposed in consideration of the IoT’s own data’s features. As for the quality control algorithm, missing value test, value pattern test, sufficient data test, statistical range abnormality test, time value abnormality test, spatial value abnormality test were performed first. After that, physical limit test, stage test, climate range test, and internal consistency test, which are QC for suggested by the KMA, were performed. To verify the proposed algorithm, it was applied to the actual IoT urban weather observed data to the weather station located in Songdo, Incheon. Through this, it is possible to identify defects that IoT devices can have that could not be identified by the existing KMA’s QC and a quality control algorithm for IoT weather observation devices to be installed in smart cities of future is proposed.

Keywords

Internet of things

Weather observation

Machine learning

Quality control

Smart city

본 연구에서는 기상청에서 수행하는 기존의 기상 관측에 대한 품질관리 절차 이외에 향후 스마트시티 등에서 활용될 수 있는 머신러닝 기반의 Internet of Things (IoT) 도시기상 관측 자료에 대한 품질검사 기준을 제안한다. 현재 기상청에서 종관기상관측(Automated Synoptic Observing System, ASOS)과 방재기상관측(Automatic Weather System, AWS) 기반으로 설정한 기준이 도시기상에 적합한지 확인하기 위하여 서울시에 설치된 SKT AWS 자료를 기반으로 사용성을 검증하였고, IoT 자체의 데이터가 가지는 특성을 고려하여 최종적으로 머신러닝 기반의 품질검사 알고리즘을 제안하였다. 품질검사 방법으로는 IoT 기기 자체에 대한 결측값 검사, 값 패턴 검사, 충분 데이터 검사, 통계적 범위 이상 검사, 시간값 이상 검사, 공간값 이상 검사를 먼저 수행하고, 기상청에서 제시하고 있는 기상 관측에 대한 품질검사인 물리한계검사, 단계검사, 지속성 검사, 기후범위 검사, 내적 일치성 검사를 5가지 기상요소에 대하여 각각 수행하였다. 제안한 알고리즘의 검증을 위하여 인천광역시 송도에 위치한 관측소에 실제 IoT 도시기상관측 데이터에 이를 적용하였다. 이를 통해 기존의 기상청 QC로는 확인할 수 없었던 IoT 기기가 가질 수 있는 결함을 확인할 수 있고, 알고리즘에 대한 검증을 진행하여 향후 스마트시티에 설치될 IoT 기상관측기기에 대한 품질검사 방법을 제안한다.

키워드

사물인터넷

기상관측

머신러닝

품질검사

스마트시티

References

Ahn, J.H. (2009). "Development of embedded weather station using ethernet." Power Electronics Annual Conference, KIPE, pp. 13-15. (in Korean)

Chae, J.H., Park, M.S., and Choi, Y.J. (2014). "The WISE quality control system for integrated meteorological sensor data." Journal of the Korea Meteorological Society, KOMES, Vol. 24, No. 3, pp. 445-456. (in Korean) 10.14191/Atmos.2014.24.3.445

Gartner (2018). How to create a business case for data quality improvement, accessed 14 October 2021, <https://www.gartner.com/smarterwithgartner/how-to-create-a-business-case-for-data-quality-improvement>.

GTONE (2019). IoT stream data quality measurement indicators and profiling method for internet of things and system therefore. KR Patent, 1020190102059, filed Aug 21, 2019, Issued Dec 24, 2019.

Jang, B.J., and Jeong, I.T. (2020). "Development of IoT sensing technology for urban weather and environment observation using the micro mobility." Proceedings of Korea Society of Civil Engineering Conference, KSCE, pp. 504-505. (in Korean)

Kim, H.J., Lee, H.S., Choi, B.J., and Kim, Y.H. (2019). "Machine learning-based quality control and error correction using homogeneous temporal data collected by IoT sensors." Journal of the Korea Convergence Society, Vol. 10, No. 4, pp. 17-23. (in Korean)

Kim, H.S. (2020). "A study on the data quality management evaluation model." Journal of the Korea Convergence Society, Vol. 11, No. 17, pp. 217-222. (in Korean)

Kim, J.S., Lee, D.J., Kim, J.G., Shin, C.S., Park, J.W., and Cho, Y.Y. (2016a). "Real-time weather observation system based on IoT." Annual Conference of KIPS, KIPS, pp. 926-927. (in Korean)

Kim, M., Lee, N.Y., and Park, J.H. (2016b). "A quality evaluation model for IoT services." Korean Information Processing Society Review, KIPS Tr. Comp. and Comm. Sys., Vol. 5, No. 9, pp. 269-274. (in Korean) 10.3745/KTCCS.2016.5.9.269

Kim, S.G., Kim, S.H., Lim, C.H., Na, S.K., Park, S.S., Kim, J.M., and Lee, Y.G. (2021). "Analysis of future demend and utilization of the urban meteorological data for the smart city." Journal of the Korea Meteorological Society, KOMES, Vol. 31, No. 2, pp. 241-249. (in Korean)

Kim, S.Y., Lee, H.C., and Ryoo, S.B. (2016c). "Development and assessment of real-time quality control algorithm for PM₁₀ data observed by continuous ambient particulate monitor." Journal of the Korea Meteorological Society, Atmosphere, Vol. 26, No. 4, pp. 541-551. (in Korean) 10.14191/Atmos.2016.26.4.541

Korea Meteorological Administration (KMA) (2019a). Enforcement rules of the weather observation standardization act. (in Korean)

Korea Meteorological Administration (KMA) (2019b). Standards and procedures for quality ratings of weather observation data. pp. 4-5. (in Korean)

Lee, Y.S., Kim, H.K., Hyun, M.J., Lee, H.A., Lee, Y.H., and Lee, J.W. (2017). "Derivation of standard values for quality control of temperature and precipitation observed data." Proceedings of the Autumn Meeting of KMS, KOMES, pp. 222-223. (in Korean)

Miller, J.S. (1996). U.S. Patent No. 5,506,984. U.S. Patent and Trademark Office, Washington, DC, U.S.

Oh, S.W., Im, N.H., Lee, S.H., and Kim, M.S. (2020). "Long-term price prediction and trend analysis of garlic using prophet model." Journal of the Korean Data Analysis Society, Vol. 22, No. 6, pp. 2325-2336. (in Korean) 10.37727/jkdas.2020.22.6.2325

Park, C.Y., and Choi, Y.E. (2012). "Validation of quality control algorithms for temperature data of the Republic of Korea." Journal of the Korea Meteorological Society, KOMES, Vol. 22, No. 3, pp. 299-307. (in Korean) 10.14191/Atmos.2012.22.3.299

Seo, J.H., Hwang, S.Y., and Kim, Y.H. (2002). "An implementation of radiosonde for atmospheric pressure information acquisition." Proceedings of the International Conference of Manufacturing Technology Engineers, KSMTE, pp. 618-620. (in Korean)

Taylor, S.J, and Letham, B. (2017). "Forecasting at scale." PeerJ Preprints Vol. 5, e3190v2. doi: 10.7287/peerj.preprints.3190v2 10.7287/peerj.preprints.3190v2

Information

Publisher :KOREA WATER RESOURECES ASSOCIATION
Publisher(Ko) :한국수자원학회
Journal Title :Journal of Korea Water Resources Association
Journal Title(Ko) :한국수자원학회 논문집
Volume : 54
No :12
Pages :1071-1081
Received Date : 2021-09-30
Revised Date : 2021-10-18
Accepted Date : 2021-10-28
DOI :https://doi.org/10.3741/JKWRA.2021.54.S-1.1071

Journal of Korea Water Resources Association ISSN:2799-8746(Print) 2799-8754(Online) 한국수자원학회 논문집

All Issue