Investigating three-dimensional convolutional and recurrent neural networks for crop classification using time-series optical images

Teimouri, Maryam; Mokhtarzade, Mehdi

doi:10.52547/jgst.12.3.1

[Home ] [Archive]

[ فارسی ]

Journal of Geomatics Science and Technology

Main Menu

Home

Browse

Journal Info

Guide for Authors

Submit Manuscript

Articles archive

For Reviewers

Contact us

Site Facilities

Reviewers

Search in website

Receive site information

Volume 12, Issue 3 (3-2023)

JGST 2023, 12(3): 1-15

Back to browse issues page

Investigating three-dimensional convolutional and recurrent neural networks for crop classification using time-series optical images

Maryam Teimouri ^*

, Mehdi Mokhtarzade

Abstract: (647 Views)

One of the serious challenges in remote sensing is extracting suitable features. With the presence of a new generation of deep neural networks, automatic and accurate feature extraction and classification of crops have become possible. On the other hand, appropriate features can partially reduce the effects of spectral similarity in the detection of different crops while improving classification accuracy. Also, the use of time-series data during the crop growth period provides useful information about crops to researchers. In this regard, this research aimed to investigate and evaluate three methods, three-dimensional convolutional neural network (3D-CNN), long short-term memory (LSTM), and gated recurrent unit (GRU), to extract appropriate features from time-series optical images. In the architecture of 3D-CNN, an attempt was made to design a structure so that the optimal spatial-temporal features could be extracted from time series images, and then the results were evaluated and compared with two other methods (i.e. LSTM, GRU). Finally, according to the results, 3D-CNN, with an overall accuracy (OA) of 90.70% and a kappa coefficient (KC) of 89.37%, which were about 3.50% and 4.00% higher than the OA and KC of LSTM, respectively, demonstrated a greater capability to identify crops. Moreover, the OA of the results of the classification by GRU was close to the OA of 3D-CNN, and only the OA of this method was 1.48% better than GRU. Therefore, the results confirmed the efficiency and suitability of 3D-CNN for crop classification.

Article number: 1

Keywords: crop classification, time series images, 3D-CNN, LSTM, GRU

Full-Text [PDF 1415 kb] (232 Downloads)

Type of Study: Research | Subject: Photo&RS
Received: 2021/04/16

References

1. X.-P. Song, P. V. Potapov, A. Krylov, L. King, C. M. Di Bella, A. Hudson, A. Khan, B. Adusei, S. V. Stehman, and M. C. Hansen, "National-scale soybean mapping and area estimation in the United States using medium resolution satellite imagery and field survey," Remote Sensing of Environment, vol. 190, pp. 383-395, 2017. [DOI:10.1016/j.rse.2017.01.008]

2. D. Haboudane, N. Tremblay, J. R. Miller, and P. Vigneault, "Remote estimation of crop chlorophyll content using spectral indices derived from hyperspectral data," IEEE Transactions on Geoscience and remote Sensing, vol. 46, pp. 423-437, 2008. [DOI:10.1109/TGRS.2007.904836]

3. E. Cloutis, D. Connery, D. Major, and F. Dover, "Airborne multi-spectral monitoring of agricultural crop status: effect of time of year, crop type and crop condition parameter," Remote Sensing, vol. 17, pp. 2579-2601, 1996. [DOI:10.1080/01431169608949094]

4. A. Verhegghen, S. Bontemps, and P. Defourny, "A global NDVI and EVI reference data set for land-surface phenology using 13 years of daily SPOT-VEGETATION observations," International Journal of remote sensing, vol. 35, pp. 2440-2471, 2014. [DOI:10.1080/01431161.2014.883105]

5. S. Niazmardi, S. Homayouni, A. Safari, H. McNairn, J. Shang, and K. Beckett, "Histogram-based spatio-temporal feature classification of vegetation indices time-series for crop mapping," International Journal of Applied Earth Observation and Geoinformation, vol. 72, pp. 34-41, 2018. [DOI:10.1016/j.jag.2018.05.014]

6. N. Kussul, L. Mykola, A. Shelestov, and S. Skakun, "Crop inventory at regional scale in Ukraine: developing in season and end of season crop maps with multi-temporal optical and SAR satellite imagery," European Journal of Remote Sensing, vol. 51, pp. 627-636, 2018. [DOI:10.1080/22797254.2018.1454265]

7. F. Vuolo, M. Neuwirth, M. Immitzer, C. Atzberger, and W.-T. Ng, "How much does multi-temporal Sentinel-2 data improve crop type classification?," International Journal of Applied Earth Observation and Geoinformation, vol. 72, pp. 122-130, 2018. [DOI:10.1016/j.jag.2018.06.007]

8. G. V. Laurin, C. Belli, R. Bianconi, P. Laranci, and D. Papale, "Early mapping of industrial tomato in Central and Southern Italy with Sentinel 2, aerial and RapidEye additional data," The Journal of Agricultural Science, vol. 156, pp. 396-407, 2018. [DOI:10.1017/S0021859618000400]

9. G. Waldhoff, U. Lussem, and G. Bareth, "Multi-Data Approach for remote sensing-based regional crop rotation mapping: A case study for the Rur catchment, Germany," International Journal of Applied Earth Observation and Geoinformation, vol. 61, pp. 55-69, 2017. [DOI:10.1016/j.jag.2017.04.009]

10. D. I. Moody, S. P. Brumby, R. Chartrand, R. Keisler, N. Longbotham, C. Mertes, S. W. Skillman, and M. S. Warren, "Crop classification using temporal stacks of multispectral satellite imagery," in Algorithms and Technologies for Multispectral, Hyperspectral, and Ultraspectral Imagery XXIII, 2017, p. 101980G. [DOI:10.1117/12.2262804]

11. H. Qiong, W.-b. WU, S. Qian, L. Miao, C. Di, Q.-y. YU, and H.-j. TANG, "How do temporal and spectral features matter in crop classification in Heilongjiang Province, China?," Journal of integrative agriculture, vol. 16, pp. 324-336, 2017. [DOI:10.1016/S2095-3119(15)61321-1]

12. B. Mulianga, A. Bégué, P. Clouvel, and P. Todoroff, "Mapping cropping practices of a sugarcane-based cropping system in Kenya using remote sensing," Remote Sensing, vol. 7, pp. 14428-14444, 2015. [DOI:10.3390/rs71114428]

13. M. Teimouri, M. Mokhtarzade, N. Baghdadi, and C. Heipke, "Fusion of Time-Series Optical and SAR Images Using 3D Convolutional Neural Networks for Crop Classification," Geocarto International, no. just-accepted, pp. 1-16, 2022. [DOI:10.1080/10106049.2022.2095446]

14. T. Xia et al., "Exploring the potential of Chinese GF-6 images for crop mapping in regions with complex agricultural landscapes," International Journal of Applied Earth Observation and Geoinformation, vol. 107, p. 102702, 2022. [DOI:10.1016/j.jag.2022.102702]

15. T. Sakamoto, "Early Classification Method for US Corn and Soybean by Incorporating MODIS-Estimated Phenological Data and Historical Classification Maps in Random-Forest Regression Algorithm," Photogrammetric Engineering & Remote Sensing, vol. 87, no. 10, pp. 747-758, 2021. [DOI:10.14358/PERS.21-00003R2]

16. C. Zhang et al., "Rapid in-season mapping of corn and soybeans using machine-learned trusted pixels from Cropland Data Layer," International Journal of Applied Earth Observation and Geoinformation, vol. 102, p. 102374, 2021. [DOI:10.1016/j.jag.2021.102374]

17. M. E. Paoletti, J. M. Haut, J. Plaza, and A. Plaza, "A new deep convolutional neural network for fast hyperspectral image classification," ISPRS Journal of Photogrammetry and Remote Sensing, vol. 145, pp. 120-147, 2018. [DOI:10.1016/j.isprsjprs.2017.11.021]

18. Z. Deng, H. Sun, S. Zhou, J. Zhao, L. Lei, and H. Zou, "Multi-scale object detection in remote sensing imagery with convolutional neural networks," ISPRS Journal of Photogrammetry and Remote Sensing, vol. 145, pp. 3-22, 2018. [DOI:10.1016/j.isprsjprs.2018.04.003]

19. M. Voelsen, M. Teimouri, F. Rottensteiner, and C. Heipke, "INVESTIGATING 2D AND 3D CONVOLUTIONS FOR MULTITEMPORAL LAND COVER CLASSIFICATION USING REMOTE SENSING IMAGES," ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. 3, pp. 271-279, 2022. [DOI:10.5194/isprs-annals-V-3-2022-271-2022]

20. A. Shakya, M. Biswas, and M. Pal, "CNN-based fusion and classification of SAR and Optical data," International Journal of Remote Sensing, vol. 41, no. 22, pp. 8839-8861, 2020. [DOI:10.1080/01431161.2020.1783713]

21. L. Zhong, L. Hu, and H. Zhou, "Deep learning based multi-temporal crop classification," Remote sensing of environment, vol. 221, pp. 430-443, 2019. [DOI:10.1016/j.rse.2018.11.032]

22. H. Zhao, S. Duan, J. Liu, L. Sun, and L. Reymondin, "Evaluation of Five Deep Learning Models for Crop Type Mapping Using Sentinel-2 Time Series Images with Missing Information," Remote Sensing, vol. 13, no. 14, p. 2790, 2021. [DOI:10.3390/rs13142790]

23. S. Ji, C. Zhang, A. Xu, Y. Shi, and Y. Duan, "3D Convolutional Neural Networks for Crop Classification with Multi-Temporal Remote Sensing Images," Remote Sensing, vol. 10, p. 75, 2018. [DOI:10.3390/rs10010075]

24. R. Fernandez-Beltran, T. Baidar, J. Kang and F. Pla, "Rice-yield prediction with multi-temporal sentinel-2 data and 3D CNN: A case study in Nepal "Remote Sensing, vol. 13, p. 1391, 2021. [DOI:10.3390/rs13071391]

25. J. Adrian, V. Sagan and M. Maimaitijiang "Sentinel SAR-optical fusion for crop type mapping using deep learning and Google Earth Engine "ISPRS Journal of Photogrammetry and Remote Sensing, vol. 175, p. 215-235, 2021. [DOI:10.1016/j.isprsjprs.2021.02.018]

26. D. Ienco, R. Gaetano, C. Dupaquier, and P. Maurel, "Land cover classification via multitemporal spatial data by deep recurrent neural networks," IEEE Geoscience and Remote Sensing Letters, vol. 14, pp. 1685-1689, 2017. [DOI:10.1109/LGRS.2017.2728698]

27. D. H. T. Minh, D. Ienco, R. Gaetano, N. Lalande, E. Ndikumana, F. Osman, and P. Maurel, "Deep recurrent neural networks for winter vegetation quality mapping via multitemporal sar sentinel-1," IEEE Geoscience and Remote Sensing Letters, vol. 15, pp. 464-468, 2018. [DOI:10.1109/LGRS.2018.2794581]

28. E. Ndikumana, D. Ho Tong Minh, N. Baghdadi, D. Courault, and L. Hossard, "Deep Recurrent Neural Network for Agricultural Classification using multitemporal SAR Sentinel-1 for Camargue, France," Remote Sensing, vol. 10, p. 1217, 2018. [DOI:10.3390/rs10081217]

29. S. Ji, W. Xu, M. Yang, and K. Yu, "3D convolutional neural networks for human action recognition," IEEE Transactions on pattern analysis and machine intelligence, vol. 35, pp. 221-231, 2012. [DOI:10.1109/TPAMI.2012.59]

30. S. Hochreiter and J. Schmidhuber, "LSTM can solve hard long time lag problems," Advances in neural information processing systems, pp. 473-479, 1997.

31. K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, "Learning phrase representations using RNN encoder-decoder for statistical machine translation," arXiv preprint arXiv:1406.1078, 2014. [DOI:10.3115/v1/D14-1179]

32. Q. Gao, S. Lim, and X. Jia, "Hyperspectral Image Classification Using Convolutional Neural Networks and Multiple Feature Learning," Remote Sensing, vol. 10, p. 299, 2018. [DOI:10.3390/rs10020299]

33. S. Morell-Monzó, J. Estornell, and M.-T. Sebastiá-Frasquet, "Comparison of Sentinel-2 and high-resolution imagery for mapping land abandonment in fragmented areas," Remote Sensing, vol. 12, no. 12, p. 2062, 2020. [DOI:10.3390/rs12122062]

34. Y.Rezaei, SB. Fatemi, "Principles of Remote Sensing," publications of Azade, 1358.

35. X.-P. Song, P. V. Potapov, A. Krylov, L. King, C. M. Di Bella, A. Hudson, A. Khan, B. Adusei, S. V. Stehman, and M. C. Hansen, "National-scale soybean mapping and area estimation in the United States using medium resolution satellite imagery and field survey," Remote Sensing of Environment, vol. 190, pp. 383-395, 2017. [DOI:10.1016/j.rse.2017.01.008]

36. D. Haboudane, N. Tremblay, J. R. Miller, and P. Vigneault, "Remote estimation of crop chlorophyll content using spectral indices derived from hyperspectral data," IEEE Transactions on Geoscience and remote Sensing, vol. 46, pp. 423-437, 2008. [DOI:10.1109/TGRS.2007.904836]

37. E. Cloutis, D. Connery, D. Major, and F. Dover, "Airborne multi-spectral monitoring of agricultural crop status: effect of time of year, crop type and crop condition parameter," Remote Sensing, vol. 17, pp. 2579-2601, 1996. [DOI:10.1080/01431169608949094]

38. A. Verhegghen, S. Bontemps, and P. Defourny, "A global NDVI and EVI reference data set for land-surface phenology using 13 years of daily SPOT-VEGETATION observations," International Journal of remote sensing, vol. 35, pp. 2440-2471, 2014. [DOI:10.1080/01431161.2014.883105]

39. S. Niazmardi, S. Homayouni, A. Safari, H. McNairn, J. Shang, and K. Beckett, "Histogram-based spatio-temporal feature classification of vegetation indices time-series for crop mapping," International Journal of Applied Earth Observation and Geoinformation, vol. 72, pp. 34-41, 2018. [DOI:10.1016/j.jag.2018.05.014]

40. N. Kussul, L. Mykola, A. Shelestov, and S. Skakun, "Crop inventory at regional scale in Ukraine: developing in season and end of season crop maps with multi-temporal optical and SAR satellite imagery," European Journal of Remote Sensing, vol. 51, pp. 627-636, 2018. [DOI:10.1080/22797254.2018.1454265]

41. F. Vuolo, M. Neuwirth, M. Immitzer, C. Atzberger, and W.-T. Ng, "How much does multi-temporal Sentinel-2 data improve crop type classification?," International Journal of Applied Earth Observation and Geoinformation, vol. 72, pp. 122-130, 2018. [DOI:10.1016/j.jag.2018.06.007]

42. G. V. Laurin, C. Belli, R. Bianconi, P. Laranci, and D. Papale, "Early mapping of industrial tomato in Central and Southern Italy with Sentinel 2, aerial and RapidEye additional data," The Journal of Agricultural Science, vol. 156, pp. 396-407, 2018. [DOI:10.1017/S0021859618000400]

43. G. Waldhoff, U. Lussem, and G. Bareth, "Multi-Data Approach for remote sensing-based regional crop rotation mapping: A case study for the Rur catchment, Germany," International Journal of Applied Earth Observation and Geoinformation, vol. 61, pp. 55-69, 2017. [DOI:10.1016/j.jag.2017.04.009]

44. D. I. Moody, S. P. Brumby, R. Chartrand, R. Keisler, N. Longbotham, C. Mertes, S. W. Skillman, and M. S. Warren, "Crop classification using temporal stacks of multispectral satellite imagery," in Algorithms and Technologies for Multispectral, Hyperspectral, and Ultraspectral Imagery XXIII, 2017, p. 101980G. [DOI:10.1117/12.2262804]

45. H. Qiong, W.-b. WU, S. Qian, L. Miao, C. Di, Q.-y. YU, and H.-j. TANG, "How do temporal and spectral features matter in crop classification in Heilongjiang Province, China?," Journal of integrative agriculture, vol. 16, pp. 324-336, 2017. [DOI:10.1016/S2095-3119(15)61321-1]

46. B. Mulianga, A. Bégué, P. Clouvel, and P. Todoroff, "Mapping cropping practices of a sugarcane-based cropping system in Kenya using remote sensing," Remote Sensing, vol. 7, pp. 14428-14444, 2015. [DOI:10.3390/rs71114428]

47. M. Teimouri, M. Mokhtarzade, N. Baghdadi, and C. Heipke, "Fusion of Time-Series Optical and SAR Images Using 3D Convolutional Neural Networks for Crop Classification," Geocarto International, no. just-accepted, pp. 1-16, 2022. [DOI:10.1080/10106049.2022.2095446]

48. T. Xia et al., "Exploring the potential of Chinese GF-6 images for crop mapping in regions with complex agricultural landscapes," International Journal of Applied Earth Observation and Geoinformation, vol. 107, p. 102702, 2022. [DOI:10.1016/j.jag.2022.102702]

49. T. Sakamoto, "Early Classification Method for US Corn and Soybean by Incorporating MODIS-Estimated Phenological Data and Historical Classification Maps in Random-Forest Regression Algorithm," Photogrammetric Engineering & Remote Sensing, vol. 87, no. 10, pp. 747-758, 2021. [DOI:10.14358/PERS.21-00003R2]

50. C. Zhang et al., "Rapid in-season mapping of corn and soybeans using machine-learned trusted pixels from Cropland Data Layer," International Journal of Applied Earth Observation and Geoinformation, vol. 102, p. 102374, 2021. [DOI:10.1016/j.jag.2021.102374]

51. M. E. Paoletti, J. M. Haut, J. Plaza, and A. Plaza, "A new deep convolutional neural network for fast hyperspectral image classification," ISPRS Journal of Photogrammetry and Remote Sensing, vol. 145, pp. 120-147, 2018. [DOI:10.1016/j.isprsjprs.2017.11.021]

52. Z. Deng, H. Sun, S. Zhou, J. Zhao, L. Lei, and H. Zou, "Multi-scale object detection in remote sensing imagery with convolutional neural networks," ISPRS Journal of Photogrammetry and Remote Sensing, vol. 145, pp. 3-22, 2018. [DOI:10.1016/j.isprsjprs.2018.04.003]

53. M. Voelsen, M. Teimouri, F. Rottensteiner, and C. Heipke, "INVESTIGATING 2D AND 3D CONVOLUTIONS FOR MULTITEMPORAL LAND COVER CLASSIFICATION USING REMOTE SENSING IMAGES," ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. 3, pp. 271-279, 2022. [DOI:10.5194/isprs-annals-V-3-2022-271-2022]

54. A. Shakya, M. Biswas, and M. Pal, "CNN-based fusion and classification of SAR and Optical data," International Journal of Remote Sensing, vol. 41, no. 22, pp. 8839-8861, 2020. [DOI:10.1080/01431161.2020.1783713]

55. L. Zhong, L. Hu, and H. Zhou, "Deep learning based multi-temporal crop classification," Remote sensing of environment, vol. 221, pp. 430-443, 2019. [DOI:10.1016/j.rse.2018.11.032]

56. H. Zhao, S. Duan, J. Liu, L. Sun, and L. Reymondin, "Evaluation of Five Deep Learning Models for Crop Type Mapping Using Sentinel-2 Time Series Images with Missing Information," Remote Sensing, vol. 13, no. 14, p. 2790, 2021. [DOI:10.3390/rs13142790]

57. S. Ji, C. Zhang, A. Xu, Y. Shi, and Y. Duan, "3D Convolutional Neural Networks for Crop Classification with Multi-Temporal Remote Sensing Images," Remote Sensing, vol. 10, p. 75, 2018. [DOI:10.3390/rs10010075]

58. R. Fernandez-Beltran, T. Baidar, J. Kang and F. Pla, "Rice-yield prediction with multi-temporal sentinel-2 data and 3D CNN: A case study in Nepal "Remote Sensing, vol. 13, p. 1391, 2021. [DOI:10.3390/rs13071391]

59. J. Adrian, V. Sagan and M. Maimaitijiang "Sentinel SAR-optical fusion for crop type mapping using deep learning and Google Earth Engine "ISPRS Journal of Photogrammetry and Remote Sensing, vol. 175, p. 215-235, 2021. [DOI:10.1016/j.isprsjprs.2021.02.018]

60. D. Ienco, R. Gaetano, C. Dupaquier, and P. Maurel, "Land cover classification via multitemporal spatial data by deep recurrent neural networks," IEEE Geoscience and Remote Sensing Letters, vol. 14, pp. 1685-1689, 2017. [DOI:10.1109/LGRS.2017.2728698]

61. D. H. T. Minh, D. Ienco, R. Gaetano, N. Lalande, E. Ndikumana, F. Osman, and P. Maurel, "Deep recurrent neural networks for winter vegetation quality mapping via multitemporal sar sentinel-1," IEEE Geoscience and Remote Sensing Letters, vol. 15, pp. 464-468, 2018. [DOI:10.1109/LGRS.2018.2794581]

62. E. Ndikumana, D. Ho Tong Minh, N. Baghdadi, D. Courault, and L. Hossard, "Deep Recurrent Neural Network for Agricultural Classification using multitemporal SAR Sentinel-1 for Camargue, France," Remote Sensing, vol. 10, p. 1217, 2018. [DOI:10.3390/rs10081217]

63. S. Ji, W. Xu, M. Yang, and K. Yu, "3D convolutional neural networks for human action recognition," IEEE Transactions on pattern analysis and machine intelligence, vol. 35, pp. 221-231, 2012. [DOI:10.1109/TPAMI.2012.59]

64. S. Hochreiter and J. Schmidhuber, "LSTM can solve hard long time lag problems," Advances in neural information processing systems, pp. 473-479, 1997.

65. K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, "Learning phrase representations using RNN encoder-decoder for statistical machine translation," arXiv preprint arXiv:1406.1078, 2014. [DOI:10.3115/v1/D14-1179]

66. Q. Gao, S. Lim, and X. Jia, "Hyperspectral Image Classification Using Convolutional Neural Networks and Multiple Feature Learning," Remote Sensing, vol. 10, p. 299, 2018. [DOI:10.3390/rs10020299]

67. S. Morell-Monzó, J. Estornell, and M.-T. Sebastiá-Frasquet, "Comparison of Sentinel-2 and high-resolution imagery for mapping land abandonment in fragmented areas," Remote Sensing, vol. 12, no. 12, p. 2062, 2020. [DOI:10.3390/rs12122062]

Send email to the article author

Add your comments about this article

‎ 10.52547/jgst.12.3.1

‎ 20.1001.1.2322102.1401.12.3.1.0

Mendeley

Zotero

RefWorks

Teimouri M, Mokhtarzade M. Investigating three-dimensional convolutional and recurrent neural networks for crop classification using time-series optical images. JGST 2023; 12 (3) : 1
URL: http://jgst.issgeac.ir/article-1-1021-en.html

Rights and permissions
	This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

Volume 12, Issue 3 (3-2023)

Back to browse issues page