Source codes of the AI techniques

Deep clustering has been used in several remote sensing applications. [1] used deep clustering for unsupervised multi-temporal segmentation of optical images. [2] extended this for segmentation and change detection in optical-SAR images. [3] used a deep clustering loss to detect wildlife in drone images.

[1] Saha, S., Mou, L., Qiu, C., Zhu, X.X., Bovolo, F. and Bruzzone, L., 2020. Unsupervised Deep Joint Segmentation of Multi-temporal High-Resolution Images. IEEE Transactions on Geoscience and Remote Sensing, 58(12), pp.8780-8792.

[2] Saha, S., Ebel, P., Zhu, X.X., 2021. Self-supervised Multisensor Change Detection. IEEE Transactions on Geoscience and Remote Sensing, in press.

[3] Zheng, X., Kellenberger, B., Gong, R., Hajnsek, I., Tuia, D. 2021. Self-Supervised Pretraining and Controlled Augmentation Improve Rare Wildlife Recognition in UAV Images. Workshop Learning to Understand Aerial Images (LUAI), International Conference on Computer Vision.

Contrastive Learning uses similar (positive) and dissimilar (negative) samples to learn representations such that similar samples stay close to each other, while the dissimilar ones are pushed far apart. One of the most influential methods in this family, Momentum Contrast (MoCo) [4], views contrastive learning as building dynamic dictionaries. Contrastive learning objectives were also used in [5] for wildlife detection.

[4]    Zheng, X., Kellenberger, B., Gong, R., Hajnsek, I., Tuia, D. 2021. Self-Supervised Pretraining and Controlled Augmentation Improve Rare Wildlife Recognition in UAV Images. Workshop Learning to Understand Aerial Images (LUAI), International Conference on Computer Vision.
[5]    He, K., Fan, H., Wu, Y., Xie, S. and Girshick, R., 2020. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 9729-9738).

Bootstrap Your Own Latent (BYOL) is a recently introduced algorithm for self-supervised learning of image representations, as an improvement over the contrastive learning paradigm, [6] using two neural networks, called online and target networks. These networks interact and learn from each other. The target and the online networks are provided different views of the same images and each network generates a latent representation for their respective data. Then, the output of the target network is predicted using a multi-layer perceptron. It is not trivial in remote sensing applications to provide the selection of negative pairs required in contrastive learning. E.g., for the forestry analysis, two randomly chosen patches may still belong to the same semantic category or may be substantially similar. BYOL may be a potential alternative.

[6] Grill, J.B., Strub, F., Altché, F., Tallec, C., Richemond, P.H., Buchatskaya, E., Doersch, C., Pires, B.A., Guo, Z.D., Azar, M.G. and Piot, B., 2020. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733.

Meta-learning uses a dataset of training tasks instead of having a single pretext task, as a generalisation of the approaches mentioned above. The individual tasks in meta-learning typically contain only few data samples and at test time some labelled samples of the target task are available to adapt a pre-trained model to a given target task. Meta-learning allows to incorporate labelled data that are (to some degree) different from the target task and violate the assumption of machine learning that datasets are sampled from an identical data distribution. In this context, model-agnostic meta-learning algorithm for land cover classification [7], [8] and biomass estimation are some of options to evaluate the applicability in this project.

[7] Rußwurm, M., Wang, S., Körner, M., and Lobell, D. 2020. Meta-learning for few-shot land cover classification. IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 788–796.

[8] Wang, S., Rußwurm, M., Körner, M., and Lobell, D. 2020. meta-learning for few-shot time series classification. IEEE International Geoscience and Remote Sensing Symposium, IGARSS 2020.

Deep change vector analysis (DCVA) is a popular unsupervised change detection framework that compares deep features extracted from the bi-temporal images pixel-wise and obtains change information [9]. DCVA uses a suitable pre-trained network as a feature extractor on the target scene.

[9] Saha, S., Bovolo, F. and Bruzzone, L., 2019. Unsupervised deep change vector analysis for multiple-change detection in VHR images. IEEE Transactions on Geoscience and Remote Sensing, 57(6), pp.3677-3693.

Long Short-Term Memory (LSTM) based unsupervised approach, in the context of the project, is considered as an adaptation of an autoencoder for sequence data using a LSTM encoder-decoder architecture. LSTM autoencoders can be exploited for time-series anomaly detection and has been adopted for remote sensing time series analysis. For instance, in [11], change detection is treated as an anomaly detection problem by exploiting a multilayer LSTM network to learn representation of the time series, which is also applied as a baseline in this project.

[11] Leenstra, M., Marcos, D., Bovolo, F., Tuia, D., 2021. Self-supervised pre-training enhances change detection in Sentinel-2 images. Pattern Recognition and Remote sensing Workshop, International Conference in Pattern Recognition

Graph neural networks based on a self‑supervised approach, as a subfield of geometric deep learning [12], [13], aims to extend invariance and equivariance properties of deep neural models to non-Euclidean domains while capturing both the spatial and temporal relationships, in the scope of this project. Even though some works already tried to use graph convolutional networks (GCN) to perform anomaly detection on graphs (either on node level, sub‑graph or graph level), considering temporal relationships with GCN are still to be applied to anomaly detection [14].

[12] Saha, S., Mou, L., Zhu, X.X., Bovolo, F. and Bruzzone, L., 2020. Semisupervised change detection using graph convolutional network. IEEE Geoscience and Remote Sensing Letters, 18(4), pp.607-611.

[13] Zhou, J., et al. “Graph neural networks: A review of methods and applications.” AI Open 1, 2020, pp. 57-81.

[14] Avolio, C, Tricomi, A., Zavagli, M., De Vendictis, L., Volpe, F., Costantini, M. Automatic Detection of Anomalous Time Trends from Satellite Image Series to Support Agricultural Monitoring. 2021, IEEE IGARSS 2021.