Make use of a completely functional ARIMA+GARCH python implementation and test it over different markets using a simple framework for visualization and comparisons.

Image for post
Image for post
Image by author.

Introduction

When it comes to financial Time Series (TS) modelling, autoregressive models (models that makes use of previous values to forecast the future) such as ARMA, ARIMA or GARCH and its various variants are usually the preferred ones to explain the foundations of TS modelling. However, practical application of these techniques in real trading strategies and it’s comparison to naïve strategies (like Buy and Hold) are not that common. Moreover, it’s not easy to find a ready to use implementation that could be easily replicated for other markets, assets, etc. Some of the codes that I had run into have failures or are just incomplete and missing something. …


Enjoy an easy-to-use unsupervised water detection algorithm for Sentinel 2 and Landsat 8 images that uses a multi-dimensional clustering coupled with naïve bayes classifier for improved performance.

Image for post
Image for post
Figure 1. Example of water mask extracted from a Camargue scene in France. Image by author.

This story is divided in two parts: Methodology and the waterdetect package. In the methodology, the main concepts of the algorithm are given, in order to provide the reader a better understanding of the package and how to tune it. The second part is a tutorial on the waterdetect package with sample codes to run it.

Methodology

Introduction

The use of deep learning techniques for remote sensing applications has been increasing in recent years. The recently published review paper “Object Detection and Image Segmentation with Deep Learning on Earth Observation Data: A Review-Part I: Evolution and Recent Trends” (Hoeser and Kuenzer 2020)[1] presents the evolution of Convolutional Neural Networks (CNN) in satellite applications, notably Object Detection and Image Segmentation, where it achieves state-of-the-art results. …


Improve the performance of your deep learning algorithms with multispectral image augmentations and Fastai v2

Image for post
Image for post
Figure 1: Augmentations applied to a Landsat 8 patch and its corresponding cloud mask. Image by Author.

We know that image augmentation is a key factor for computer vision tasks. It helps the algorithm to avoid overfitting, as well as it limits the need for huge training datasets [1]. Most deep learning frameworks have a vision module that implements augmentation “out-of-the-box”, as it is the case of Keras, PyTorch and Fastai libraries. The problem starts to rise when we need to feed the model with images that don’t match the 3-channel standard (RGB). That’s the case of most remote sensing applications (ex. Figure 1) and a number of other areas.

To overcome this limitation, I will show how to implement a multispectral augmentation using the Albumentations library [2] and plug it into a Fastai v2 DataBlock for further training. The principles about how to create a DataBlock for satellite images can be found in my previous story “How to create a DataBlock for Multispectral Satellite Image Segmentation with the Fastai-v2”. …


Enjoy sate-of-the art results in your Earth Observation tasks with simple coding and the Fastai’s deep learning API.

Image for post
Image for post
Water pixels segmentation example in Jirau reservoir (Brazil), using U-Net architecture and Fastai-v2.

Fastai is an open source deep learning library that adds higher level functionalities to PyTorch and makes it easier to achieve state-of-the-art results with little coding.

The vision module is really handy when we need to quickly create an image dataset, apply data augmentations, resize, crop or even overlay a segmentation mask (Figure 1). However, all this simplification comes with a cost. Most of these higher level APIs for computer vision are optimized to RGB images, and these image libraries don’t support multispectral or multichannel images.


Image for post
Image for post
Figure — Example of patch creation for image segmentation of water bodies.

Introduction

In some previous stories (here, here and here) we’ve used PyTorch and Fast.ai library to segment clouds in satellite images, using as reference a public dataset (Kaggle’s 38-Cloud: Cloud Segmentation in Satellite Images). However, there are cases when we need to prepare our own dataset from the beginning, and that can be time-consuming without the proper tools.

As it is not my objective here to explain GEE in depth, I will cover just the basics needed to accomplish our final goal, that is to obtain training patches ready to be consumed by any deep learning framework. …


Image for post
Image for post
Figure 1. Example of unsupervised agglomerative clustering for water detection in satellite images. Lake Qaraoun (Lebanon).

Introduction

The clustering analysis is one type of unsupervised technique used to identify (i.e. group) similar samples in a multi-dimensional data space. In the context of remote sensing data, it is largely used for pixel classification, where the multi-dimensional space represents the pixel reflectances in n different wavelengths (sensor bands) or any other combination.

During my first year of PhD, at the Paul Sabatier University (Toulouse), more specifically working at the GET laboratory, we have been working with satellite images from Sentinel-2 and Landsat-8 missions to assess inland water surface. As the water reflectance can vary a lot depending on the region, atmospheric conditions, and constituents it is not viable to employ a single threshold, or a set of fixed rules to separate water from land. …


Image for post
Image for post
Cloud segmentation: RGB image (left), ground truth (middle) and our model predictions (right).

In my previous story (here), I showed how to create a multi-channel dataset for satellite images from scratch, without using the torchvision module.

Now, we will move on to create a simple deep learning model, for semantic segmentation of satellite images and check how it performs using the 38-Cloud: Cloud Segmentation in Satellite Image, from Kaggle.

To make things easier, this code is available in Kaggle notebook 38-Cloud-Simple_Unet, available here.

The Model

My different model architectures can be used for a pixel-level segmentation of images. The 2019 Guide to Semantic Segmentation is a good guide for many of them, showing the main differences in their concepts. …


Image for post
Image for post
Cloud Segmentation

In my last Medium story (here) I proposed an approach using the high level API Fast.ai to detect cloud contours in satellite images. Detecting object contours (i.e. all the pixels belonging to the same object) is called semantic segmentation. The dataset that was used for the task is the 38-Cloud: Cloud Segmentation in Satellite Image, from Kaggle.

Although we could achieve a relatively good accuracy of 96% with a few lines of code, the model was not able to consider all the input channels, Red, Green, Blue and NIR(Near Infrared) provided in the dataset. The problem is that most of the semantic segmentation models found in deep learning frameworks like Keras, Fast.ai and even PyTorch are designed to, and come with pre-trained weights, to work with RGB images. Besides that, the vision module of these libraries are also stuck to RGB files. …


Image for post
Image for post
Photo by Asael Peña on Unsplash

Neural nets can be both, threatening and "passionate" at the same time. The more you learn and the more you go deep into it, you unleashes its power and a whole bunch of new ideas and applications come to our minds. Environmental applications are amongst those subjects that attracts more attention daily and remote sensing is one powerful tool for researchers, students and policy makers to understand environmental processes.

But, how could we bridge between neural nets and remote sensing into something useful? A quick research on the internet will show you up many ideas. The purpose of my first Medium publication, is to provide a simple introductory notebook on using Convolutional Neural Networks (CNNs) to segment clouds from satellite images. …

About

Maurício Cordeiro

Doctorate student @ Université Paul Sabatier (Toulouse). Water Resources Specialist at the Brazilian National Water Agency.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store