1,720,990 research outputs found
Real-world load time series
No full textIn this chapter, we consider three different real-world datasets, which contain real-valued time series of measurements of electricity and telephonic activity load. For each dataset, we set up a short-term load forecast problem of 24 hours ahead prediction. Two of the datasets under analysis include time series of measurements of exogenous variables, which are used to provide additional context to the network and thus to improve the accuracy of the prediction. For each dataset, we perform an analysis to study the nature of the time series, in terms of its correlation properties, seasonal patterns, correlation with the exogenous time series, and nature of the variance. According to the result of our analysis, we select a suitable preprocessing strategy before feeding the data into the recurrent neural networks. As shown in the following, the forecast accuracy in a prediction problem can be considerably improved by proper preprocessing of data (Zhang and Qi 2005)
Introduction
No full textA short-term load forecast is the prediction of the consumption of resources in a distribution network in the near future. The supplied resource can be of any kind, such as electricity in power grids or telephone service in telecommunication networks. An accurate forecast of the demand is of utmost importance for the planning of facilities, optimization of day-to-day operations, and an effective management of the available resources. In the context of energy and telecommunication networks, the load data are usually represented as real-valued time series characterized by strong temporal dependencies and seasonal patterns. We begin by reviewing several methods that have been adopted in the past years for the task of short-term load forecast and we highlight their main advantages and limitations. We then introduce the framework of recurrent neural networks, a particular class of artificial neural networks specialized in the processing of sequential/temporal data. We explain how recurrent neural networks can be an effective tool for prediction, especially in those cases where the extent of the time dependencies is unknown a-priori
Conclusions
No full textIn this chapter we summarize the main points of our overview and draw our conclusions. We discuss our interpretations about the reasons behind the different results and performance achieved by the Recurrent Neural Network architectures analyzed. We conclude by hypothesizing possible guidlines for selecting suitable models depending on the specific task at hand
Recurrent neural network architectures
No full textIn this chapter, we present three different recurrent neural network architectures that we employ for the prediction of real-valued time series. All the models reviewed in this chapter can be trained through the previously discussed backpropagation through time procedure. First, we present the most basic version of recurrent neural networks, called Elman recurrent neural network. Then, we introduce two popular gated architectures, which are long short-term memory and the gated recurrent units. We discuss the main advantages of these more sophisticated architectures, especially regarding their capability to process much longer dependencies in time by maintaining an internal memory for longer periods. For each one of the reviewed network, we provide the details and we show the equations for updating the internal state and computing the output at each time step. Then, for each recurrent neural network we also provide a quick overview of its main applications in previous works in the context of real-valued time series forecasting
Synthetic time series
No full textIn this chapter, we describe three different synthetic datasets that we considered to evaluate the performance of the reviewed recurrent neural network architectures in a controlled environment. The generative models of the synthetic time series are the Mackey–Glass system, NARMA, and multiple superimposed oscillators.Those are benchmark tasks commonly considered in the literature to evaluate the performance of a predictive model. The three forecasting exercises that we study have varying levels of difficulty, given by the nature of the signal and the complexity of the task to be solved by the RNN
Properties and training in recurrent neural networks
No full textIn this chapter, we describe the basic concepts behind the functioning of recurrent neural networks and explain the general properties that are common to several existing architectures. We introduce the basis of their training procedure, the backpropagation through time, as a general way to propagate and distribute the prediction error to previous states of the network. The learning procedure consists of updating the model parameters by minimizing a suitable loss function, which includes the error achieved on the target task and, usually, also one or more regularization terms. We then discuss several ways of regularizing the system, highlighting their advantages and drawbacks. Beside the standard stochastic gradient descent procedure, we also present several additional optimization strategies proposed in the literature for updating the network weights. Finally, we illustrate the problem of the vanishing gradient effect, an inherent problem of the gradient-based optimization techniques which occur in several situations while training neural networks. We conclude by discussing the most recent and successful approaches proposed in the literature to limit the vanishing of the gradients
Inference Guided Few-Shot Segmentation
Get PDFFew-shot segmentation has in recent years gotten a lot of attention. The reason is its ability to segment images from classes based on only a handful of labeled support images. This opens up many possibilities when the need for a big dataset is removed.
To do this a few-shot segmentation network need to extract as much quality information from each support image as possible.
In this thesis we are exploring if an existing few-shot segmentation network can be improved by making the inference phase more target class specific. To do this we are introducing our Inference Guided Few-Shot Segmentation (IGFSS) method. It can be applied to an existing few-shot segmentation network. It changes the inference phase from a static network to one that adapts certain class specific parts of the network to each new target class. We tested our method with the Self-Guided Cross-Guided (SGCG) network as backbone. Here we optimized either the prototypes or the decoder. We used the Pascal dataset to compare the results from both methods. This is done on a fixed list from the dataset to be able to make a fair comparison.
In the 5-shot setup, where new classes are segmented based on 5 support images. Here we get a solid improvement when our method is applied to both the prototypes and the decoder. The mean IoU score was increased with 3.7% and 7.5% respectively.
The dataset was analysed with regard to image and object distributions. This gives us a better understanding of the results of our IGFSS method.
While our IGFSS method does benefit all classes this could be a first step towards a Class-Adaptive Inference Guided Few-Shot Segmentation method
Experiments
No full textIn this section, we compare the prediction performance achieved by the recurrent neural network architectures presented in the previous sections on both the synthetic tasks and the real-world datasets. For each architecture, we report the optimal configuration of its hyperparameters for the task at hand, and the best learning strategy adopted for training the model weights. We perform several independent evaluation of the prediction results due to the stochastic initialization of the internal model weights. The accuracy of the forecast is evaluated in terms of normalized mean squared error and the results are reported both as numerical value and graphical depictions of the predicted time series
Other recurrent neural networks models
No full textIn this chapter we review two additional types of Recurrent Neural Network, which present important differences with respect to the architectures described so far. More specifically, we introduce the nonlinear auto-regressive with eXogenous inputs (NARX) neural network and the Echo State Network. Both these networks have been largely employed in Short Term Load Forecast applications and they have been shown to be more effective than other methods based on statistical models. The main differences of NARX networks and Echo State Networks with respect to the other previously described models, are both in terms of their architecture and, in particular, in their training procedure. Indeed, both these architectures are designed in such a way that Back Propagation Through Time is not necessary. Specifically, in NARX the network output is replaced by the expected ground truth and this allows to train the network like a feedforward architecture. On the other hand, in a Echo State Network only the outermost linear layer is trained, usually by means of ridge regression. Due to these fundamental differences, some of the properties and training approaches discussed in the previous sections do not hold for the NARX and Echo State Network models and we reserved a separate chapter to review these models
Deep Image Clustering with Tensor Kernels and Unsupervised Companion Objectives
Get PDFDeep image clustering is a rapidly growing branch of machine learning and computer vision, in which deep neural networks are trained to discover groups within a set of images, in an unsupervised manner. Deep neural networks have proven to be immensely successful in several machine learning tasks, but the majority of these advances have been in supervised settings. The process of labeling data for supervised applications can be extremely time-consuming, or even completely infeasible in many domains. This has led researchers to shift their focus towards the deep clustering field. However, this field is still in its infancy, meaning that it includes several open research questions, regarding e.g. the design and optimization of the algorithms, the discovery of meaningful clusters, and the initialization of model parameters.
In an attempt to address some of these open questions, a new algorithm for deep image clustering is developed in this thesis. The proposed Deep Tensor Kernel Clustering (DTKC) consists of a convolutional neural network (CNN), which is trained to reflect a common cluster structure at the output of all its intermediate layers. Encouraging a consistent cluster structure throughout the network has the potential to guide it towards meaningful clusters, even though these clusters might appear to be nonlinear in the input space. The cluster structure is enforced through the idea of companion objectives, where separate loss functions are attached to each of the layers in the network. These companion objectives are constructed based on a proposed generalization of the Cauchy-Schwarz (CS) divergence, from vectors to tensors of arbitrary rank. Generalizing the CS divergence to tensor-valued data is a crucial step, due to the tensorial nature of the intermediate representations in the CNN. Furthermore, an alternate initialization strategy based on self-supervised learning, is also employed. To the author's best knowledge, this is the first attempt at using this particular self-supervised learning approach to initialize a deep clustering algorithm.
Several experiments are conducted to thoroughly assess the performance of the proposed DTKC model, with and without self-supervised pre-training. The results show that the models outperform, or perform comparable to, a wide range of benchmark algorithms from the literature
- …
