1,721,016 research outputs found
Diversifying Non-dissipative Reservoir Computing Dynamics
No full textThe Euler State Network (EuSNs) model is a recently proposed Reservoir Computing methodology that provides stable and non-dissipative untrained dynamics by discretizing an appropriately constrained ODE. In this paper, we propose alternative formulations of the reservoirs for EuSNs, aiming at improving the diversity of the resulting dynamics. Our empirical analysis points out the effectiveness of the proposed approaches on a large pool of time-series classification tasks
Non-dissipative Reservoir Computing Approaches for Time-Series Classification
No full textReservoir Computing (RC) is a consolidated framework for designing fastly trainable recurrent neural systems, where the dynamical component is fixed and initialized to implement a fading memory over the input signal. In this paper, we study the behavior of a recently introduced class of alternative RC approaches in which the fixed dynamical component implements a stable but non-dissipative system, so that the driving temporal signal can be propagated through multiple time steps effectively. We analyze the behavior of two classes of non-dissipative RC in terms of dynamical stability and show the resulting advantages in time-series classification tasks in comparison to conventional RC
Euler State Networks: Non-dissipative Reservoir Computing
Full text linkInspired by the numerical solution of ordinary differential equations, in
this paper we propose a novel Reservoir Computing (RC) model, called the Euler
State Network (EuSN). The presented approach makes use of forward Euler
discretization and antisymmetric recurrent matrices to design reservoir
dynamics that are both stable and non-dissipative by construction.
Our mathematical analysis shows that the resulting model is biased towards a
unitary effective spectral radius and zero local Lyapunov exponents,
intrinsically operating near to the edge of stability. Experiments on long-term
memory tasks show the clear superiority of the proposed approach over standard
RC models in problems requiring effective propagation of input information over
multiple time-steps. Furthermore, results on time-series classification
benchmarks indicate that EuSN is able to match (or even exceed) the accuracy of
trainable Recurrent Neural Networks, while retaining the training efficiency of
the RC family, resulting in up to 490-fold savings in computation
time and 1750-fold savings in energy consumption.Comment: paper submitted to journa
Minimal Euler State Networks
No full textThe Euler State Network (EuSN) is a recently proposed Reservoir Computing (RC) model where the fixed state dynamics are obtained by discretizing an ordinary differential equation under stability and non-dissipative conditions. As a result, the model is able to effectively propagate input information over time, hugely improving the performance of RC models in tasks requiring long-term memory. Aiming at both reducing the complexity of the reservoir structure and further improving its efficiency, in this paper we propose a minimalistic EuSN architecture where the reservoir is constrained to a fixed bi-directional chain structure. We explore progressive simplifications where the recurrent and the input connections of the reservoir are fully described by a single weight value. While reducing the complexity of the base EuSN, the proposed minimal EuSN approach shows comparable performance on several tasks on time-series classification, thus offering considerable potential advantages, especially in embedded applications and physical implementations
Reservoir Computing neural networks for estimating mechanical properties of hot steel strips
No full textThe steelmaking industry could benefit greatly from a reliable technique for predicting the mechanical properties of rolling products. This would lead to significant reductions in time and costs associated with the process.In this paper, we present a novel approach to predict the ultimate tensile strength of hot-rolled steel strips, utilizing the capabilities of recurrent neural models to process temporal data. Our focus is on Reservoir Computing (RC) models, selected for their efficient training characteristics, which are advantageous in production support contexts. In the paper, we introduce two custom RC-based architectures, designed to handle input features distinctively based on their relevance to the steelmaking process. The proposed approach is experimentally validated on a real use case with data originating from a hot rolling steel strip plant. It is compared against standard RC and fully-trainable recurrent neural networks. The results demonstrate the ability of the proposed method to reach a significantly good predictive performance, largely within the threshold set by industry experts. Our custom RC models offer an outstanding balance between predictive performance and computational efficiency, making them highly suitable for this application. In comparison to baseline RC approaches, at substantially the same computational cost, we manage to reduce the prediction error by more than 50%. Moreover, in comparison with fully trained models, we achieve even slightly more accurate predictions while reducing computational cost by more than 20 times. Finally, our results indicate that it is possible to further improve predictive performance through the differentiation of predictive models based on chemical composition similarities
Reservoir Topology in Deep Echo State Networks
Full text linkDeep Echo State Networks (DeepESNs) recently extended the applicability of Reservoir Computing (RC) methods towards the field of deep learning. In this paper we study the impact of constrained reservoir topologies in the architectural design of deep reservoirs, through numerical experiments on several RC benchmarks. The major outcome of our investigation is to show the remarkable effect, in terms of predictive performance gain, achieved by the synergy between a deep reservoir construction and a structured organization of the recurrent units in each layer. Our results also indicate that a particularly advantageous architectural setting is obtained in correspondence of DeepESNs where reservoir units are structured according to a permutation recurrent matrix
Ring Reservoir Neural Networks for Graphs
Full text linkMachine Learning for graphs is nowadays a research topic of consolidated relevance. Common approaches in the field typically resort to complex deep neural network architectures and demanding training algorithms, highlighting the need for more efficient solutions. The class of Reservoir Computing (RC) models can play an important role in this context, enabling to develop fruitful graph embeddings through untrained recursive architectures. In this paper, we study progressive simplifications to the design strategy of RC neural networks for graphs. Our core proposal is based on shaping the organization of the hidden neurons to follow a ring topology. Experimental results on graph classification tasks indicate that ring-reservoirs architectures enable particularly effective network configurations, showing consistent advantages in terms of predictive performance
Residual Echo State Networks: Residual recurrent neural networks with stable dynamics and fast learning
No full textResidual connections have been established as a staple for modern deep learning architectures. Most of their applications are cast towards feedforward computing. In this paper, we study the architectural bias of residual connections in the context of recurrent neural networks (RNNs), specifically in the temporal dimension. We frame our discussion from the perspective of Reservoir Computing and dynamical system theory, focusing on important aspects of neural computation like memory capacity, long-term information processing, stability, and nonlinear computation capability. Experiments corroborate the striking advantage brought by temporal residual connections for a plethora of different time series processing tasks, comprehending memory-based, forecasting, and classification problems
Edge of Stability Echo State Network
No full textEcho state networks (ESNs) are time series processing models working under the echo state property (ESP) principle. The ESP is a notion of stability that imposes an asymptotic fading of the memory of the input. On the other hand, the resulting inherent architectural bias of ESNs may lead to an excessive loss of information, which in turn harms the performance in certain tasks with long short-term memory requirements. To bring together the fading memory property and the ability to retain as much memory as possible, in this article, we introduce a new ESN architecture called the Edge of Stability ESN (). The introduced model is based on defining the reservoir layer as a convex combination of a nonlinear reservoir (as in the standard ESN), and a linear reservoir that implements an orthogonal transformation. In virtue of a thorough mathematical analysis, we prove that the whole eigenspectrum of the Jacobian of the map can be contained in an annular neighborhood of a complex circle of controllable radius. This property is exploited to tune the ’s dynamics close to the edge-of-chaos regime by design. Remarkably, our experimental analysis shows that model can reach the theoretical maximum short-term memory capacity (MC). At the same time, in comparison to conventional reservoir approaches, is shown to offer an excellent trade-off between memory and nonlinearity, as well as a significant improvement of performance in autoregressive nonlinear modeling and real-world time series modeling
Architectural richness in deep reservoir computing
No full textReservoir computing (RC) is a popular class of recurrent neural networks (RNNs) with untrained dynamics. Recently, advancements on deep RC architectures have shown a great impact in time-series applications, showing a convenient trade-off between predictive performance and required training complexity. In this paper, we go more in depth into the analysis of untrained RNNs by studying the quality of recurrent dynamics developed by the layers of deep RC neural networks. We do so by assessing the richness of the neural representations in the different levels of the architecture, using measures originating from the fields of dynamical systems, numerical analysis and information theory. Our experiments, on both synthetic and real-world datasets, show that depth—as an architectural factor of RNNs design—has a natural effect on the quality of RNN dynamics (even without learning of the internal connections). The interplay between depth and the values of RC scaling hyper-parameters, especially the scaling of inter-layer connections, is crucial to design rich untrained recurrent neural systems
- …
