105 research outputs found
Monitoraggio ed esplorazione dei contenuti dinamici utilizzando gli spazi vettoriali
No full textIn modern Natural Language Processing (NLP) and Information Retrieval (IR), individual words are typically embedded in vector space, called `word vectors' or `word embedding', to enable differentiable optimization in neural networks. This leads to a new NLP paradigm that could deal with individual words in neural networks.
The first issue of the above paradigm is that components in neural networks (like word vectors and hidden states) usually do not convey any concrete physical meaning.
One typical way is to use probabilities as well-constrained quantities to better understand neural network components.
The challenge of traditional probability theory is that it cannot treat words as atomic discrete events since words are embedded as dense vectors that are not necessarily mutually orthogonal.
This thesis proposes a novel framework based on Quantum Probability Theory (QPT) that defines probability axioms in vector space, to probabilistically ground word representation, semantic composition, and semantic abstraction in a unified space.
Another issue of the paradigm is that the inductive bias of learning word vectors relies on only the distributional hypothesis: \textit{linguistic items with similar distributions have similar meanings}, while other aspects are usually ignored. This thesis focuses on one of the most nontrivial aspects, namely the spatially or temporally sequential aspect of words. The spatially sequential aspect refers to capture the spatial position of words in any bag-of-words document encoders, while the temporally sequential aspect refers to mine the time-specific word meaning in the scenario when word meanings may evolve with time.
Interestingly, the complex-valued word embedding (with amplitude terms and phase terms), which is induced from QPT, could be naturally used to model sequence (both for spacial sequence and temporal sequence) by directly encoding sequential order in phase terms. The benefit is that the rotation nature of phases in waves makes sequential encoding being always bounded no matter how long the length of the sequence/dynamics is.
Furthermore, a side effect of the thesis is to bridge the gap between \textit{complex-valued word embeddings} and \textit{sinusoidal position embedding}; it therefore reinterprets commonly-used yet `magic' sinusoidal position embedding in a principled way: sinusoidal position embedding is a real-valued variant of the proposed complex-valued word embeddings.
Beyond the spatial dimension, the thesis also explores sinusoidal embeddings in temporally-sequential dimension, called `Word2Fun', for the temporal evolution of words. Word2Fun is proved to be able to approximate any continuous word meaning evolution.
The thesis implements the QPT framework with 1) a Quantum Probability Driven neural Network (QPDN) for document modeling that achieves comparable performance with SOTA approaches in text classification benchmarks; and 2) a further extension for text matching, called `complex-valued network for matching' (CNM) , that achieves comparable performance with SOTA approaches in question answering (a typical matching task) benchmarks.
This additionally shows the potential to use complex-valued word embedding in general document representation. For the complex-valued word embedding in sequential modeling, the empirical study also evidences the superiority of the `complex-valued word embedding' in spatial sequence modeling and Word2Fun in temporal sequence modeling.In modern Natural Language Processing (NLP) and Information Retrieval (IR), individual words are typically embedded in vector space, called `word vectors' or `word embedding', to enable differentiable optimization in neural networks. This leads to a new NLP paradigm that could deal with individual words in neural networks.
The first issue of the above paradigm is that components in neural networks (like word vectors and hidden states) usually do not convey any concrete physical meaning.
One typical way is to use probabilities as well-constrained quantities to better understand neural network components.
The challenge of traditional probability theory is that it cannot treat words as atomic discrete events since words are embedded as dense vectors that are not necessarily mutually orthogonal.
This thesis proposes a novel framework based on Quantum Probability Theory (QPT) that defines probability axioms in vector space, to probabilistically ground word representation, semantic composition, and semantic abstraction in a unified space.
Another issue of the paradigm is that the inductive bias of learning word vectors relies on only the distributional hypothesis: \textit{linguistic items with similar distributions have similar meanings}, while other aspects are usually ignored. This thesis focuses on one of the most nontrivial aspects, namely the spatially or temporally sequential aspect of words. The spatially sequential aspect refers to capture the spatial position of words in any bag-of-words document encoders, while the temporally sequential aspect refers to mine the time-specific word meaning in the scenario when word meanings may evolve with time.
Interestingly, the complex-valued word embedding (with amplitude terms and phase terms), which is induced from QPT, could be naturally used to model sequence (both for spacial sequence and temporal sequence) by directly encoding sequential order in phase terms. The benefit is that the rotation nature of phases in waves makes sequential encoding being always bounded no matter how long the length of the sequence/dynamics is.
Furthermore, a side effect of the thesis is to bridge the gap between \textit{complex-valued word embeddings} and \textit{sinusoidal position embedding}; it therefore reinterprets commonly-used yet `magic' sinusoidal position embedding in a principled way: sinusoidal position embedding is a real-valued variant of the proposed complex-valued word embeddings.
Beyond the spatial dimension, the thesis also explores sinusoidal embeddings in temporally-sequential dimension, called `Word2Fun', for the temporal evolution of words. Word2Fun is proved to be able to approximate any continuous word meaning evolution.
The thesis implements the QPT framework with 1) a Quantum Probability Driven neural Network (QPDN) for document modeling that achieves comparable performance with SOTA approaches in text classification benchmarks; and 2) a further extension for text matching, called `complex-valued network for matching' (CNM) , that achieves comparable performance with SOTA approaches in question answering (a typical matching task) benchmarks.
This additionally shows the potential to use complex-valued word embedding in general document representation. For the complex-valued word embedding in sequential modeling, the empirical study also evidences the superiority of the `complex-valued word embedding' in spatial sequence modeling and Word2Fun in temporal sequence modeling
Quantum-inspired Complex Word Embedding
No full textA challenging task for word embeddings is to capture the emergent meaning or polarity of a combination of individual words. For example, existing approaches in word embeddings will assign high probabilities to the words ”Penguin” and ”Fly” if they frequently co-occur, but it fails to capture the fact that they occur in an opposite sense - Penguins do not fly. We hypothesize that humans do not associate a single polarity or sentiment to each word. The word contributes to the overall polarity of a combination of words depending upon which other words it is combined with. This is analogous to the behavior of microscopic particles which exist in all possible states at the same time and interfere with each other to give rise to new states depending upon their relative phases. We make use of the Hilbert Space representation of such particles in Quantum Mechanics where we subscribe a relative phase to each word, which is a complex number, and investigate two such quantum inspired models to derive the meaning of a combination of words. The proposed models achieve better performances than state-ofthe-art non-quantum models on the binary sentence classification task
On Elastic Language Models
No full textLarge-scale pretrained language models have achieved compelling performance
in a wide range of language understanding and information retrieval tasks.
Knowledge distillation offers an opportunity to compress a large language model
to a small one, in order to reach a reasonable latency-performance tradeoff.
However, for scenarios where the number of requests (e.g., queries submitted to
a search engine) is highly variant, the static tradeoff attained by the
compressed language model might not always fit. Once a model is assigned with a
static tradeoff, it could be inadequate in that the latency is too high when
the number of requests is large or the performance is too low when the number
of requests is small. To this end, we propose an elastic language model
(ElasticLM) that elastically adjusts the tradeoff according to the request
stream. The basic idea is to introduce a compute elasticity to the compressed
language model, so that the tradeoff could vary on-the-fly along scalable and
controllable compute. Specifically, we impose an elastic structure to enable
ElasticLM with compute elasticity and design an elastic optimization to learn
ElasticLM under compute elasticity. To serve ElasticLM, we apply an elastic
schedule. Considering the specificity of information retrieval, we adapt
ElasticLM to dense retrieval and reranking and present ElasticDenser and
ElasticRanker respectively. Offline evaluation is conducted on a language
understanding benchmark GLUE; and several information retrieval tasks including
Natural Question, Trivia QA, and MS MARCO. The results show that ElasticLM
along with ElasticDenser and ElasticRanker can perform correctly and
competitively compared with an array of static baselines. Furthermore, online
simulation with concurrency is also carried out. The results demonstrate that
ElasticLM can provide elastic tradeoffs with respect to varying request stream.Comment: 27 pages, 11 figures, 9 table
- …
