1,720,988 research outputs found
Recommended from our members
LEARNING PROGRAM EMBEDDING FROM UNLABELED SOURCE CODE
Get PDFMachine-learning models can reach very high performance with supervised training, where they learn from labeled data. However, supervised training requires annotating data with desired output labels, which can be a difficult and time-consuming task. Meanwhile, advancements in deep learning models and technology have made it possible to train very large models, which was not feasible a few years ago. Although training such big models requires a substantial amount of supervised data, models can overcome this limitation by first learning from un-labeled data. Pre-trained language models enable us to achieve state-of-the-art performance from large-scale models with limited supervised data. During the pre-training stage, models are exposed to unlabeled data, and their weights can be adjusted using self-supervised tasks such as filling masks or spans, de- noising artificially induced noises, or simple auto-regressive token generation. These tasks help the model learn token distributions and the context of the programming language. After acquiring this knowledge, the model can be easily fine-tuned or used even without any fine-tuning for a specific task. This thesis began with work on foundation models, which are pre-trained on simple tasks like mask-filling and de-noising, and then are fine-tuned for task-specific applications. We show how to effectively apply pre-trained language models in SE tasks, including traditional ones such as code correction and novel ones such as decompiling binaries. We also investigate the effectiveness of multilingual training and demonstrate how knowledge can be transferred from one language to another, thereby improving the model’s performance on three tasks: code summarization, code search, and method name prediction. We also investigated what foundation models learn during the pre-training stage. It is evident that learning syntactical distribution is relatively easier and can be done using tasks such as MLM. However, it is unclear whether the model learns semantic distribution and is robust with respect to meaning-preserving transformations. In our work, we found that the model learns semantics and is robust to meaning-preserving transformations.More recently, Large language models (LLMs), with a few billion to 540 billion parameters, trained on billions of tokens, have become available. They are proficient at few-shot learning, where just a few examples provided at the time of querying are sufficient. These models have largely eliminated the need for a fine-tuning stage and can perform even with a few training samples or none at all. It’s worth noting that during zero-shot and few-shot learning, we don’t change the parameters of the model; we just use the prompt to divert the model to condition its text generation in a more desirable manner. However, the model’s performance heavily relies on the prompt, and prompt engineering for different tasks has become a focus of attention for numerous researchers. In this work, we demonstrate how to automatically semantically augment prompts for code summarization, achieving state-of-the-art performance on this task
Recommended from our members
Progress in AI Safety via Normative NLP Research and Software Engineering Research
Get PDFWe are in a period of rapidly increasing machine intelligence. When reviewing progress, we conclude that there is a clear possibility of machines performing a dominant fraction of economic activity within the next few decades (reaching “high level machine intelligence (HLMI)” or “artificial general intelligence (AGI)”), which requires a focus on safety.We contribute to this effort in two parts. In the first part, we focus on “Normative NLP”, or designing Natural Language Processing systems that follow certain norms. First, we recognize that with increasing capabilities, machines are at risk of deceiving humans by pretending to be human (“deceptive anthropomorphism”). We present work on this topic published at prominent NLP venues. In the R-U-A-Robot dataset (2021), we collected over 2,500 phrasings related to the intent of “Are you a robot?”. Even when explicitly asked, we show popular systems of the time often failed to confirm their non-human identity, and we design machine learning classifiers as well as a user study to improve on this. In addition, we contribute the Robots-Dont-Cry dataset (2022), which studies implicit deceptive anthropomorphism. We collect over 900 dialogue turns in popular datasets of the time, showing many are not viewed as possible for a machine. This work has been used by other scientists to study anthropomorphism and robust NLP classifiers.In the second part, we focus on connecting Software Engineering research to AGI safety. We discuss traditional Software Engineering research problems and their connection to AGI safety (2023). We then focus on two problems. First, we contribute techniques for estimating confidence of correctness in code-generating models (2025). This work aims to aid our ability to know when to audit machine outputs. In other work (2020), we study the problem of code summarization, where we characterize datasets of the time and help to improve the rigor of metrics. Faithful and high-quality summaries of complex machine output might also help manage a world where machines are producing vast amounts of complex output.These contributions are presented toward the UC Davis PhD requirements, and add to knowledge for building a better future within AI, NLP, and Software Engineering
Recommended from our members
STraceBert: Source code retrieval using semantic application traces
Get PDFSoftware reverse engineering is an essential task in software engineering and security, but it can be a challenging process, especially when artifacts are engineered (adversarially) to resist reverse engineering.. To address this challenge, we present STraceBERT, a novel approach that utilizes a Java dynamic analysis tool to record calls to core Java libraries, and pretrain a BERT-style model on the recorded application traces for effective method source code retrieval from a candidate set. Our experiments demonstrate the effectiveness of STraceBERT in retrieving the source code compared to existing approaches. Our proposed approach offers a promising solution to the problem of code retrieval in software reverse engineering and opens up new avenues for further research in this area
Recommended from our members
Modeling Source Code For Developers
Get PDFSoftware Engineering practices are changing in an age of artificial intelligence. While the core activities of design, develop, maintain, test and evaluate remain, the methods used in these activities are evolving. The prevalence of generative programming models has the potential to reconstitute the duties of a software engineer. Widely adopted models like Copilot and Bard are IDE-based pair-programming assistants that create code from virtually any input: contextual code, natural language, specifications, input output pairs, etc. The way developers interact with these models will redefine some core ideas of software engineering. These models empower virtually anyone, of varying coding proficiency, to create software. Models with the capacity to code will surely manage to inherit software design and analysis capabilities [186], albeit for now, with specific training or prompting.Naturally, one wonders how language modeling, or more specifically the modeling of source code and its features, will impact developers. Researchers often conjecture on the varying degree of influence these methods will have, but certainly, these tools will support developers in new and existing tasks: code completion, bug and vulnerability detection, code summarization, type annotation, and more are already prominent use cases. One can envision a world where software developers delegate portions of their work to machine learning pipelines, such as unit testing and vulnerability testing of their code; how much of that code they actually write is up for debate as well. Developers will likely automate portions of their work flow but simultaneously gain new tasks and responsibilities. These tasks might include passing automatic code reviews that detect code smells, place code comments automatically, and detect refactorings; maybe using models from [56], [133], [59]. These capabilities come from modeling source code and its features directly by distilling down meaningful representations for the task at hand.This thesis explores learning meaningful representations from code through a variety of applications for developer supporting tools. The first application is a type-prediction model using representations learned with masked-language-modeling. While effective, we find that the off-the-shelf model fails at an aspect of modeling source code, namely the use of local user-defined types. The next application modifies the model learned representations with one characterized by an objective function capturing how developers actually use types. Along this body of work, the next two chapters present a type inference dataset for the community and a framework for new machine learning models with a Visual Studio plugin. This thesis concludes with a study of large language models on single statement bug introduction and proposes avoidance strategies. Finally I present some future work to improve these models. By reading this thesis, I hope the reader has a few takeaways:(1) Machine learning is an essential tool for capturing code and its meta data. Models trained on code and its features are capable of generalizing and improving old and new processes.(2) The data that models train on is not perfect, and the resulting models often inherit biases towards vulnerable and buggy code; researchers must evaluate the risk vs. reward with broadly trained models.(3) The objectives optimized for software models may not align with our goals; models that incorporate human feedback may ultimately align better to our values and understanding of code.(4) Large language models are powerful tools for software engineering, but they’re only part of the picture. Models that learn data and control flow, project and file meta data, local and global scope semantics, and information associated with code traces, are better informed on the source code it consumes and produces.This thesis attempts to quantify the utility of off-the-shelf LLMs like BERT, the misalignment of LLM representations to human derived representations of coding constructs, and the present risks of using LLM predictions at face value. Hopefully, in each case, the chapters leave you optimistic that many of the aforementioned concerns can be minimized, or mitigated with just a bit of ingenuity
TestNMT: Function-to-test neural machine translation
Get PDFTest generation can have a large impact on the software engineering process by decreasing the amount of time and effort required to maintain a high level of test coverage. This increases the quality of the resultant software while decreasing the associated effort. In this paper, we present TestNMT, an experimental approach to test generation using neural machine translation. TestNMT aims to learn to translate from functions to tests, allowing a developer to generate an approximate test for a given function, which can then be adapted to produce the final desired test. We also present a preliminary quantitative and qualitative evaluation of TestNMT in both cross-project and within-project scenarios. This evaluation shows that TestNMT is potentially useful in the within-project scenario, where it achieves a maximum BLEU score of 21.2, a maximum ROUGE-L score of 38.67, and is shown to be capable of generating approximate tests that are easy to adapt to working tests
Going Beyond Counting First Authors in Author Co-citation Analysis
Get PDFThe present study examines one of the fundamental aspects of author co-citation analysis (ACA) - the way co-citation
counts are defined. Co-citation counting provides the data on which all subsequent statistical analyses and mappings
are based, and we compare ACA results based on two different types of co-citation counting - the traditional type that
only counts the first one among a cited work's authors on the one hand and a non-traditional type that takes into
account the first 5 authors of a cited work on the other hand. Results indicate that the picture produced through this non-traditional author co-citation counting contains more coherent author groups and is therefore considerably clearer. However, this picture represents fewer specialties in the research field being studied than that produced through the traditional first-author co-citation counting when the same number of top-ranked authors is selected and analyzed. Reasons for these effects are discussed
Better patching using LLM prompting, via Self-Consistency
Get PDFLarge Language models (LLMs) can be induced to solve non-trivial problems
with "few-shot" prompts including illustrative problem-solution examples. Now
if the few-shots also include "chain of thought" (CoT) explanations, which are
of the form problem-explanation-solution, LLMs will generate a "explained"
solution, and perform even better. Recently an exciting, substantially better
technique, self-consistency [1] (S-C) has emerged, based on the intuition that
there are many plausible explanations for the right solution; when the LLM is
sampled repeatedly to generate a pool of explanation-solution pairs, for a
given problem, the most frequently occurring solutions in the pool (ignoring
the explanations) tend to be even more likely to be correct! Unfortunately, the
use of this highly-performant S-C (or even CoT) approach in software
engineering settings is hampered by the lack of explanations; most software
datasets lack explanations. In this paper, we describe an application of the
S-C approach to program repair, using the commit log on the fix as the
explanation, only in the illustrative few-shots. We achieve state-of-the art
results, beating previous approaches to prompting-based program repair, on the
MODIT dataset; we also find evidence suggesting that the correct commit
messages are helping the LLM learn to produce better patches.Comment: Accepted at ASE-NIER (2023) trac
Variations on the Author
Get PDF“Variations on the Author” discusses two of Eduardo Coutinho’s recent films (Um Dia na Vida, from 2010, and Últimas Conversas, posthumously released in 2015) and their contribution to the general question of documentary authorship. The director’s filmography is characterized by a consistent yet self-effacing form of authorial self-inscription: Coutinho often features as an interviewer that rather than express opinions propels discourses; an interviewer that is good at listening. This mode of self-inscription characterizes him as an author who is not expressive but who is nonetheless markedly present on the screen. In Um Dia na Vida, however, Coutinho is completely absent form the image, while Últimas Conversas, on the contrary, includes a confessional prologue that moves the director from the margins to the center of his films. This article examines the ways in which these works stand out in the filmography of a director who offers new insights into the notion of cinematic authorship
- …
