151 research outputs found
Recommended from our members
LEARNING PROGRAM EMBEDDING FROM UNLABELED SOURCE CODE
No full textMachine-learning models can reach very high performance with supervised training, where they learn from labeled data. However, supervised training requires annotating data with desired output labels, which can be a difficult and time-consuming task. Meanwhile, advancements in deep learning models and technology have made it possible to train very large models, which was not feasible a few years ago. Although training such big models requires a substantial amount of supervised data, models can overcome this limitation by first learning from un-labeled data. Pre-trained language models enable us to achieve state-of-the-art performance from large-scale models with limited supervised data. During the pre-training stage, models are exposed to unlabeled data, and their weights can be adjusted using self-supervised tasks such as filling masks or spans, de- noising artificially induced noises, or simple auto-regressive token generation. These tasks help the model learn token distributions and the context of the programming language. After acquiring this knowledge, the model can be easily fine-tuned or used even without any fine-tuning for a specific task. This thesis began with work on foundation models, which are pre-trained on simple tasks like mask-filling and de-noising, and then are fine-tuned for task-specific applications. We show how to effectively apply pre-trained language models in SE tasks, including traditional ones such as code correction and novel ones such as decompiling binaries. We also investigate the effectiveness of multilingual training and demonstrate how knowledge can be transferred from one language to another, thereby improving the model’s performance on three tasks: code summarization, code search, and method name prediction. We also investigated what foundation models learn during the pre-training stage. It is evident that learning syntactical distribution is relatively easier and can be done using tasks such as MLM. However, it is unclear whether the model learns semantic distribution and is robust with respect to meaning-preserving transformations. In our work, we found that the model learns semantics and is robust to meaning-preserving transformations.More recently, Large language models (LLMs), with a few billion to 540 billion parameters, trained on billions of tokens, have become available. They are proficient at few-shot learning, where just a few examples provided at the time of querying are sufficient. These models have largely eliminated the need for a fine-tuning stage and can perform even with a few training samples or none at all. It’s worth noting that during zero-shot and few-shot learning, we don’t change the parameters of the model; we just use the prompt to divert the model to condition its text generation in a more desirable manner. However, the model’s performance heavily relies on the prompt, and prompt engineering for different tasks has become a focus of attention for numerous researchers. In this work, we demonstrate how to automatically semantically augment prompts for code summarization, achieving state-of-the-art performance on this task
Recommended from our members
Progress in AI Safety via Normative NLP Research and Software Engineering Research
No full textWe are in a period of rapidly increasing machine intelligence. When reviewing progress, we conclude that there is a clear possibility of machines performing a dominant fraction of economic activity within the next few decades (reaching “high level machine intelligence (HLMI)” or “artificial general intelligence (AGI)”), which requires a focus on safety.We contribute to this effort in two parts. In the first part, we focus on “Normative NLP”, or designing Natural Language Processing systems that follow certain norms. First, we recognize that with increasing capabilities, machines are at risk of deceiving humans by pretending to be human (“deceptive anthropomorphism”). We present work on this topic published at prominent NLP venues. In the R-U-A-Robot dataset (2021), we collected over 2,500 phrasings related to the intent of “Are you a robot?”. Even when explicitly asked, we show popular systems of the time often failed to confirm their non-human identity, and we design machine learning classifiers as well as a user study to improve on this. In addition, we contribute the Robots-Dont-Cry dataset (2022), which studies implicit deceptive anthropomorphism. We collect over 900 dialogue turns in popular datasets of the time, showing many are not viewed as possible for a machine. This work has been used by other scientists to study anthropomorphism and robust NLP classifiers.In the second part, we focus on connecting Software Engineering research to AGI safety. We discuss traditional Software Engineering research problems and their connection to AGI safety (2023). We then focus on two problems. First, we contribute techniques for estimating confidence of correctness in code-generating models (2025). This work aims to aid our ability to know when to audit machine outputs. In other work (2020), we study the problem of code summarization, where we characterize datasets of the time and help to improve the rigor of metrics. Faithful and high-quality summaries of complex machine output might also help manage a world where machines are producing vast amounts of complex output.These contributions are presented toward the UC Davis PhD requirements, and add to knowledge for building a better future within AI, NLP, and Software Engineering
Recommended from our members
STraceBert: Source code retrieval using semantic application traces
No full textSoftware reverse engineering is an essential task in software engineering and security, but it can be a challenging process, especially when artifacts are engineered (adversarially) to resist reverse engineering.. To address this challenge, we present STraceBERT, a novel approach that utilizes a Java dynamic analysis tool to record calls to core Java libraries, and pretrain a BERT-style model on the recorded application traces for effective method source code retrieval from a candidate set. Our experiments demonstrate the effectiveness of STraceBERT in retrieving the source code compared to existing approaches. Our proposed approach offers a promising solution to the problem of code retrieval in software reverse engineering and opens up new avenues for further research in this area
Recommended from our members
Modeling Source Code For Developers
No full textSoftware Engineering practices are changing in an age of artificial intelligence. While the core activities of design, develop, maintain, test and evaluate remain, the methods used in these activities are evolving. The prevalence of generative programming models has the potential to reconstitute the duties of a software engineer. Widely adopted models like Copilot and Bard are IDE-based pair-programming assistants that create code from virtually any input: contextual code, natural language, specifications, input output pairs, etc. The way developers interact with these models will redefine some core ideas of software engineering. These models empower virtually anyone, of varying coding proficiency, to create software. Models with the capacity to code will surely manage to inherit software design and analysis capabilities [186], albeit for now, with specific training or prompting.Naturally, one wonders how language modeling, or more specifically the modeling of source code and its features, will impact developers. Researchers often conjecture on the varying degree of influence these methods will have, but certainly, these tools will support developers in new and existing tasks: code completion, bug and vulnerability detection, code summarization, type annotation, and more are already prominent use cases. One can envision a world where software developers delegate portions of their work to machine learning pipelines, such as unit testing and vulnerability testing of their code; how much of that code they actually write is up for debate as well. Developers will likely automate portions of their work flow but simultaneously gain new tasks and responsibilities. These tasks might include passing automatic code reviews that detect code smells, place code comments automatically, and detect refactorings; maybe using models from [56], [133], [59]. These capabilities come from modeling source code and its features directly by distilling down meaningful representations for the task at hand.This thesis explores learning meaningful representations from code through a variety of applications for developer supporting tools. The first application is a type-prediction model using representations learned with masked-language-modeling. While effective, we find that the off-the-shelf model fails at an aspect of modeling source code, namely the use of local user-defined types. The next application modifies the model learned representations with one characterized by an objective function capturing how developers actually use types. Along this body of work, the next two chapters present a type inference dataset for the community and a framework for new machine learning models with a Visual Studio plugin. This thesis concludes with a study of large language models on single statement bug introduction and proposes avoidance strategies. Finally I present some future work to improve these models. By reading this thesis, I hope the reader has a few takeaways:(1) Machine learning is an essential tool for capturing code and its meta data. Models trained on code and its features are capable of generalizing and improving old and new processes.(2) The data that models train on is not perfect, and the resulting models often inherit biases towards vulnerable and buggy code; researchers must evaluate the risk vs. reward with broadly trained models.(3) The objectives optimized for software models may not align with our goals; models that incorporate human feedback may ultimately align better to our values and understanding of code.(4) Large language models are powerful tools for software engineering, but they’re only part of the picture. Models that learn data and control flow, project and file meta data, local and global scope semantics, and information associated with code traces, are better informed on the source code it consumes and produces.This thesis attempts to quantify the utility of off-the-shelf LLMs like BERT, the misalignment of LLM representations to human derived representations of coding constructs, and the present risks of using LLM predictions at face value. Hopefully, in each case, the chapters leave you optimistic that many of the aforementioned concerns can be minimized, or mitigated with just a bit of ingenuity
New Initiative: The Naturalness of Software
No full textAbstract—This paper describes a new research consortium, studying the Naturalness of Software. This initiative is supported by a pair of grants by the US National Science Foundation, totaling 600,000 helped kickstart an inter-disciplinary effort, and demonstrate feasibility; a follow-on full grant of $2,000,000 was recently awarded. The initiative is led by the author, who is at UC Davis, and includes investigators from Iowa State University and Carnegie-Mellon University (Language Technologies Institute). I
TestNMT: Function-to-test neural machine translation
No full textTest generation can have a large impact on the software engineering process by decreasing the amount of time and effort required to maintain a high level of test coverage. This increases the quality of the resultant software while decreasing the associated effort. In this paper, we present TestNMT, an experimental approach to test generation using neural machine translation. TestNMT aims to learn to translate from functions to tests, allowing a developer to generate an approximate test for a given function, which can then be adapted to produce the final desired test. We also present a preliminary quantitative and qualitative evaluation of TestNMT in both cross-project and within-project scenarios. This evaluation shows that TestNMT is potentially useful in the within-project scenario, where it achieves a maximum BLEU score of 21.2, a maximum ROUGE-L score of 38.67, and is shown to be capable of generating approximate tests that are easy to adapt to working tests
Converging Work-Talk Patterns in Online Task-Oriented Communities.
No full textMuch of what we do is accomplished by working collaboratively with others, and a large portion of our lives are spent working and talking; the patterns embodied in the alternation of working and talking can provide much useful insight into task-oriented social behaviors. The available electronic traces of the different kinds of human activities in online communities are an empirical goldmine that can enable the holistic study and understanding of these social systems. Open Source Software (OSS) projects are prototypical examples of collaborative, task-oriented communities, depending on volunteers for high-quality work. Here, we use sequence analysis methods to identify the work-talk patterns of software developers in online communities of Open Source Software projects. We find that software developers prefer to persist in same kinds of activities, i.e., a string of work activities followed by a string of talk activities and so forth, rather than switch them frequently; this tendency strengthens with time, suggesting that developers become more efficient, and can work longer with fewer interruptions. This process is accompanied by the formation of community culture: developers' patterns in the same communities get closer with time while different communities get relatively more different. The emergence of community culture is apparently driven by both "talk" and "work". Finally, we also find that workers with good balance between "work" and "talk" tend to produce just as much work as those that focus strongly on "work"; however, the former appear to be more likely to continue to be active contributors in the communities
GENOA—a customizable, front-end-retargetable source code analysis framework
No full text
Code analysis
tools provide support for such software engineering tasks as program understanding, software metrics, testing, and reengineering. In this article we describe GENOA, the framework underlying application generators such as Aria and GEN++ which have been used to generate a wide range of practical code analysis tools. This experience illustrates
front-end retargetability
of GENOA; we describe the features of the GENOA framework that allow it to be used with different front ends. While permitting arbitrary parse tree computations, the GENOA specification language has special, compact iteration operators that are tuned for expressing simple, polynomial-time analysis programs; in fact, there is a useful sublanguage of the GENOA language that can express precisely all (and only)
polynomial-time
(PTIME) analysis programs on parse trees. Thus, we argue that the GENOA language is a simple and convenient vehicle for implementing a range of analysis tools. We also argue that the “front-and reuse” approach of GENOA offers an important advantage for tools aimed at large software projects: the reuse of complex, expensive build procedures to run generated tools over large source bases. In this article, we describe the GENOA framework and our experiences with it.
</jats:p
- …
