94 research outputs found
Leveraging synthetic trace generation of modeling operations for intelligent modeling assistants using large language models
No full textContext: Due to the proliferation of generative AI models in different software engineering tasks, the research community has started to exploit those models, spanning from requirement specification to code development. Model-Driven Engineering (MDE) is a paradigm that leverages software models as primary artifacts to automate tasks. In this respect, modelers have started to investigate the interplay between traditional MDE practices and Large Language Models (LLMs) to push automation. Although powerful, LLMs exhibit limitations that undermine the quality of generated modeling artifacts, e.g., hallucination or incorrect formatting. Recording modeling operations relies on human-based activities to train modeling assistants, helping modelers in their daily tasks. Nevertheless, those techniques require a huge amount of training data that cannot be available due to several factors, e.g., security or privacy issues. Objective: In this paper, we propose an extension of a conceptual MDE framework, called MASTER-LLM, that combines different MDE tools and paradigms to support industrial and academic practitioners. Method: MASTER-LLM comprises a modeling environment that acts as the active context in which a dedicated component records modeling operations. Then, model completion is enabled by the modeling assistant trained on past operations. Different LLMs are used to generate a new dataset of modeling events to speed up recording and data collection. Results: To evaluate the feasibility of MASTER-LLM in practice, we experiment with two modeling environments, i.e., CAEX and HEPSYCODE, employed in industrial use cases within European projects. We investigate how the examined LLMs can generate realistic modeling operations in different domains. Conclusion: We show that synthetic traces can be effectively used when the application domain is less complex, while complex scenarios require human-based operations or a mixed approach according to data availability. However, generative AI models must be assessed using proper methodologies to avoid security issues in industrial domains
Foster the use of Hackathons in Collaborative Research Projects: Methodology, Experience Report and Lesson Learned
No full textGPTSniffer: A CodeBERT-based classifier to detect source code written by ChatGPT
No full textSince its launch in November 2022, ChatGPT has gained popularity among users, especially programmers who use it to solve development issues. However, while offering a practical solution to programming problems, ChatGPT should be used primarily as a supporting tool (e.g., in software education) rather than as a replacement for humans. Thus, detecting automatically generated source code by ChatGPT is necessary, and tools for identifying AI -generated content need to be adapted to work effectively with code. This paper presents GPTSniffer- a novel approach to the detection of source code written by AI - built on top of CodeBERT. We conducted an empirical study to investigate the feasibility of automated identification of AI -generated code, and the factors that influence this ability. The results show that GPTSniffer can accurately classify whether code is human -written or AI -generated, outperforming two baselines, GPTZero and OpenAI Text Classifier. Also, the study shows how similar training data or a classification context with paired snippets helps boost the prediction. We conclude that GPTSniffer can be leveraged in different contexts, e.g., in software engineering education, where teachers use the tool to detect cheating and plagiarism, or in development, where AI -generated code may require peculiar quality assurance activities
Prompt engineering and its implications on the energy consumption of Large Language Models
No full textReducing the environmental impact of AI-based software systems has become critical. The intensive use of large language models (LLMs) in software engineering poses severe challenges regarding computational resources, data centers, and carbon emissions. In this paper, we investigate how prompt engineering techniques (PETs) can impact the carbon emission of the Llama 3 model for the code generation task. We experimented with the CodeXGLUE benchmark to evaluate both energy consumption and the accuracy of the generated code using an isolated testing environment. Our initial results show that the energy consumption of LLMs can be reduced by using specific tags that distinguish different prompt parts. Even though a more in-depth evaluation is needed to confirm our findings, this work suggests that prompt engineering can reduce LLMs' energy consumption during the inference phase without compromising performance, paving the way for further investigations
Measurement of top quark pairs production cross-section in the semi-leptonic channel with the ATLAS experiment
No full textThis thesis is about three major aspects of the identification of top quarks. First comes the understanding of their production mechanism, their decay channels and how to translate theoretical formulae into programs that can simulate such physical processes using Monte Carlo techniques. In particular, the author has been involved in the introduction of the POWHEG generator in the framework of the ATLAS experiment. POWHEG is now fully used as the benchmark program for the simulation of ttbar pairs production and decay, along with MC@NLO and AcerMC: this will be shown in chapter one. The second chapter illustrates the ATLAS detectors and its sub-units, such as calorimeters and muon chambers. It is very important to evaluate their efficiency in order to fully understand what happens during the passage of radiation through the detector and to use this knowledge in the calculation of final quantities such as the ttbar production cross section. The last part of this thesis concerns the evaluation of this quantity deploying the so-called "golden channel" of ttbar decays, yielding one energetic charged lepton, four particle jets and a relevant quantity of missing transverse energy due to the neutrino. The most important systematic errors arising from the various part of the calculation are studied in detail. Jet energy scale, trigger efficiency, Monte Carlo models, reconstruction algorithms and luminosity measurement are examples of what can contribute to the uncertainty about the cross-section
HybridRec: A recommender system for tagging GitHub repositories
No full textAbstractSoftware repositories are increasingly essential to support the management of typical artifacts building up projects, including source code, documentation, and bug reports. GitHub is at the forefront of this kind of platforms, providing developer with a reservoir of code contained in more than 28M repositories. To help developers find the right artifacts, GitHub uses topics, which are short texts assigned to the stored artifacts. However, assigning inappropriate topics to a repository might hamper its popularity and reachability. In our previous work, we implemented MNBN and TopFilter to recommend GitHub topics. MNBN exploits a stochastic network to predict topics, while TopFilter relies on a syntactic-based function to recommend topics. In this paper, we extend our work by building HybridRec, a recommender system based on stochastic and collaborative-filtering techniques to generate more relevant topics. To deal with unbalanced datasets, we employ a Complement Naïve Bayesian Network (CNBN). Furthermore, we apply a preprocessing phase to clean and refine the input data before feeding the recommendation engine. An empirical evaluation demonstrates that HybridRec outperforms three state-of-the-art baselines, obtaining a better performance with respect to various metrics. We conclude that the conceived framework can be used to help developers increase their projects’ visibility.</jats:p
Towards Synthetic Trace Generation of Modeling Operations using In-Context Learning Approach
No full textProducing accurate software models is crucial in model-driven software
engineering (MDE). However, modeling complex systems is an error-prone task
that requires deep application domain knowledge. In the past decade, several
automated techniques have been proposed to support academic and industrial
practitioners by providing relevant modeling operations. Nevertheless, those
techniques require a huge amount of training data that cannot be available due
to several factors, e.g., privacy issues. The advent of large language models
(LLMs) can support the generation of synthetic data although state-of-the-art
approaches are not yet supporting the generation of modeling operations. To
fill the gap, we propose a conceptual framework that combines modeling event
logs, intelligent modeling assistants, and the generation of modeling
operations using LLMs. In particular, the architecture comprises modeling
components that help the designer specify the system, record its operation
within a graphical modeling environment, and automatically recommend relevant
operations. In addition, we generate a completely new dataset of modeling
events by telling on the most prominent LLMs currently available. As a proof of
concept, we instantiate the proposed framework using a set of existing modeling
tools employed in industrial use cases within different European projects. To
assess the proposed methodology, we first evaluate the capability of the
examined LLMs to generate realistic modeling operations by relying on
well-founded distance metrics. Then, we evaluate the recommended operations by
considering real-world industrial modeling artifacts. Our findings demonstrate
that LLMs can generate modeling events even though the overall accuracy is
higher when considering human-based operations
On the use of large language models in model-driven engineering
No full textModel-driven engineering (MDE) has seen significant advancements with the integration of machine learning (ML) and deep learning techniques. Building upon the groundwork of previous investigations, our study provides a concise overview of current large language models (LLMs) applications in MDE, emphasizing their role in automating tasks like model repository classification and developing advanced recommender systems. The paper also outlines the technical considerations for seamlessly integrating LLMs in MDE, offering a practical guide for researchers and practitioners. Looking forward, the paper proposes a focused research agenda for the future interplay of LLMs and MDE, identifying key challenges and opportunities. This concise roadmap envisions the deployment of LLM techniques to enhance the management, exploration, and evolution of modeling ecosystems. Moreover, we also discuss the adoption of LLMs in various domains by means of model-driven techniques and tools, i.e., MDE for supporting LLMs. By offering a compact exploration of LLMs in MDE, this paper contributes to the ongoing evolution of MDE practices, providing a forward-looking perspective on the transformative role of large language models in software engineering and model-driven practices
Dealing with Popularity Bias in Recommender Systems for Third-party Libraries: How far Are We?
No full textRecommender systems for software engineering (RSSEs) assist software engineers in dealing with a growing information overload when discerning alternative development solutions. While RSSEs are becoming more and more effective in suggesting handy recommendations, they tend to suffer from popularity bias, i.e., favoring items that are relevant mainly because several developers are using them. While this rewards artifacts that are likely more reliable and well-documented, it would also mean that missing artifacts are rarely used because they are very specific or more recent. This paper studies popularity bias in Third-Party Library (TPL) RSSEs. First, we investigate whether state-of-the-art research in RSSEs has already tackled the issue of popularity bias. Then, we quantitatively assess four existing TPL RSSEs, exploring their capability to deal with the recommendation of popular items. Finally, we propose a mechanism to defuse popularity bias in the recommendation list. The empirical study reveals that the issue of dealing with popularity in TPL RSSEs has not received adequate attention from the software engineering community. Among the surveyed work, only one starts investigating the issue, albeit getting a low prediction performance
- …
