24,530 research outputs found
“Estimating Software Project Effort Using Analogies”: Reflections After 28 Years
No full textA version of the article is available at arXiv:2501.14582v2 [cs.SE] (https://arxiv.org/abs/2501.14582). Comments: 5 pages, invited and accepted IEEE TSE paper for the journal's 50th year anniversary on most influential papers. (This version corrects three typos.). Submission history
From: Martin Shepperd: [v1] Fri, 24 Jan 2025 15:44:25 UTC (14 KB); [v2] Thu, 30 Jan 2025 16:44:38 UTC (14 KB).This invited paper is the result of an invitation to write a retrospective article on a “TSE most influential paper” as part of the journal's 50th anniversary. The objective is to reflect on the progress of software engineering prediction research using the lens of a selected, highly cited research paper and 28 years of hindsight. The paper examines (i) what was achieved, (ii) what has endured and (iii) what could have been done differently with the benefit of retrospection. While many specifics of software project effort prediction have evolved, key methodological issues remain relevant. The original study emphasised empirical validation with benchmarks, out-of-sample testing and data/tool sharing. Four areas for improvement are identified: (i) stronger commitment to Open Science principles, (ii) focus on effect sizes and confidence intervals, (iii) reporting variability alongside typical results and (iv) more rigorous examination of threats to validity
Software project economics: A roadmap
No full textThe objective of this paper is to consider research progress in the field of software project economics with a view to identifying important challenges and promising research directions. I argue that this is an important sub-discipline since this will underpin any cost-benefit analysis used to justify the resourcing, or otherwise, of a software project. To accomplish this I conducted a bibliometric analysis of peer reviewed research articles to identify major areas of activity. My results indicate that the primary goal of more accurate cost prediction systems remains largely unachieved. However, there are a number of new and promising avenues of research including: how we can combine results from primary studies, integration of multiple predictions and applying greater emphasis upon the human aspects of prediction tasks. I conclude that the field is likely to remain very challenging due to the people-centric nature of software engineering, since it is in essence a design task. Nevertheless the need for good economic models will grow rather than diminish as software becomes increasingly ubiquitous
Four commentaries on the use of students and professionals in empirical software engineering experiments
No full textThe relative pros and cons of using students or practitioners in experiments in empirical software engineering have been discussed for a long time and continue to be an important topic. Following the recent publication of “Empirical software engineering experts on the use of students and professionals in experiments” by Falessi, Juristo, Wohlin, Turhan, Münch, Jedlitschka, and Oivo (EMSE, February 2018) we received a commentary by Sjøberg and Bergersen. Given that the topic is of great methodological interest to the community and requires nuanced treatment, we invited two editorial board members, Martin Shepperd and Per Runeson, respectively, to provide additional views
Exploring Student Engagement and Outcomes: Experiences from Three Cycles of an Undergraduate Module
No full textMany studies in educational data mining address specific learner groups, such
as first-in-family to attend Higher Education, or focus on differences in
characteristics such as gender or ethnicity, with the aim of predicting
performance and designing interventions to improve outcomes. For Higher
Education, this is reflected in significant interest in institutional-level
analysis of student cohorts and in tools being promoted to Higher Education
Institutions to support collection, integration and analysis of data. For those
leading modules/units on degree programmes, however, the reality can be far
removed from the seemingly well-supported and increasingly sophisticated
approaches advocated in centrally-led data analysis. Module leaders often find
themselves working with a number of student-data systems that are not
integrated, may contain conflicting data and where significant effort is
required to extract, clean and meaningfully analyse the data. This paper
suggests that important lessons may be learned from experiences at module level
in this context and from subsequent analysis of related data collected across
multiple years. The changes made each year are described and a range of data
analysis methods are applied, post hoc, to identify findings in relation to the
four areas of focus. The key findings are that non-engagement with the Virtual
Learning Environment in the first three weeks was the strongest predictor of
failure and that early engagement correlated most strongly with final grade.
General recommendations are drawn from the findings which should be valuable to
module leaders in environments where access to integrated, up-to-date student
information remains a day-to-day challenge, and insights will be presented into
how such bottom-up activities might inform institutional/top-down planning in
the use of relevant technologies
The scientific basis for prediction research
No full textCopyright @ 2012 ACMIn recent years there has been a huge growth in using statistical and machine learning methods to find useful prediction systems for software engineers. Of particular interest is predicting project effort and duration and defect behaviour. Unfortunately though results are often promising no single technique dominates and there are clearly complex interactions between technique, training methods and the problem domain. Since we lack deep theory our research is of necessity experimental. Minimally, as scientists, we need reproducible studies. We also need comparable studies. I will show through a meta-analysis of many primary studies that we are not presently in that situation and so the scientific basis for our collective research remains in doubt. By way of remedy I will argue that we need to address these issues of reporting protocols and expertise plus ensure blind analysis is routine
Making inferences with small numbers of training sets
No full textA potential methodological problem with empirical studies that assess project effort prediction system is discussed. Frequently, a hold-out strategy is deployed so that the data set is split into a training and a validation set. Inferences are then made concerning the relative accuracy of the different prediction techniques under examination. This is typically done on very small numbers of sampled training sets. It is shown that such studies can lead to almost random results (particularly where relatively small effects are being studied). To illustrate this problem, two data sets are analysed using a configuration problem for case-based prediction and results generated from 100 training sets. This enables results to be produced with quantified confidence limits. From this it is concluded that in both cases using less than five training sets leads to untrustworthy results, and ideally more than 20 sets should be deployed. Unfortunately, this raises a question over a number of empirical validations of prediction techniques, and so it is suggested that further research is needed as a matter of urgency
The consistency of empirical comparisons of regression and analogy-based software project cost prediction
No full textOBJECTIVE - to determine the consistency within and between results in empirical studies of software engineering cost estimation. We focus on regression and analogy techniques as these are commonly used. METHOD – we conducted an exhaustive search using predefined inclusion and exclusion criteria and identified 67 journal papers and 104 conference papers. From this sample we identified 11 journal papers and 9 conference papers that used both methods. RESULTS – our analysis found that about 25% of studies were internally inconclusive. We also found that there is approximately equal evidence in favour of, and against analogy-based methods. CONCLUSIONS – we confirm the lack of consistency in the findings and argue that this inconsistent pattern from 20 different studies comparing regression and analogy is somewhat disturbing. It suggests that we need to ask more detailed questions than just: “What is the best prediction system?
Feature weighting techniques for CBR in software effort estimation studies: A review and empirical evaluation
No full textContext : Software effort estimation is one of the most important activities in the software development process. Unfortunately, estimates are often substantially wrong. Numerous estimation methods have been proposed including Case-based Reasoning (CBR). In order to improve CBR estimation accuracy, many researchers have proposed feature weighting techniques (FWT). Objective: Our purpose is to systematically review the empirical evidence to determine whether FWT leads to improved predictions. In addition we evaluate these techniques from the perspectives of (i) approach (ii) strengths and weaknesses (iii) performance and (iv) experimental evaluation approach including the data sets used. Method: We conducted a systematic literature review of published, refereed primary studies on FWT (2000-2014). Results: We identified 19 relevant primary studies. These reported a range of different techniques. 17 out of 19 make benchmark comparisons with standard CBR and 16 out of 17 studies report improved accuracy. Using a one-sample sign test this positive impact is significant (p = 0:0003). Conclusion: The actionable conclusion from this study is that our review of all relevant empirical evidence supports the use of FWTs and we recommend that researchers and practitioners give serious consideration to their adoption
Jack Alive / Martin Dead : The Location of the "Author" in Jack London\u27s Martin Eden
No full textThis essay is an attempt to read Martin Eden, Jack Londonʼs autobiographical novel, in terms of the inextricable relationship between the author and the protagonist. Critics have often taken the unbalanced plot and the lack of ironic distance between narrator and character in Martin Eden as the technical weakness of London, but this paper argues that the achievement of this novel owes a great deal to the attachment of London to Martin. The unbalanced structure is a necessary product of the severe struggle of the author to kill his romantic alter ego. // Martin, who aspires to win Ruth Morse, tries to cross class boundaries by making a career of a writer. Even after realizing the emptiness of Ruth, who turns out to be nothing but a typical figure of the bourgeoisie, he somehow persists in loving her. The notion underlying here is that, for Martin, love, career and art are fundamentally inseparable. He objects to the aestheteʼs view of Brissenden on account of his separation of art from career. Martinʼs identity and life consist only in the triunity of love/career/art; the alternative is the repudiation of life. Thus, the unnatural delay of his disappointment in love can be regarded as Londonʼs strategy to set the suicide of Martin as the necessary consequence of the story. // By finishing the story and killing Martin, London finally detaches himself from Martin, reconstructs his self, and, unlike Martin, survives as a professional writer. In this sense, Martin Eden is a story about “writerʼs self-reconstruction.
An Investigation of Rule Induction Based Prediction Systems
No full textTraditionally, researchers have used either off-the-shelf models such as COCOMO, or developed local models using statistical techniques such as stepwise regression, to predict software effort estimates. More recently, attention has turned to a variety of machine learning methods such as artificial neural networks (ANNs), case-based reasoning (CBR) and rule induction (RI). This position paper outlines some preliminary research into the use of rule induction methods to build software cost models. We briefly describe the use of rule induction methods and then apply the technique to a dataset of 81 software projects derived from a Canadian software house in the late 1980s. We show that RI methods tend to be unstable and generally predict with quite variable accuracy. Pruning the feature set, however, has a significant impact upon accuracy. We also compare our results with a prediction system based upon a standard regression procedure. We suggest that further work is carried out to examine the effects of the relationships among, and between, the features of the attributes on the generated rules in an attempt to improve on current prediction techniques and enhance our understanding of machine learning methods
- …
