Evaluating Dynamic Analysis Techniques for Program Comprehension
http://www.st.ewi.tudelft.nl/~cornel/pub/PhD_thesis_BasCornelissen.pdf
Introduction
Program comprehension is an essential part of software development and software maintenance, as software must be sufficiently understood before it can be properly modified. one of the common approaches in getting to understand a program is the study of its execution, also known as dynamic analysis. While many such approaches have been proposed in the literature, their empirical evalucation is, as we will see, often missing or inconclusive, and their added values to existing solutions are seldomly quantified.
* 기존의 Dynamic analysis 연구가 사실 미흡한 점들이 많았다는 것을 설명한다.
This dissertation aims to characterize, and extend upon, the existing work on program comprehension through dynamic analysis. A strong emphasis in our research is put on empirical evaluation. To structure the current state of the art and to identify research opportunities, we first conduct a systematic survey of all publications on this topic in the past decades. We then propose both a traditional and a more advanced visualization technique, discuss the necessary abstractoin techniques, and provide tool implementations. Our approaches are validated through extensive case studies and a controlled experiment. Finally, we conclude with a set of recommendations for future directions.
* 과거의 많은 연구들을 분석하여 트랜드 및 연구 요소를 분석하고 새로운 기술을 소개하겠다고 설명한다.
A Survery of Program Comprehension through Dynamic Analysis
Program comprehension is an important activity in software maintenance, as software must be sufficiently understood before it can be properly modified. The study of a program's execution, known as dynamic analysis, has become a common technique in this respect and has received substantial attention from the research community, particularly over the last decade. These efforts have resulted in a large research body of which currently there exists no comprehensive overview. This chapter reports on a systematic literature survey amied at the identification and structuring of research on program comprehension through dynamic analysis. From a research body consisting of 4,795 articles published in 14 relevant venues between July 1999 and June 2008 and the references therein, we have ystematically selected 172 articles and characterized them in terms of four main facets: activity, target, method, and evaluation. The resulting overview offers insight in what constitutes the main contributions of the field, supports the task of identifying gaps and opportunities, and has motivated our discussion of several important research directions that merit additional consideration in the near future.
* 과거의 연구를 분석하여 앞으로 연구가 진행되어야할 방향을 제시하였다.
Sequence Diagram Reconstruction
In this chapter, we elaborate on our first efforts in supporting the program comprehension process by means of dynamic analysis. Our focus here is on traditional visualization techniques, as they are readily available and because we expect most people to be accustomed to them. Inspired upon the popularity of testing during, e.g., Agile software development methods, we focus on a system's test suite as a starting point for knowledge gathering. We propose to use a variant of UML sequence diagrams to visualize the information obtained from the execution of such a test suite, and employ a series of abstraction techniques to make the resuling diagrmas more scalable. Our approach is
implemented in two different tool implementations, and then validated through case studies that involve two different systems.
* testsuite을 dynamic analysis의 입력 시나리오로 사용하겠다는 설명을 한다. 그리고 실제 실험적으로 적용하고 그결과가 의미가 있다는 것을 보인다.
An Assessment Methodology for Trace Reduction Techniques
In the previous chapter, we proposed the use of a traditional visualization technique to convey a system's runtime information to the user. Although various trace reduction techniques have been proposed in the literature to address the scalability concerns involved in such approaches, their applicability in different contexts often remains unclear because extensive comparisons are lacking. In this chapter, we propose an assessment methodology for the evalucation and comparison of trace reduction techniques. We illustrate the methodology using a selection of four types of reduction methods found in literature, which we evaluate and compare using a test set of seven large execution traces. Our approach enables a systematic assessment of trace reduction techniques, which eases the selection of suitable reductions in different settings, and allows for a more effective use of dynamic analysis tools in software maintenance.
* 아마도 연구의 방향을 trace reduction technique으로 잡은 듯 하다. 그래서 이 부분에 대해서 보다 심층적인 자료 수집 및 연구가 진행된 듯 하다.
Advanced Trace Visualization with Extravis
An alternative to traditional visualization techniques is the use of more advanced visualizations. Such techniques are a potentially powerful means to tracke the scalability issues associated with program comprehension and dynamic analysis, particularly when designed with these challenges in mind. In this chapter, we propose two such visualizations: the massive sequence view and the circular bundle view, both reflecting a strong emphasis on scalability. They have been implemented in a tool called Extravis. By means of distict usage scenarios that were conducted on three different software systems, we show how our approach is applicable in three typical program comprehension tasks: trace exploration, feature location, and top-down analysis with domain knowledge.
* Extravis라는 새로운 기법을 가진 도구를 소개한다.
Trace Visualization: A Controlled Experiment
The previous chapter presented Extravis, our tool for supporting program comprehension through the visualization of large traces. Similar to other trace visualization techniques in the literature, our tool was validated through anecdotal evidence, but should also be quantitatively evaluated to assess its added value to existing solutions. In this chapter, we report on a controlled experiment concerning trace visualization for program comprehension. We designed eight typical tasks aimed at gaining an understanding of a representative subject system, and measured how a control group (using the Eclipse IDE) and an experimental group (using both Eclipse and Extravis) performed in terms of correctness an time spent. The results are statically significant in both regards, showing a 21% decrease in time and a 43% increase in correctness for the latter group
* Eclipse+Extravis 를 사용하는 그룹과 Eclipse 만을 사용하는 그룹을 대상으로 실험적 평가진행
- Research question
1. Does the availability of Extravis reduce the time that is needed to complete typical comprehension tasks?
2. Does the availability of Extravis increase the correctness of the answers given during those tasks?
3. Based on the results, which types of tasks can we identify that benefit most from the use of Extravis?
- Hypothesis
Hypothesis H1: The availability of Extravis reduces the time needed to complete typical comprehension tasks.
Hypothesis H2: The availability of Extravis increases the correctness of answers given during those tasks.
- 실험에 참여하는 사람들
The subjects in this experiment are 14 Ph.D. candidates, five M.Sc. students, three postdocs, one associate professor, and one participant from industry. The assignment is to complete the eight comprehension tasks within 90 minutes The first dependent variable is the time spent on each task, and The second dependent variable is the correctness of the given answers.
- 실험의 정확도 계산
For the time as well as the correctness variable we maintain a typical confidence level of 95% (a=0.05), which means that statistical significance is attained in cases where the p-value is found to be lower than 0.05.
1.gif
Conclusion
Program comprehension is a challenging task for developers who are faces with maintaining a software system. In particular, the lack of sufficient knowledge of a system's internals means that a time-consuming examination is required in order to properly understand it. The study of a system's runtime behavior, known as dynamic analysis, has the potential to facilitate this process. Unfortunately, most of the literature on this topic does not provide thorough empirical validations. In this thesis, we sought to raise the bar with respect to the evaluation of techniques for program comprehension through dynamic analysis. We systematically surveryed the existing work, defined reusable frameworks, conducted extensive case studies, provided publicly available tools, and quantitatively measured the added value of one of these tools compared to a traditional IDE. Our contributions demonstrate the benefits of dynamic analysis for program comprehension and enable fellow researchers to better evaluate their own solutions in thid field.