Successful Ph.D. Dissertation Defense by Qin Zhang (08/30/2006)

Posted By: Gerald  Lau ( glau@hawaii.edu )

Reference: http://www2.hawaii.edu/~qzhang/

Qin Zhang has successfully defended his dissertation, as announced below ...
----------------------------------------------------------------
Improving Software Development Process and Project Management
                                         with
                          Software Project Telemetry

                              Dissertation Defense
                       POST 302 (ICS Conference Room)
                            Wednesday, Aug 30, 2006
                                   2:00pm-4:00pm

                                    Qin Zhang
             Collaborative Software Development Laboratory
           Department of Information and Computer Sciences

Abstract

Software development is slow, expensive and error prone, often resulting in 
products with a large number of defects which cause serious problems in 
usability, reliability, and performance. To combat this problem, software 
measurement provides a systematic and empirically-guided approach to control 
and improve development processes and final products. Experience has shown 
excellent results so long as measurement programs are conscientiously 
implemented and followed. However, due to the high cost associated 
with ?metrics collection? and difficulties in ?metrics decision-making?, many 
organizations fail to benefit from measurement programs. 

In this thesis, I propose a light-weight approach to software measurement 
called ?software project telemetry?. It addresses the ?metrics collection cost 
problem? through highly automated measurement machinery: software sensors are 
written to collect metrics automatically and unobtrusively. It addresses 
the ?metrics decision-making problem? through a domain-specific language 
designed for the representation of high-level perspectives on software 
development process. In contrast to traditional model-based approaches which 
involve the comparison of metrics from different projects, software project 
telemetry focuses on the comparison of metrics from the same project at 
different times. This within-project data comparison involves a much smaller 
time scale: typically with intervals of days or weeks. The idea is that 
comparison can be made more effectively between two different periods of the 
same project than between two different projects. It thus avoids many problems 
a model-based approach suffers from, such as spending the cost of accumulating 
a historical project database first and then constantly worrying about whether 
the current project is ?comparable? to those in the database. In software 
project telemetry, the metrics from the initial period of the project are used 
to establish a baseline and bootstrap the process. Project management and 
process improvement decisions are made by detecting changes in telemetry 
trends and comparing trends between two periods of the same project. In-
process control for a project that is still under development is made possible 
precisely because comparisons are made within the same project instead of 
across projects. 

The claim of this thesis is that software project telemetry provides an 
effective approach to (1) automated metrics collection and analysis, and (2) 
in-process, empirically-guided software development process problem detection 
and diagnosis.Two empirical studies were carried out to evaluate the claim: 
one in software engineering classes, and the other in the Collaborative 
Software Development Lab (CSDL). The classroom study was relatively simple. I 
distributed a questionnaire at the end of the semester to collect the 
student's opinions about software project telemetry. To increase my confidence 
in the validity of their self-reported opinions, I analyzed their telemetry 
analysis invocation pattern to determine the extent to which their opinions 
were based on the actual system usage. In the CSDL study, I pursued a much 
more in-depth data collection and analysis strategy over a much longer period 
of time. I introduced software project telemetry as a metrics-based process 
improvement program and explored its use in the lab extensively. I collected 
data from observations and interviews, and used the grounded theory approach 
to generate hypotheses. I also tested the hypotheses in a limited way by 
making changes to the telemetry system or implementing new facilities to see 
if the hypothesized outcome would come true. The results from the two studies 
suggested that software project telemetry had acceptably-low metrics 
collection and analysis overhead, and that it provided decision-making value 
at least in the exploratory context of the two studies.  

The studies also generated a number of insights with respect to real-world use 
of software project telemetry. Software project telemetry delivers best value 
when it can be customized to the specific need of an organization. The 
customization includes both setting up sensors to collect metrics and 
designing charts to perform analyses. Due to automated nature of metrics 
collection, broken sensor might be undetected for some time. Special-purpose 
telemetry charts can be designed to help make quick assessment whether the 
underlying sensor data are likely broken or not by flagging suspicious 
metrics. Data privacy concern, especially with effort-related personal process 
metrics, might be an adoption barrier for the technology. Though the 
implementation of software project telemetry has a mechanism to limit the 
kinds of data that could be access by people other than the owner, overcoming 
this issue seems largely dependent on what the data are used for in an 
organization. 

Two pieces of software are made publicly available as a result from this 
research. One of them is a server-side component that enables a user 
to ?actively? explore relationships among different metrics using a web 
browser. The other is a client-side application that can be configured to 
retrieve and display a sequence of telemetry charts automatically and 
continuously, providing the user with ?passive? awareness of project status.