Using analysis and measurement as drivers of the enhancement process is one major difference between iterative development and other models of development. It provides support for determining the effectiveness of the processes and the quality of product. It allows one to study, and therefore improve the processes for the particular environment. Data collected by Test Metrics in previous iteration can be used to improve future work estimates, redefine the focus areas and to improve efficiency. While the need of metrics is well recognized, their implementation is area where large variation is required according to nature of project and testing environment.
Basic Process
Appropriate process for use of test metrics is vital to achieve adequate quality. Basic steps in process of implementation of metrics are:
Identify the pain area or goals: Identification of pain area or problem area is an important step. In this step, risk analysis is done for the future iteration. The objective of Risk Analysis is to identify potential problems that could affect cost or outcome of the project.
Define metrics for each goal: Based on identified pain areas appropriate metrics should be defined. Base metrics will be used for data that can be captured directly. Base metrics like Number of Test cases executed/passed/failed/deferred etc. are used. Similar metrics for time & defects are also used. Calculated Metrics or Derived Metrics convert the base metrics data into more useful information. Test metrics are meaningful if they provide objective feedback to the project team regarding any of the development processes from analysis, to coding, to testing. Hence defining metrics those will provide objective result is important.
Next step is to setup a procedure to collect data. In this step, means of collecting data are decided. The collected data should be relevant, valid, and measurable, which can be used for analysis to derive conclusions about process effectiveness. Collecting irrelevant data leads to wastage of effort, time and team starts loosing faith in need of implementation of metrics.
Communicating metrics implementation plan to all stakeholders is very important. Unless the team members (test team as well as other entities) keep deliverable updated, data collected by metrics will not be valid. Many times, members of a project team intuitively understand that changes in the development life cycle could improve the quality and reduce the cost of a project, but they are unwilling to implement changes without objective proof of where the changes should occur. Hence it is important to spread awareness amongst stakeholders about objectives of metrics implementation.
Collecting Data & Analyzing it can be carried out using number of tools starting from Microsoft Excel to more specialized tools like McCabe QA, LDRA Test bed, Functional Point Analysis tools & Rational Suit. This stage should be carried out with minimum manual intervention as manual recording of data is subject to bias (deliberate or unconscious), error, omission, and delay. Therefore automatic data capture is desirable, and sometimes also essential. Derived/Calculated metrics are calculated in this stage.
Communicating results of metrics implementation should be communicated to all stakeholders periodically. Results should be in the format that is easily readable and consistent in its format. Analysis reports should include graphs highlighting the information based on which further decisions are expected.
Applying Results to achieve improvement should be done in diplomatic fashion. The results should not be used as measure of performance of individuals. Instead they should be used in constructive manner to achieve continuous improvement. Before taking decision, feedback should be taken from concerned members of team. In this stage it is necessary to revisit effectiveness of defined metrics. Metrics should be evaluated and restructured to get effective results in next iteration of development.
Test Metrics In Iterative Model
Implementation of test metrics in Iterative model has became comparatively difficult due to time constraints, variation in focus areas & variation in client requirements. Effective implementation of test metrics requires quick actions, accurate selection of metrics and collective commitment. Further the time and efforts spent on metrics collection should be optimum. If more time and effort is spent on this activity, it is likely that metrics will be perceived as futile activity. This generally leads in lost faith of project management and finally abandoning of metrics collection.
Test metrics are important to iterative development as they provide quick glimpse of the incremental quality. Problems surfacing in early iterations can be fixed and problems in future iterations can be guessed by applying techniques like risk analysis and past experience.
Test metrics also help in making decisions about test and development related estimation for upcoming iterations. Implementation of test metrics should not be responsibility of a small team. Instead all stakeholders should be aware of importance of implementation and need of doing that. Data collection and analysis activity should be automated to avoid biased decisions and manual mistakes.
Conclusions
Test Metrics implementation in Iterative development should be based on following:
1.Invest less time and efforts for metrics implementation by measures.
2.Automate data collection and analysis so as to avoid manual mistakes and biased decisions.
3.Accurate selection of metrics and data point to avoid collection of data that is of no use.
4.As requirements change frequently in iterative development and new risks are identified, introduce new metrics or reform the existing metrics whenever required.
5.Quick decisions and their implementation in order to get proper results.
6.Total involvement of project management and other stakeholders.
7.Timely and frequent communication with stakeholders.
8.Avoid using metrics for evaluation of individual performance.
9.Challenges are never ending but measurements and preventive actions will always help getting proper results.
10.Do not make metrics collection as responsibility of a single individual or small team. Involve all stake holders so that data collection is effective and efficient.
This post is based on my paper 'Test Metrics in Iterative Development' published in STC2008. Copyright of this paper are with QAI and use for profit is not allowed.
Basic Process
Appropriate process for use of test metrics is vital to achieve adequate quality. Basic steps in process of implementation of metrics are:
Identify the pain area or goals: Identification of pain area or problem area is an important step. In this step, risk analysis is done for the future iteration. The objective of Risk Analysis is to identify potential problems that could affect cost or outcome of the project.
Define metrics for each goal: Based on identified pain areas appropriate metrics should be defined. Base metrics will be used for data that can be captured directly. Base metrics like Number of Test cases executed/passed/failed/deferred etc. are used. Similar metrics for time & defects are also used. Calculated Metrics or Derived Metrics convert the base metrics data into more useful information. Test metrics are meaningful if they provide objective feedback to the project team regarding any of the development processes from analysis, to coding, to testing. Hence defining metrics those will provide objective result is important.
Next step is to setup a procedure to collect data. In this step, means of collecting data are decided. The collected data should be relevant, valid, and measurable, which can be used for analysis to derive conclusions about process effectiveness. Collecting irrelevant data leads to wastage of effort, time and team starts loosing faith in need of implementation of metrics.
Communicating metrics implementation plan to all stakeholders is very important. Unless the team members (test team as well as other entities) keep deliverable updated, data collected by metrics will not be valid. Many times, members of a project team intuitively understand that changes in the development life cycle could improve the quality and reduce the cost of a project, but they are unwilling to implement changes without objective proof of where the changes should occur. Hence it is important to spread awareness amongst stakeholders about objectives of metrics implementation.
Collecting Data & Analyzing it can be carried out using number of tools starting from Microsoft Excel to more specialized tools like McCabe QA, LDRA Test bed, Functional Point Analysis tools & Rational Suit. This stage should be carried out with minimum manual intervention as manual recording of data is subject to bias (deliberate or unconscious), error, omission, and delay. Therefore automatic data capture is desirable, and sometimes also essential. Derived/Calculated metrics are calculated in this stage.
Communicating results of metrics implementation should be communicated to all stakeholders periodically. Results should be in the format that is easily readable and consistent in its format. Analysis reports should include graphs highlighting the information based on which further decisions are expected.
Applying Results to achieve improvement should be done in diplomatic fashion. The results should not be used as measure of performance of individuals. Instead they should be used in constructive manner to achieve continuous improvement. Before taking decision, feedback should be taken from concerned members of team. In this stage it is necessary to revisit effectiveness of defined metrics. Metrics should be evaluated and restructured to get effective results in next iteration of development.
Test Metrics In Iterative Model
Implementation of test metrics in Iterative model has became comparatively difficult due to time constraints, variation in focus areas & variation in client requirements. Effective implementation of test metrics requires quick actions, accurate selection of metrics and collective commitment. Further the time and efforts spent on metrics collection should be optimum. If more time and effort is spent on this activity, it is likely that metrics will be perceived as futile activity. This generally leads in lost faith of project management and finally abandoning of metrics collection.
Test metrics are important to iterative development as they provide quick glimpse of the incremental quality. Problems surfacing in early iterations can be fixed and problems in future iterations can be guessed by applying techniques like risk analysis and past experience.
Test metrics also help in making decisions about test and development related estimation for upcoming iterations. Implementation of test metrics should not be responsibility of a small team. Instead all stakeholders should be aware of importance of implementation and need of doing that. Data collection and analysis activity should be automated to avoid biased decisions and manual mistakes.
Conclusions
Test Metrics implementation in Iterative development should be based on following:
1.Invest less time and efforts for metrics implementation by measures.
2.Automate data collection and analysis so as to avoid manual mistakes and biased decisions.
3.Accurate selection of metrics and data point to avoid collection of data that is of no use.
4.As requirements change frequently in iterative development and new risks are identified, introduce new metrics or reform the existing metrics whenever required.
5.Quick decisions and their implementation in order to get proper results.
6.Total involvement of project management and other stakeholders.
7.Timely and frequent communication with stakeholders.
8.Avoid using metrics for evaluation of individual performance.
9.Challenges are never ending but measurements and preventive actions will always help getting proper results.
10.Do not make metrics collection as responsibility of a single individual or small team. Involve all stake holders so that data collection is effective and efficient.
This post is based on my paper 'Test Metrics in Iterative Development' published in STC2008. Copyright of this paper are with QAI and use for profit is not allowed.
No comments:
Post a Comment