Introduction to the project
The Project
ViVa is an ESPRIT project, running till February 1997. The purpose is to develop a method and a toolset for validation and verification (V&V) of knowledge based systems (KBS).
V&V is an important and necessary part of any industrial software development process. The overall goal of V&V is to increase the confidence that the software and related documents are consistent and complete with respect to the technical state-of-the-art of software engineering and comply with the user requirements to the utmost extent.
In recent years, knowledge-based systems have successfully been applied to real-world problems in engineering domains. In the space engineering domain, several KBS applications have been developed, mainly for monitoring, diagnosis or scheduling purposes. For many reasons, confidence in KBS technology has been low for non-trivial and safety-critical applications. In order to increase this confidence, the ViVa project focuses on V&V of KBS-specific software parts, rules, class hierarchy and objects.
The benefits are expected to be:
- Improved KBS quality through the methodical checking of the accuracy, relevance and consistency of knowledge, of design, and of implementation.
- Improved system development efficiency through reduction of errors and redesign.
- Access to the safety and mission critical markets for KBS applications through increased level of confidence.
Organisations can take advantage of the ViVa method/toolset product for: - performing V&V during KBS development,
- ensuring that development has been successfully finished,
- validating existing KBS.
Such organisations include:
- System houses developing KBS for a client.
- Software engineering departments within large organisations developing KBS.
- Users or purchasers of KBS.
- Consultancy organisations.
The project aims at a practically applicable method and a mature toolset for V&V of KBS, and in order to obtain this, the method/toolset product is evaluated as part of the project. ESA/ESOC (European Space Agency/European Space Operations Center) - though not a formal partner - will provide the trial applications for the method and the toolset. The applications will come from the ATOS-4 project, which is an operational project running in parallel with ViVa. ATOS-4 is part of ESA/ESOC's ATOS ("Advanced Technology Operations System") programme, the aim of which is to develop the design basis for space mission operations systems using advanced technologies such as KBS [Jones et al, 1994].
The consortium partners have the following responsibilities:
- CRI, who has considerable experience and interests in the area of KBS, is coordinator for the project. CRI is further responsible for integration and evaluation of the project results.
- Siemens Österreich, an experienced KBS development organisation, is responsible for developing the ViVa Method.
- Softlab Austria, who specialises in monitoring and control of software projects in close cooperation with customers, cooperates with Siemens on the method development.
- Université de Savoie (LIA), who has a recognised profile in V&V for KBS, is responsible for the development of the toolset.
The Method
The ViVa Method prescribes V&V activities to be performed in each phase of the development of a KBS, including demonstrating whether the results produced satisfy the requirements. In order to be useful, the activities described in the method must be related to the specific life-cycle model used in the KBS development project. In this way, the method provides an "add-on" to the KBS development activities.
The ViVa Method is developed in two steps:
- A generic ViVa method - the ViVa Framework [Lackinger, Schlee, 1995] - is developed independently of specific life-cycle models, development methods and software engineering standards.
- The ViVa Method [Kaindl, 1995, Schlee, Lackinger, 1995] is developed as a specialisation of the Framework towards the KBS development method KADS [Schreiber et al, 1993] and ESA's software engineering standard PSS-05 [ESA, 1991]. The new software engineering standard ECSS-E-40 under development by the European Cooperation for Space Standardization will influence the ViVa Method when ready.
The ViVa Framework has three dimensions:
- The V&V reference method - describing what needs to be done.
- The V&V role model - describing who does it.
- The V&V process architecture - describing how it will be done.
The Framework includes a generic description of the products produced during KBS development, and associations between these. Both products and associations can be characterized by quality properties. A quality property describes a (measurable) quality aspect of the product or association. For each quality property, the Framework indicates one or more V&V techniques which can be used to determine its value.
The ViVa Method defines all activities needed in order to perform V&V. This includes:
- Definition of validation requirements. A validation requirement is a user requirement which defines the desired values of one or more quality properties.
- Planning of V&V activities, based on the validation requirements: Who is responsible for what, and when should it be done.
- How to perform the planned V&V activities: Guidelines for actually performing the work.
Validation requirement example: "The KBS must have correct behaviour on a set of test cases, which are agreed as being critical and representative by the expert and client".
The V&V activities in grey ellipses are supported by the toolset.
The Toolset
The ViVa Toolset supports selected parts of the method. In method terms, the functionality of the tools cover implementation of V&V techniques for finding values of quality properties of the code product.
The project has chosen to focus on functionalities which have already been tried in research or industrial contexts, in order to realise and integrate these into sufficiently mature software.
The tools provide functionality in the following areas:
- Static consistency checking.
- Static completeness checking.
- Support for testing
- Support for V&V management.
The following components of the knowledge base are statically checked for consistency:
- The class hierarchy
- Ensuring that classes have attribute types and arities defined which are consistent with the ones of the superclasses.
- The objects in the knowledge base
- Checking that actual attribute values and arities are consistent with the class definitions
- The rule base
- Looking for redundant rules, rule cycles, conflicting rules, etc.
The completeness of the knowledge base is verified by checking the following components:
- Input and output specifications
- All input specifications must be contained in at least one rule premise, and all output specifications must be contained in at least one rule conclusion.
- Ruleset
- All rule premises must be either an input pattern or contained in a rule conclusion, and all rule conclusions must be either an output pattern or contained in a rule premise.
- Object base
- Each attribute defined in the class hierarchy must be referenced in the object base.
Tool functionality for test support is as follows:
- Test case input generation. Test case input can be generated automatically for the purposes of random testing, limit testing (using limit values for input), or robustness testing (using illegal input values).
- Automatic execution of a KBS, using one or more test cases.
- Checking the output from KBS execution. ViVa will allow the user to state the expected output from running a test case, and the actual output will then be compared to this.
- Completion testing: Ensuring that the KBS produces correct answers for a set of test cases representing a problem domain.
- Obtaining coverage information: Which rules were fired, and how many times, in a given execution of the KBS.
- KBS metric calculation. Automatic calculation of specific metrics. A metric is a numerically typed property of the KBS and its behaviour.
Support for V&V management consists in:
- Management of problem reports, generated by different tools or manually entered by the users, is supported by a specialised browser/editor. All tools which are capable of detecting errors will automatically generate problem reports.
- Support for management of the V&V process, by presenting an overview of the tool activities performed in the current session.
The Method/Toolset Implementation
The toolset is implemented as an Intranet application, where a World Wide Web browser provides a common user interface for the tools. The tools are further integrated through their common use of the ViVa repository database to store and retrieve results. Figure 1 is a snapshot, where coverage information is presented to the user, dynamically generated from the repository as a WWW page. The coverage values shown are obtained as a result of running test cases on instrumented versions of the KBS. Instruments are procedure calls, which are inserted into the code of the KBS, and which produce traces in the repository. In the snapshot example the instrument records when a rule conclusion succeeds. Coverage values are calculated for each test case and for groups of test cases.
Figure 1: The use of a WWW browser for the toolset user interface.
The method is provided in the form of a WWW hypertext document. It contains operational guidelines for performing V&V of KBS and links to a glossary and to literature references. The method document, the toolset and the toolset user manual. which is also provided as a WWW document, are integrated by HTML links to each other, as illustrated in figure 2.
Figure 2: The integration of method and toolset by WWW techniques.
The underlying client/server architecture is illustrated in figure 3. The user on the client side views HTML document in the WWW browser and clicks on hyperlinks and buttons, which sends link addresses (URL) to the HTTP server. The URL contains tool identifier and arguments. The server invokes the tool, some of which are implemented as C++ programs, some are PL/SQL programs. The invoked program answers with a dynamically generated HTML document containing data retrieved from the repository, and for some tools containing JavaScript functions and Java applets.
Figure 3: The client/server architecture of the ViVa toolset.
The choice of WWW techniques for the implementation has many advantages:
- The method and toolset can be used on many different platforms.
- Users can work on the same project from different locations.
- Prototyping of the user interface is very easy.
- Demonstrations and information dissemination are facilitated.
References
M. Jones, J. Wheadon, W. O'Mullane, P. Sever, D. Whitgift, Improving the Cost Effectiveness of Mission Operations Systems through Integration and Reuse, ESOC paper, 1994.
F. Lackinger, S. Schlee, The ViVa Framework for V&V of KBS (method deliverable D-4), Softlab Austria, 1995.(Download as postscript file (700381 bytes))
H. Kaindl, The ViVa Method for V&V of KBS during Development According to KADS (method deliverable D-5), Siemens Österreich, 1995. (Download as postscript file (94478 bytes+1175730 bytes))
S.Schlee, F. Lackinger, The ViVa Method for V&V of KBS during Development Following PSS-05 Guidelines (method deliverable D-6), Softlab Austria, 1995.(Download as postscript file (598183 bytes))
G. Schreiber, B. Wielinga, J. Breuker, eds., KADS, A Principled Approach to Knowledge-Based System Development, Academic Press, 1993.
ESA Software Engineering Standards, Issue 2, ESA-PSS-05-0, 1991.
H. Kaindl, S. Kramer, References on Validation and Verification of Knowledge-Based Systems, Siemens Österreich, 1995.
vivawww@cri.dk
Last update: 8 April 1997.