Professional Documents
Culture Documents
Introduction
NIST works with industry to develop standards and tests to improve the quality of software and achieve interoperable solutions Many years experience with
Formal standards organizations and Consortia Developing conformance test suites, tools, reference implementations Developing validation and certification testing programs
Todays Objectives
Present basic information about conformance and conformance related topics Achieve a common understanding Lead a discussion about how conformance applies to ebXML Present overview of TA Spec. Conformance Clause Offer our assistance to WG in discussing and developing conformance for their Spec.
Outline
Conformance Testing
Test suites Test tools
Conformance clause
Validation
Certification
Certificates Branding
Procedures, reporting
Background
Terminology
Conformance
Specification
Conformance Testing Conformance Clause test suite, test tools process, reporting certificate, brand
Validation
Certification
Terminology - Conformance
These requirements are specified in a standard or specification as part of a conformance clause or in the body of the specification
CONFORMANCE CLAUSE - a section of a specification that states all the requirements or criteria that must be satisfied to claim conformance
CONFORMANCE TESTING a way to verify implementations of a specification to determine whether or not deviations from the specifications exist (through the use of test suites).
tests capture the technical requirements within the specification testing is bound in scope by the specification internal structure of product-under-test not accessible (black box testing)
Testing Methodology
Falsification Testing
find errors by means of experimentation show presence of errors not their absence prove non-conformance; can never prove conformance
Specification
Requirements
???? conforming
VALIDATION - process necessary to perform conformance testing in accordance with a prescribed procedure and official test suite
ensures that testing can be repeatable and reproducible ensures that conclusions are consistent with facts presented in the evaluation
CERTIFICATION - acknowledgement that a validation was completed and the criteria established by CIO for issuing certificates (brands) was met.
Testing Ingredients
Certification
qualified bodies to do the testing and certification control board - advisory and arbiter
Validation
Process - policy and procedures for testing
Profile is a subset of the overall specifications that includes all of the functionality necessary to satisfy the requirements of a particular community of users. Levels are nested subsets of the specifications. Level 1 = core that all must implement; subsequent levels include lower level + added functionality
Register of extensions
document parallel to but distinct from specification list of recognized, published extensions may eventually migrate into future versions
5. Alternative Approaches
specification may describe several different ways to accomplish an operation (e.g., choice of bindings) specify whether or not all approaches need to be implemented
b. Test Suite
- Prescribe a test methodology? - How test suite will be delivered/used (e.g., web based, downloadable)?
c. Testing Service
- Guidelines for establishing a test service
Testing program
Who will own the testing program Policy and procedures Testing laboratory Issue Certificates Control board Test suite maintenance
TA Conformance Guidance
General conformance guidelines for ebXML All ebXML specs shall contain a conformance clause
Can have levels of ebXML conformance hierarchical Explains how conformance requirements can be expressed (mandatory, conditional, optional)
Discussion