Re: [regrep] 7/29/2003: Conformance Efforts and ebXML IIC

Farrukh and all,

 

   I've >snipped< portions of our 
previous conversation regarding ebRS conformance testing,

for brevity. Below are my additional 
comments highlighted in BOLD

 

----- Original Message ----- 

[MIKE] - I see an ebXML RS conformance test 
suite specification document,
>which contains a list of conformance test 
requirements, and a conformance test suite 

>Also included  is supporting test material 
like that put together
>by Len Gallagher in the initial ebRS conformance 
test suite 

[Farrukh]  Your detailed responses have clarified things 
very much and I feel we 
are on the same page.

[Mike] I would 
suggest a "staged" approach, in which test assertions are
>compiled for a 
selected "profile" or "level"

[Farrukh]  +1 on this very pragmatic 
advice.

[Farrukh] Who produces automated test suite for verifying 
Conformance?

[MIKE] - The IIC is normally tasked with this 
effort.  Although again, due to our
>own resource limitations,we 
would ask for help from the registry TC in contributing tests.

[Farrukh]  I think that would be very 
reasonable. Is any one in the RegRep TC 
available to start the effort by 
taking ownership of marking up the RS 
spec for potential assertions to 
harvest into an initial skeletal CTS ?

>[MIKE] -  If you take a 
"divide and conquer" approach to RS testing, you may be
>able to make this 
easier.  For instance, choosing a single binding (e.g. SOAP) to write test 
requirements, 
>
[Farrukh]  +1 on choosing SOAP binding as a 
starting point

[MIKE] selecting LifeCycleManager
>functionality as 
a higher priority for test requirements than 
QueryManager
>functionality...  could get the ball rollling 
sooner.
>
[Farrukh]  I think we cannot do only LCM and not QM. 
Maybe we should consider doing 
More LCM and some QM.

 

[MIKE] - I agree.  We must test 
both.. but initially focusing on LCM might be best.

[MIKE] - I 
started "marking up" the ebRS 2.5 specification, pages 1-40, ( 
sectoins
>#1 - #6) I found that portion of the specification to be 
straightforward regarding
>identifying conformance test 
requirements.
>
>The LifeCycleManager section (section #7) appears 
to be straight-forward and
>easy to identify testing requirements, grouped 
into fundamental
>sub-categores of "parameter", "response",  
"exception" and  "audit trail"
>types of testing
>

[Farrukh]  +1

[MIKE] The 
QueryManager (section #8) could be a very rigorous exercise ... I 
would
>suggest a "generalized" approach to FilterQuery or 
SQLQuery
>testing, in which "duplicate" query tree node testing is 
eliminated to
>simplify both test requirement writing, as well as actual 
test generation.
>

[Farrukh]  I think that tests will require 
publish using LCM and verifying using QM 
that data written can be read back 
as expected.

 

[MIKE] - Agreed

[MIKE] 
Section #9, Content Management Services, is an optional service, that can 
be
>tested by creating/testing an IIC defined content
>management 
service to the registry, then verifying its 
conformant
>functionality.  This could be done by creation and 
publishing of a
>"dummy" validation service, I believe.  Because this 
is an optional feature,
>test requirements for this service could be 
described.. but
>actual implementation of test cases "could" be deferred 
if time/resources
>are an issue.
>

[Farrukh]  We could only do the default 
XML Cataloging Service initially which is 
the onloy required part of CMS 
chapter.

 

[MIKE] - This is a good minimalist approach 
initially.

[MIKE]  Section #10, Event Notification Service, is also an 
optional service, that
>is well documented and fairly simple to generate 
test requirements.
>Acutal test case scenarios should be easy to 
construct.  Again, because this
>is an optional feature, it is a 
candidate for test requirement
>generation, but actual test cases COULD be 
deferred for a later time.
>

[Farrukh] I could see defering this initially altogether.

 

[MIKE] - Agreed

[MIKE] Section #11, Cooperating 
Registries Support is a complex scenario requiring
>federated registries 
for conformance testing.  The requirements
>are well stated, but 
implementation of test cases will be complex and
>time-intensive.  
The specification does not eplicitly say whether this is an
>optional 
feature, although I assume that it is.
>  
>

[Farrukh]  Some features are required but most are optionall. I could 
see defering 
this initially altogether.

[MIKE] - Agreed

[MIKE] Section #12,  Security an be a large exercise, particularly 
with all of the
>possible XMLDSIG options.  But this is a "required" 
feature, and can perhaps
>be described with a small initial set of test 
assertions and test cases, and
>a limited testing scope.
>

[Farrukh] Testing default ACcess Control POlicy is a minimal 
requirement IMO.

[MIKE] - Agreed

[MIKE] So from a deliverables 
standpoint, if you are doing the entire ebRS
>specification  I would 
suggest ( working in parallel if resources are
>available ) 
:
>

[Farrukh]  Indeed. That is why I ask team again to speak up idf 
you are bale to help.

[MIKE] 1) "Marking up" the ebRS document to 
identify testing requirements - 4 weeks
>

[Farrukh]  Maybe we can divide chapters among the peopel 
available.

 

[MIKE] - I will submit a basic guideline document to the RegRep TC 
outlining the

fundamental procedures and principles in compiling a Conformance 
Test Suite Specification

document.  We are working on that in the IIC.  This will 
assist your TC in getting traction

on initial requirements and test cases.  

[MIKE] 6) To create a "full blown" ebRS Conformance Test Suite 
12 months
>7) To create a Conformance Test Suite Specification - 
(integrating test
>requirements, test cases and supporting documentation) 
- 1 month
>
>This is my estimate, based upon IIC work with ebXML 
Messaging Services.
>

[Farrukh]  Your estimates seem right on  to me and based on 
first hand experience.

Lets hope that some colleagues are able to 
volunteer time to begin this 
work soon.

 

[MIKE] - I will try to re-arrange my schedule so that I can have 
Thursday afternoons free for

regrep conference calls, and we can discuss these issues and 
measure progress.  I will not be

able to make today's call, if there is one.

[Farrukh]  On a personal level I can commit to all review 
activity. I am reluctant 
to take on any primary role I am over committed 
between spec work and 
implementation.

[MIKE] - Understood.

Thanks again for laying this out so clearly for us to begin this 

important work.

-- 
Farrukh