RE: [gnso-wpm-dt] WPM-DT: "Red Team" Idea

["Gomes, Chuck" <cgomes@xxxxxxxxxxxx>]

Ken,
 
Thanks for the added red team detail.  Very helpful.  I would just qualify your 
suggestions in this way:  I don't think we have time to have the red team do an 
exhaustive review or to test the process like we have done.  Rather I think a 
fairly quick review of the process focusing on the questions you suggest below 
should suffice. 
 
Chuck


________________________________

        From: owner-gnso-wpm-dt@xxxxxxxxx [mailto:owner-gnso-wpm-dt@xxxxxxxxx] 
On Behalf Of Ken Bour
        Sent: Thursday, December 24, 2009 1:36 PM
        To: gnso-wpm-dt@xxxxxxxxx
        Subject: [gnso-wpm-dt] WPM-DT: "Red Team" Idea
        
        

        WPM-DT Members:

         

        I have been thinking about Adrian's suggestion and Olga's question 
concerning how we might take advantage of the "Red Team" concept.  

         

        According to "Red Teams: An Audit Tool, Technique and Methodology for 
Information Assurance 
<http://www.isaca.org/Template.cfm?Section=Home&CONTENTID=30762&TEMPLATE=/ContentManagement/ContentDisplay.cfm>
 ," the typical functions of such teams are to:  

        ·         Provide a surrogate adversary to "sharpen skills, expose 
vulnerabilities that adversaries might exploit and increase the understanding 
of the options and responses available to adversaries and competitors." The red 
team may accomplish this by emulating the adversary. 

        ·         Play "devil's advocate." The red team can offer different 
alternatives to current plans, operations, processes and assumptions. 

        ·         Offer sources of judgment that are external to the 
organization and act as a "sounding board" for new ideas that may arise from 
red team engagements

        Thus far, our first product is a two-dimensional rating approach (still 
in development/test phase), but there may be others as we begin tackling how 
the resultant charts/graphs can be used for project prioritization -- the 
ultimate goal.   Although we are making excellent progress, we are still some 
distance from drafting a complete package that can be recommended to the 
Council.  

         

        Once we reach that end-state, it might be useful to subject the 
ultimate solution set to an exhaustive independent test just before going 
"live."   We have seen in our own testing that it is through exercising the 
processes that we have uncovered potential defects (e.g. adding project 
acronyms instead of sequence numbers; definition for Y to include GNSO more 
specifically).   Since the WPM-DT has been intellectually close to the 
development from the outset, it is possible that we have overlooked certain 
fundamental elements that might cause problems in production.   One way to 
minimize that eventuality is to bring in another set of objective evaluators 
(or Red Team) and ask them to execute the entire rating/prioritization process 
-- start to finish -- as laid out.   By exercising the methodology in this way, 
a Red Team could offer a fresh perspective including asking naïve questions and 
probing the underlying rationale in ways that might not have been adequately 
challenged.  

         

        Following the above outline, when the time is right, we might ask 
Adrian to lead a small group (size will depend on the ultimate solution) to 
actually perform the entire set of procedures that are packaged into the final 
recommendation.   The Red Team can check to ensure that the DT's original goals 
have been met and that the process:   

        ·         is user-friendly, unambiguous, and straightforward to 
execute; 

        ·         produces realistic outputs that will enable the Council to 
make effective prioritization decisions; and

        ·         is structured not only as a one-time exercise, but considers 
the inclusion of new projects as they are proposed in the future.  

         

        In addition, other potential evaluative questions might include:  

        ·         Are the objectives clear?  Have we satisfactorily answered 
the obvious question, "Why are we doing this activity?"   

        ·         Does the process make sense in terms of the leading to the 
stated objectives?

        ·         Does the methodology require accepting any assumptions that 
have not been disclosed?

        ·         Are the instructions, guidelines, and definitions clear and 
sensible? 

        ·         If tools are provided, do they work as described?  

        ·         Others?  

         

        To summarize, once the DT has reached closure on a final Work 
Prioritization Model and recommendation, it could engage Adrian (and others?) 
to provide a neutral and objective critique -- "emulating the adversary" -- or, 
in this case, simulating how other GNSO Councilors might react to and interact 
with the final proposed solution.    

         

        Regards,

         

        Ken Bour