RE: [gnso-wpm-dt] WPM-DT: "Red Team" Idea

[Adrian Kinderis <adrian@xxxxxxxxxxxxxxxxxx>]

I agree with Chuck. A "blink" review of the logic that has been applied and the 
decisions that were made I think would work well.

Adrian Kinderis


From: owner-gnso-wpm-dt@xxxxxxxxx [mailto:owner-gnso-wpm-dt@xxxxxxxxx] On 
Behalf Of Gomes, Chuck
Sent: Friday, 25 December 2009 6:19 AM
To: Ken Bour; gnso-wpm-dt@xxxxxxxxx
Subject: RE: [gnso-wpm-dt] WPM-DT: "Red Team" Idea

Ken,

Thanks for the added red team detail.  Very helpful.  I would just qualify your 
suggestions in this way:  I don't think we have time to have the red team do an 
exhaustive review or to test the process like we have done.  Rather I think a 
fairly quick review of the process focusing on the questions you suggest below 
should suffice.

Chuck

________________________________
From: owner-gnso-wpm-dt@xxxxxxxxx [mailto:owner-gnso-wpm-dt@xxxxxxxxx] On 
Behalf Of Ken Bour
Sent: Thursday, December 24, 2009 1:36 PM
To: gnso-wpm-dt@xxxxxxxxx
Subject: [gnso-wpm-dt] WPM-DT: "Red Team" Idea
WPM-DT Members:

I have been thinking about Adrian's suggestion and Olga's question concerning 
how we might take advantage of the "Red Team" concept.

According to "Red Teams: An Audit Tool, Technique and Methodology for 
Information 
Assurance<http://www.isaca.org/Template.cfm?Section=Home&CONTENTID=30762&TEMPLATE=/ContentManagement/ContentDisplay.cfm>,"
 the typical functions of such teams are to:

*         Provide a surrogate adversary to "sharpen skills, expose 
vulnerabilities that adversaries might exploit and increase the understanding 
of the options and responses available to adversaries and competitors." The red 
team may accomplish this by emulating the adversary.

*         Play "devil's advocate." The red team can offer different 
alternatives to current plans, operations, processes and assumptions.

*         Offer sources of judgment that are external to the organization and 
act as a "sounding board" for new ideas that may arise from red team engagements
Thus far, our first product is a two-dimensional rating approach (still in 
development/test phase), but there may be others as we begin tackling how the 
resultant charts/graphs can be used for project prioritization -- the ultimate 
goal.   Although we are making excellent progress, we are still some distance 
from drafting a complete package that can be recommended to the Council.

Once we reach that end-state, it might be useful to subject the ultimate 
solution set to an exhaustive independent test just before going "live."   We 
have seen in our own testing that it is through exercising the processes that 
we have uncovered potential defects (e.g. adding project acronyms instead of 
sequence numbers; definition for Y to include GNSO more specifically).   Since 
the WPM-DT has been intellectually close to the development from the outset, it 
is possible that we have overlooked certain fundamental elements that might 
cause problems in production.   One way to minimize that eventuality is to 
bring in another set of objective evaluators (or Red Team) and ask them to 
execute the entire rating/prioritization process -- start to finish -- as laid 
out.   By exercising the methodology in this way, a Red Team could offer a 
fresh perspective including asking naïve questions and probing the underlying 
rationale in ways that might not have been adequately challenged.

Following the above outline, when the time is right, we might ask Adrian to 
lead a small group (size will depend on the ultimate solution) to actually 
perform the entire set of procedures that are packaged into the final 
recommendation.   The Red Team can check to ensure that the DT's original goals 
have been met and that the process:

*         is user-friendly, unambiguous, and straightforward to execute;

*         produces realistic outputs that will enable the Council to make 
effective prioritization decisions; and

*         is structured not only as a one-time exercise, but considers the 
inclusion of new projects as they are proposed in the future.

In addition, other potential evaluative questions might include:

*         Are the objectives clear?  Have we satisfactorily answered the 
obvious question, "Why are we doing this activity?"

*         Does the process make sense in terms of the leading to the stated 
objectives?

*         Does the methodology require accepting any assumptions that have not 
been disclosed?

*         Are the instructions, guidelines, and definitions clear and sensible?

*         If tools are provided, do they work as described?

*         Others?

To summarize, once the DT has reached closure on a final Work Prioritization 
Model and recommendation, it could engage Adrian (and others?) to provide a 
neutral and objective critique -- "emulating the adversary" -- or, in this 
case, simulating how other GNSO Councilors might react to and interact with the 
final proposed solution.

Regards,

Ken Bour