<<

. 5
( 9)



>>

This process goes on and on, but I think you see the point”decompose the
TPM into its constituents, de¬ne them, and measure them.

53 DESIGN REVIEWS Y
FL
53a (NO) All Design Reviews were not completed according to
required processes.
AM


All Design Reviews were not completed according to required processes when
the events of the Design Review are not directly traceable to the requirements
stipulated in standard processes, customer (contract and contract-referenced)
TE




processes, or enterprise processes.

RECOVERY

Lay out the enterprise requirements, the customer requirements, and the
standard requirements for Design Reviews. Interrelate all the requirements and
summarize and organize them into a checklist that will drive the Design Reviews
part of the Program Plan. Retain that information to solidify all data trails. You
should end up with a general matrix that will boil down to an outline similar
to the one below. If it is not possible to accommodate the above steps, go di-
rectly to the outline below and use and update it as necessary.
Design Review Package Content and Review Outline:

’ Mission and Requirements Analysis
’ ConOps (Concept of Operations)
BLUEPRINT FOR PROJECT RECOVERY
126


’ Functional Flow Analysis
’ Use Cases
’ Preliminary Requirements Allocation
’ System/Cost Effectiveness Analysis
’ Trade Studies (e.g. addressing system functions in mission and support
hardware/¬rmware/software)
’ Synthesis
’ Logistics Support Analysis


Specialty Discipline Studies (i.e., hardware and software reliability analysis,
maintainability analysis, armament integration, electromagnetic compatibility,
survivability/vulnerability (including nuclear), inspection methods/techniques
analysis, energy management, environmental considerations):


’ System Interface Studies
’ Generation of Speci¬cation
’ Program Risk Analysis
’ Integrated Test Planning
’ Producibility Analysis Plans
’ Technical Performance Measurement Planning
’ Engineering Integration
’ Data Management Plans
’ Con¬guration Management Plans
’ System Safety
’ Human Factors Analysis
’ Value Engineering Studies
’ Life Cycle Cost Analysis
’ Preliminary Manufacturing Plans
’ Manpower Requirements/Personnel Analysis
’ Milestone Schedules
’ Communications Plan
RECOVERING FROM TECHNICAL PROBLEMS 127


’ Training Plan
’ Security (Threat) Analysis


The source for the above list is MIL-STD-1521, Paragraph 10.3, plus embel-
lishment. Because it is a governmental (DOD) standard, it may well be overly
complex. But it™s a lot easier to eliminate a line item than to create one. Modify
the list for your speci¬c needs.

Additional Resources:

MIL-STD-1521

53b (NO) The customer has not approved each Design Review.

The customer will not have approved each Design Review unless the cus-
tomer has signed a sheet that con¬rms that the customer (through a representa-
tive, if necessary) agrees to the Design Review package, the Design Review, and
the Design Review minutes, including Design Review action items. Note: Any
exceptions taken should be included in the Action Items and thus achievable.

RECOVERY

Create and use a Design Review Approval Sheet containing information simi-
lar to Figure 5-2 on the following page.
Details regarding the Design Review Approval Form can be found in Attach-
ment 15.

54 IN-PROCESS REVIEWS

54a (NO) All required In-Process Reviews were not conducted
according to required processes.

All required In-Process Reviews were not conducted according to required
processes when the events of the In-Process Reviews are not directly traceable
to the requirements stipulated in standard processes, customer (contract and
contract referenced) processes, and enterprise processes.
BLUEPRINT FOR PROJECT RECOVERY
128


Figure 5-2 ” Design Review Approval Form
DESIGN REVIEW APPROVAL


The ________(1)_____________ Design Review Minutes

containing the ______(1)_________Design Review Package

labeled _______(2)____________

and dated _____(3)________

and

The _________(1)___________Design Review

conducted on _____(3)__________ together with the Design Review Action Items are

hereby approved

and

__________(4)____________________ is hereby directed to proceed to the next stage

of the program.

Signed _____(5)________ of __________(6)______________ Date _______________




RECOVERY

Lay out the enterprise requirements (see glossary), the customer require-
ments (from the requirements document), and the standard requirements (see
glossary) for In-Process Reviews. Consolidate all the requirements and summa-
rize and organize into a checklist that will drive the In-Process Reviews part of
the Program Plan. Retain that information to solidify all data trails. If it is not
possible to accommodate the above steps, use the following topics to guide your
In-Process Reviews.

In-Process Review Process

In-Process Reviews should be used to inspect individual products during the
product (hardware and/or software) development life cycle. A record of each
review must be maintained and presented at the end of the review.
RECOVERING FROM TECHNICAL PROBLEMS 129


In-Process Review Team

In-Process Review Teams normally consist of three to ¬ve members. One
member should be designated as team leader or moderator. The remaining team
members conduct the inspection.

Schedule

An In-Process Review should be conducted at the completion of each desig-
nated phase or subphase of the product life cycle.

Agenda

All internal reviews must be conducted to an agenda that has been distrib-
uted in advance of the review meeting. The agenda should call for a brie¬ng,
the inspection, notes preparation, and a debrie¬ng.

Review Participants

Quali¬ed participants from process control, quality assurance, representa-
tives from the preceding phase and from the subsequent phase as well as repre-
sentatives from other appropriate organizations should take part in the review.

Documentation

The In-Process Review should be documented in an In-Process Review Ap-
proval Form similar to that provided in Attachment 16.

54b (NO) The appropriate authority has not approved each In-
Process Review.

The appropriate authority will not have approved each In-Process Review
unless the appropriate authority has signed a sheet that con¬rms that the appro-
priate authority (through their representative, if necessary) agrees to the In-
Process Review package, the In-Process Review and the In-Process Review min-
utes, including In-Process Review action items. Note: Any exceptions taken
should be included in the action items and thus achievable.
BLUEPRINT FOR PROJECT RECOVERY
130


RECOVERY

Create and use an In-process Review Approval sheet containing information
similar to Figure 5-3.

Figure 5-3 ” In-Process Review Approval Form
IN-PROCESS REVIEW APPROVAL FORM

The ___(1)____ In-Process Review Minutes containing

the __(1)_____In-Process Review Package

labeled ___(2)_______ and

dated ____(3)_______

and

The __(1)___In-Process Review

conducted on ___(3)_______ together with the In-Process Review Action Items are

hereby approved

therefore

____(4)____ is hereby directed to proceed to the next stage of the program.


Signed ___(5)_____ of ______(6)________ Date _______________




55 PROTOTYPES

55a (NO) The prototypes do not re¬‚ect the requirements.

The prototypes do not re¬‚ect the requirements when the customer or client
does not agree that the prototype satis¬es or demonstrates the requirements.
This is perhaps subjective, but is the nature of prototyping. It is common in
the prototyping process that the customers have new or added requirements or
have changed their minds. This is fundamental to the prototyping process.
When this happens, the product and the requirement must be compared and if
RECOVERING FROM TECHNICAL PROBLEMS 131


they do not agree, the product must be reworked until they do agree or the
contractual agreement must be modi¬ed.
The best way to eliminate subjectivity is to conduct a physical or functional
audit. For a detailed description of functional con¬guration audits and physical
con¬guration audits, see MIL-STD-1521. These audits are essentially physical
and functional inventories against the requirements.
There is a dichotomy inherent in the prototyping process. While it is a quick
way to get a technical result and customer feedback, it is fraught with program-
matic problems. This is due to the fact that technical people are talking to
technical people, both wanting to solve the issue technically, usually without
regard to the programmatic issues. The differences lie in what you (your proj-
ect) have agreed to provide. It is not uncommon for a project to start with a
general set of technical requirements and agree to provide X number of man-
hours to achieve that result. Even though the technical requirement may not
have been met, the contractual requirement may be met.

RECOVERY

The ¬rst step is to de¬ne the issue. Have the technical requirements been
met? Have the contractual requirements been met? Likely not, so let™s separate
the issues into logical pairs, as in Table 5-5, and recover from there.

Ta b l e 5 - 5 ” I s s u e P a i r s a n d R e c o v e r i e s

Technical Contractual Recovery
Not Met Not Met Continue until one or the other is met then proceed as shown below
Met Not Met If you have met the technical objectives but not met the contractual
objectives, it usually means you are in an overrun condition. If you
have a cost plus contract, you should be able to adjust the manpower
or schedule to ¬t the actual pro¬le. If you have a ¬xed price contract,
you may well have to absorb the costs. The general rule-of-thumb is
that you do not take an R&D task on a ¬xed price basis.
Not Met Met One of three alternatives: 1) Change the technical requirement, 2)
Change the contractual conditions, 3) Quit.*
Met Met No issue

*Assumes neither Alternative (1) nor (2) will work and there is common agreement with the customer.
BLUEPRINT FOR PROJECT RECOVERY
132


Additional Resources:

Guides to physical inventory can be found in:

IEEE Std 610.12-1990
MIL-STD-973, 1521, 2167, and 498
DID DD-1423
MIL-STD-1521


Guides to functional inventory can be found in:

MIL-STD-973
DID DD-1423
ISO-9000-3:1991(E)
MIL-STD-1521

55b (NO) Prototypes were not constructed incrementally.

If prototypes are not constructed incrementally and two or more modules
are put together and fail testing, you will likely have no idea where the problems
lie. Prototypes can be constructed incrementally either vertically or horizontally.
A vertical increment means that modules are constructed serially and the subse-
quent modules are added on to existing modules. A horizontal increment means
that increments are constructed concurrently then integrated one by one to
form another module, a subsystem, or a system.

RECOVERY

In order to recover properly, you must go all the way back to the original
input requirement and then follow the ˜˜decomposition™™ process into the vari-
ous WBS elements (see Cause Description 51c). The next step is to go through
the composition and unit testing of the modules (see Cause Description Family
59) and the interface requirements of the modules. You are looking for speci¬c
inconsistencies, so this document cannot possibly address the questions, much
less the answers, to these issues.
RECOVERING FROM TECHNICAL PROBLEMS 133


55c (NO) Prototype changes were not incorporated into the design
using the Change Control Process.
Prototype changes were not incorporated into the design using the Change
Control Process (see Technical Cause Descriptions 61a, 61b, and 61c) when
the changes are not traceable through the product to the documentation that
authorized the change.

RECOVERY

Prototypes must be treated the same as First Articles in their development.
This is particularly true if your process takes the prototype (or Breadboard or
Brassboard) directly to development or production. Granted, many changes are
devised during the prototype process and incorporated into the prototype to
validate their ef¬cacy. Indeed, that™s what the prototype process is all about.
However, if it is concluded that the in-process change should be a part of the
prototype, the changes must be incorporated through the Change Control
process.
See the glossary for de¬nitions of all these terms.

55d (NO) Each prototype change was not reviewed and accepted by
the originator of the requirements.
Each prototype change was not reviewed and accepted by the originator of
the requirements whenever an acceptance of the change is not documented and
made a part of the project documentation.

RECOVERY

When you get a verbal requirement, stop and document the requirement,
even if it™s just a note. You can use that documentation as a basis for sign-off
by the originator of the requirements.

56 SUBCONTRACTS
56a (NO) The sum of all subcontracts does not re¬‚ect all tasks
allocated.
The sum of all subcontracts does not re¬‚ect all tasks allocated if the totality
of all subcontracts and all work to be performed internally do not add up to the
total requirements in the requirements document (contract).
BLUEPRINT FOR PROJECT RECOVERY
134

RECOVERY

This situation results in either a duplicity of effort (overlaps) or a shortfall
of effort (holes). Neither is acceptable. However, when evaluating subcontracts
for holes and overlaps, consideration must be given to the fact that there will
be some perceived overlaps with respect to required processes. For instance, if
a certain quality program is required of the overall program, it must be ¬‚owed
down to each of the subcontractors. In this case, it might appear to be an over-
lap, but it is not. It is, in fact, an appropriate allocation of requirements.
Obviously, the ¬rst thing to be done is to identify the holes and the overlaps.
You must have some idea that there are holes or overlaps or you wouldn™t be
here in the ¬rst place. Once again, create or review the Requirements Traceabil-
ity Matrix (RTM) and the Requirements Flow-Down Plan.
If you do not have an RTM, you can use Table 5-6 as a start. Modify the
table for your own needs. Just be sure not to change the concepts of content
and ¬‚ow.
Ta b l e 5 - 6 ” R e q u i r e m e n t s Tr a c e a b i l i t y M a t r i x ( R T M )

Unit System
SOW/ WBS S/C SOW/ Test Test
Spec Para Requirement Number Spec Para Number Para Monitor
SOW
4.3.1 Security 06-03-02 N/A T-0304 4.4.1 Smith


Spec
3.2.1 System weight 02-04-03 3.4.6 T-0045 3.4.1 Jones
shall be less
than 10,000
pounds




If you do not have an RFM, you can use the following table as a start. Modify
Table 5-7 on the following page for your own needs. Just be sure not to change
the concepts of content and ¬‚ow.

56b (NO) Each subcontract does not contain all tasks allocated.
Each subcontract does not contain all tasks allocated when the tasks con-
tained in the subcontract are not equal to the assigned tasks from the Flow-
Down Matrix and/or the assigned tasks from the Requirements Traceability
Matrix (RTM).




.......................... 9758$$ $CH5 12-09-02 08:30:15 PS
RECOVERING FROM TECHNICAL PROBLEMS 135

Ta b l e 5 - 7 ” R e q u i r e m e n t s F l o w - D o w n M a t r i x ( R F M )

Company Design S/C Plan S/C A S/C B
Spec Para Reqt WBS Plan Para Para Para Para
1.3.2 02-03-01 5.3.2 5.3.2 1.3.2 1.3.2
1.3.3 02-03-02 5.3.3 5.3.3 1.3.3 N/A
1.3.4 02-03-03 5.3.4 5.3.4 1.3.4 1.3.4
QA Plan 04-01-01 8.2.6 8.2.6 4.3.6 4.3.6
CM Plan 05-01-01 9.3.1 9.3.1 5.6.2 5.6.2




RECOVERY

If you do not have an RFM you can use the following table as a start. Modify
Table 5-8 below for your own needs. Just be sure not to change the concepts of
content and ¬‚ow.
Ta b l e 5 - 8 ” R e q u i r e m e n t s F l o w - D o w n M a t r i x ( R F M )
Y
FL
Company Design S/C Plan S/C A S/C B
Spec Para Reqt WBS Plan Para Para Para Para
AM


1.3.2 02-03-01 5.3.2 5.3.2 1.3.2 1.3.2
1.3.3 02-03-02 5.3.3 5.3.3 1.3.3 N/A
1.3.4 02-03-03 5.3.4 5.3.4 1.3.4 1.3.4
TE




QA Plan 04-01-01 8.2.6 8.2.6 4.3.6 4.3.6
CM Plan 05-01-01 9.3.1 9.3.1 5.6.2 5.6.2




If you do not have an RTM, you can use Table 5-9 on the following page as
a start. Modify the table for your own needs. Just be sure not to change the
concepts of content and ¬‚ow.

57 PURCHASE ORDERS
57a (NO) The sum of all Purchase Orders does not re¬‚ect all
purchases to be made.
The sum of all Purchase Orders does not re¬‚ect all tasks allocated if the
totality of all Purchase Orders does not add up to the total requirements to be
purchased in the requirements document (contract) and in the Design Plan.




.......................... 9758$$ $CH5 12-09-02 08:30:16 PS
BLUEPRINT FOR PROJECT RECOVERY
136


Ta b l e 5 - 9 ” R e q u i r e m e n t s Tr a c e a b i l i t y M a t r i x ( R T M )

Unit System
SOW/ WBS S/C SOW/ Test Test
Spec Para Requirement Number Spec Para Number Para Monitor
SOW
4.3.1 Security 06-03-02 N/A T-0304 4.4.1 Smith


Spec
3.2.1 System weight 02-04-03 3.4.6 T-0045 3.4.1 Jones
shall be less
than 10,000
pounds




RECOVERY

This situation results in either a duplicity of effort (overlaps) or a shortfall
of effort (holes). Neither is acceptable. However, when evaluating purchase or-
ders for holes and overlaps, consideration must be given to the fact that there
will be some perceived overlaps with respect to required processes. For instance,
if the ˜˜Buy America™™ clause is required of the overall program, it must be ¬‚owed
down to each of the purchase orders. In this case, it might appear to be an
overlap, but it is not. It is, in fact, an appropriate allocation of requirements.
Obviously, the ¬rst thing to be done is to identify the holes and the overlaps.
You must have some idea that there are holes or overlaps or you wouldn™t be
here in the ¬rst place. Once again, create or review the Requirements Traceabil-
ity Matrix, the Requirements Flow-Down Plan and the Design Plan.
If you do not have an RTM, you can use the Table 5-10 on the following
page as a start. Modify the table for your own needs. Just be sure to not change
the concepts of content and ¬‚ow.
If you do not have a Requirements Flow-Down Matrix, you can use Table
5-11 on the following page as a start. Modify the table for your own needs. Just
be sure to not change the concepts of content and ¬‚ow.

57b (NO) Each Purchase Order is not complete.
Each purchase order is not complete unless it contains: reference number,
order date, vendor, contact information, name of item, stock (catalog) number,
number of units, price, delivery schedule, delivery location, and purchaser.
RECOVERING FROM TECHNICAL PROBLEMS 137


Note: Don™t wait for a failure here. It is a good idea to create a list of all
subcontracts and Purchase Orders and maintain a constant status of each. Post this
list in a conspicuous place such as the Planning/Status Room (commonly called the
War Room), available to all.

RECOVERY

Compile all purchases made and outstanding and update each to include all
information listed above.
In future, create a Purchase Order form as a reminder. The Purchase Order
is a simple transaction, and it™s easy to create your own form for control. If you


Ta b l e 5 - 1 0 ” R e q u i r e m e n t s Tr a c e a b i l i t y M a t r i x ( R T M )

Unit System
SOW/ WBS S/C SOW/ Test Test
Spec Para Requirement Number Spec Para Number Para Monitor
SOW
4.3.1 Security 06-03-02 N/A T-0304 4.4.1 Smith


Spec
3.2.1 System weight 02-04-03 3.4.6 T-0045 3.4.1 Jones
shall be less
than 10,000
pounds




Ta b l e 5 - 1 1 ” R e q u i r e m e n t s F l o w - D o w n M a t r i x ( R F M )

Company Design S/C Plan S/C A S/C B
Spec Para Reqt WBS Plan Para Para Para Para
1.3.2 02-03-01 5.3.2 5.3.2 1.3.2 1.3.2
1.3.3 02-03-02 5.3.3 5.3.3 1.3.3 N/A
1.3.4 02-03-03 5.3.4 5.3.4 1.3.4 1.3.4
QA Plan 04-01-01 8.2.6 8.2.6 4.3.6 4.3.6
CM Plan 05-01-01 9.3.1 9.3.1 5.6.2 5.6.2
BLUEPRINT FOR PROJECT RECOVERY
138

are creating your own form though, give consideration to status. Create the
form so that the top line or some other single line contains: name of item,
vendor, order date and delivery date, and other critical information (number of
units, for instance). This one line will be carried forward to the Status List.
Engineering, manufacturing, or anyone else should be able to look at the Status
List and update their schedules from that one line.
Create a Subcontracts/Purchase Order Status List. Use the ˜˜one-liner™™ cre-
ated above to establish a list for status.
58 PRODUCTION/MANUFACTURING
58a (NO) All production/manufacturing processes are not traceable
to standard, customer, or enterprise processes (see
glossary).
All production/manufacturing processes are not traceable to standard, cus-
tomer, or enterprise processes when the heritage of the process is not clearly
referenced in the process.
RECOVERY

Create a table similar to Table 5-12 below with your data inserted.
The purpose in creating the table is to determine where the ˜˜holes™™ exist in
the trail. Of course, you should start with the references and work toward the
appearances. If you work the other way, the table may be self-satisfying and of
no use.
After you complete that part of the process, start with the production/manu-
facturing process itself and work backward. If a clear trail exists, the question
will answer itself. If not, the questions to be asked are: ˜˜Why is this process
here?™™ ˜˜What created the need for it?™™ If there is a need but no requirement,
you should forward that need to the responsible department in the enterprise.
If there is a requirement but no need, you should forward that comment also.

Ta b l e 5 - 1 2 ” S t a n d a r d s Tr a c e a b i l i t y M a t r i x ( S T M )

STANDARDS APPEARANCE
Industry Customer Enterprise Project Plan Technical Plan
ISO-9001 ISO-9001 Enterprise Quality Para 4.6.8 Part I, Para
Policy 09350 4.5.6
MIL-STD-100 Enterprise Engineering N/A Part II, Para
Standards 06050 1.2.3




.......................... 9758$$ $CH5 12-09-02 08:30:18 PS
RECOVERING FROM TECHNICAL PROBLEMS 139

You must seriously consider this situation. Is there really no need or do you
just not recognize the need? Discuss the issue with the responsible department
involved.
58b (NO) The line(s) were not properly designed and set up for this
(these) product(s).
The line(s) were not properly designed and set up for this (these) product(s)
if the line does not produce the product according to the requirements.
If the line design, the processes or the materials have not changed, chances
are that the design of the line is okay. Changing either the processes (Cause
Factor 58d/58d(NO)) or the materials (Cause Factor 58a/58a (NO)) can have
an effect on production that was unanticipated. If the line design has changed,
there has likely been an effect on the product that was unanticipated.
RECOVERY

Because line design, processes, and materials are interrelated, they must each
be shown to be proper or improper to isolate the problem and ¬x it. This
involves ¬ve steps:

First: Isolate line design by eliminating processes and materials.
Second: Isolate the part of the line that is contributing to the problem.
Third: Fix the problem.
Fourth: Research the processes and materials and ensure that they are still
compatible.
Fifth: Make a trial run to ensure that all are compatible.

If necessary, iterate the process. When you are satis¬ed that the process is
correct, return to full production
58c (NO) Shop Orders were not correct or thorough.
Shop Orders were not correct or thorough when the end product produced
is not the product that was speci¬ed by the customer. (This statement is made
on the assumption that input materials, labor processes, and the like are proper
and correct.)
RECOVERY

Create a Shop Order that contains all the information necessary to produce
the end product speci¬ed by the customer. If you do not have a Shop Order to
use, consider the following information as a start:




.......................... 9758$$ $CH5 12-09-02 08:30:19 PS
BLUEPRINT FOR PROJECT RECOVERY
140


’ Product to be produced
’ Station producing the product or a portion of the product speci¬cation
reference for the product/portion
’ Process reference for the product/portion
’ Materials needed
’ Tools needed
’ Measurements required
’ Tolerances required/allowed
’ Time allowed
’ Test/performance requirements


Introduce the new Shop Order into the system and carefully monitor the
production of the ¬rst unit. If necessary, iterate the process. When you are
satis¬ed that the process is correct, return to full production.

Additional Resources:

Shop/work order software from AyaNova
Contact: Support@ayanova

58d (NO) The materials were not proper for the processes and the
product(s) and/or did not meet the requirements.

The materials were not proper for the product(s) when the product did not
meet the requirements. This begins a process of ˜˜back tracing™™ the materials
from the product to the source. The trail will, or should, lead from the product
to the Shop Order to the process to the subcontract or Purchase Order to the
original requirement or the derivative of that requirement.

RECOVERY

The ¬rst step that must be accomplished is to determine what materials were
not proper. The next step is to determine why (in functional terms) they were
not proper. While this sounds simple and straightforward, it is not always that
way. The issue itself may well take analysis, not just observation. If this is the
case, conduct or have someone conduct the analysis. It is absolutely essential to
RECOVERING FROM TECHNICAL PROBLEMS 141


know what is not proper and why it is not proper. When you get to the why, it
is likely you will need evidence as to exactly why. Certainly when you get to the
next step, you will need this evidence.
After determining what and why, the next step is to go to the source that
created the problem. Did the department, vendor, or subcontractor that created
the errant element provide what was speci¬ed (either by the provider or by
you)? If it did, the problem is yours. If it did not, absolute evidence is required
to prove the point. If, at this point, there is disagreement regarding responsi-
bility, it will be necessary to make a decision. The decision is whether to shut
the process down until agreement can be achieved or to ¬nd alternative sources
to resolve the problem. The answer to this problem is usually one of time and
money. If the responsible party agrees to ¬x the problem quickly, then that is
the solution. If not, you will likely be better off ¬nding another source and
letting the lawyers handle this one. It™s sort of like being rear-ended in your
car. You were absolutely in the right, but you still can™t drive your car until it™s
¬xed.
Back now to the other side of the problem. That is that the provider pro-
duced exactly what was supposed to be produced and the problem is yours.
Once again, you have several options. First, you can change the speci¬cation in
the subcontract or the Work Order and have the job redone. Second, you can
¬nd another source (if this is a purchased rather than a developed product),
buy that part, and continue. A quick note here”you will likely be responsible
for a thing called ˜˜liquidated damages™™ (see glossary) if you choose this route.
In other words, you ordered 5,000 widgets against your incorrect requirement
and caused a vendor to gear up for that production. You only bought one
hundred, so the vendor has a bunch of these things (or at least the cost) already
built. You are likely to be hit for the costs to make the vendor ˜˜whole™™ again.
Is there any wonder it is so necessary to make absolutely sure the requirements
are traceable and correct?

59 UNIT TEST

59a (NO) Each Unit1 Test does not correctly re¬‚ect the requirement.

Each unit test does not correctly re¬‚ect the requirement when each element
of the unit test is not directly traceable to each element of the unit requirement
(Speci¬cation).
BLUEPRINT FOR PROJECT RECOVERY
142


RECOVERY

Each testable unit should have its own requirement (Speci¬cation). This is
true whether the unit will be built internally or by subcontract. The subcontract
will contain procurement data in excess of the requirement (Speci¬cation) but
will otherwise be the same.
The requirement (Speci¬cation) should be the basis of a Test Plan for the
unit. The Unit Test Plan must de¬ne the:


’ Test performance ¬gures, standards, etc., that the unit must meet
’ Conditions under which the test will be run
’ Support equipment
’ Test equipment
’ Conditions for acceptance of the test
’ Process for documenting and resolving test discrepancies
’ The details and step-by-step process of the test itself


Test Plans vary widely in their composition and content, but the de¬nitions
above are consistent throughout all plans.

Additional Resources:

Hardware Plans Software Plans
MIL-STD-1519/1 ISO/IEC 12207
MIL-STD-2076 MIL-STD-498
MIL-T-18303B MIL-STD-2165
IEEE-STD 416 DI-ATTS-80002
MIL-STD-483 DOD-STD-2168
MIL-STD-499
DI-ATTS-80005
DI-ATTS-81270
DI-ATTS-81273
DI-NDTI-81284
DI-NDTI-81307
DI-SDMP-81475
(continues)
RECOVERING FROM TECHNICAL PROBLEMS 143


DI-ATTS-80002
DI-ATTS-80005
DI-TMSS-80007
DI-QCIC-80204


59b (NO) Each design element that applies to the routine/module/
subsystem does not have its own test case.

Each design element that applies to the routine/module/subsystem does not
have its own test case when there is no direct correlation between the require-
ment and the elements tested in the unit, subsystem, or system test.

RECOVERY

Return, once again, to the Requirements Traceability Matrix (RTM) and fol-
low the columns to the right. There should be entries in the testing columns
showing where the requirement is tested and proved. If this does not ring true
or if you do not have an RTM, this is the time to build one. Refer to Attachment
7, Requirements Traceability Matrix.
In general, your RTM should look similar to Table 5-13.
Once the RTM is complete and the test reference data is appropriately en-
tered, it should expose the holes in the test string.


Ta b l e 5 - 1 3 ” R e q u i r e m e n t s Tr a c e a b i l i t y M a t r i x ( R T M )

Unit System
SOW/ WBS S/C SOW/ Test Test
Spec Para Requirement Number Spec Para Number Para Monitor
SOW
4.3.1 Security 06-03-02 N/A T-0304 4.4.1 Smith


Spec
3.2.1 System weight 02-04-03 3.4.6 T-0045 3.4.1 Jones
shall be less
than 10,000
pounds
BLUEPRINT FOR PROJECT RECOVERY
144


59c (NO) Unit Test ¬ndings were not reviewed for completeness and
not forwarded to be incorporated into Subsystem Tests and
the System Test.
If this is the condition with which you are faced at the subsystem or system
level, you have a lot of work to do. The initial requirements must be proved
somewhere in the test process.

RECOVERY

The quick and dirty method of recovery is to lay out the initial requirements
as stipulated in the speci¬cation and then identify where the requirement was
proved in the System Test. If you can prove that the requirements were incorpo-
rated and your customer accepts the process, you are in luck. If not, you will be
relegated to the ˜˜complete documentation™™ method.
The complete documentation method requires starting with the Require-
ments Traceability Matrix (RTM) and ¬‚owing each requirement into the unit
and subsystem that will be built as a part of the system. The requirements must
¬rst be ¬‚owed down from the requirements document (contract) to the speci-
¬cation for each unit and subsystem and then ¬‚owed up in the Unit Test Plan,
the Subsystem Test Plan and the System Test Plan. This methodology is espe-
cially critical in the development of a secure system where security quali¬cation
must be proved and certi¬ed at every level of development. That philosophy
really should ensure whether the system is secure or not; then, there is no ques-
tion.
If you do not have an RTM, refer to Cause Description ˜˜51a All Critical
Success Factors (CSFs) such as MTTR, MTBF, etc., have been documented and
understood™™ and Attachment 7 for suggestions for how to develop your RTM.
If you do not have a Unit Test Plan (UTP), refer to Cause Description ˜˜59a
Each Unit Test correctly re¬‚ects the requirement™™ for suggestions for how to
develop your UTP.
If you do not have a System Test Plan (STP) refer to Cause Description ˜˜60c
(NO) The System Test has not tested all elements of the system concurrently™™
for suggestions for how to develop your STP.

59d (NO) All Problem Test Reports (PTRs) were not captured,
dispositioned, or worked off.
All (PTRs) were not captured, dispositioned, or worked off when there is not
complete accountability for every error that occurred during test conduct.
RECOVERING FROM TECHNICAL PROBLEMS 145


Those errors were not assigned to responsible individuals for correction and
the results were not worked into the system. The System Test as written was
subsequently not run without error.

RECOVERY

Every test run should have PTR forms available to capture any anomalies
that occur during the conduct of the test. Further, there must be a PTR log in
which to record the PTRs and account for each and every one. Usually, a se-
quence number is assigned to a PTR preceded by a unique Alpha that relates to
the test. For example, the ¬rst PTR for the System Test could be numbered as
ST-001.
If you do not have a PTR system, consider using the information to create a
form for your own use:

Y
’ PTR No.

FL
Priority
’ System
AM


’ Subsystem
’ Test Conductor
’ Test Title Run No.
TE




’ Short Title of Problem
’ Description of Problem
’ Disposition: Responsibility
’ Scheduled Correction Date
’ Action Taken
’ Completed By
’ Date
’ Accepted By
’ Date


In addition to a PTR Form, you should have a PTR Log to collect the actions
of all the PTRs opened. The PTR Log should contain:
BLUEPRINT FOR PROJECT RECOVERY
146


’ PTR No.
’ Priority
’ Short title of problem
’ Name of person to whom assigned
’ Date initiated
’ Date to be completed
’ Date actually completed
’ Date accepted
’ Name of person accepting the PTR closure

60 SYSTEM TEST

60a (NO) The System Test Plan/Procedure was not approved by the
customer.

The System Test Plan was not approved by the customer when the System
Test Plan/Procedure has not been provided to the customer with lead time ade-
quate for customer review or the customer does not agree with or approve the
¬nal content or the customer returns the plan/procedure without review and/
or approval or the customer has not been previously apprised of the content of
the plan/procedure.
If the ¬rst time the customer sees the System Test Plan is when the customer
arrives for the System Test, you can plan on a lot of stoppages, a lot of explana-
tions, and possibly a disapproval of the entire System Test.
At this point, it should be clear that the System Test Plan should be ¬nalized
and forwarded to the customer with adequate time for review. If the customer
does not review the plan or does not approve the plan, revise the schedule to
ensure that the plan has been approved by the customer. Proceeding into test
without customer approval of the Test Plan and Test Procedure is a sure way to
ensure failure. Believe it or not, there are some customers around who will
sandbag the process just to keep their options open. You cannot allow this to
happen.

RECOVERY

The best solution to this problem is to not let it happen in the ¬rst place.
That is achieved by scheduling the completion of the System Test Plan and
RECOVERING FROM TECHNICAL PROBLEMS 147


System Test Procedure as a part of the Data Plan. An understanding and agree-
ment must be made that the documents will be reviewed and approved by the
customer within a certain time period (two weeks to one month are usual).
Your agreement and schedule can allow for iteration, if necessary, but the Sys-
tem Test should not be scheduled until ¬nal approval of the System Test Plan
and the System Test Procedure are approved by the customer.
Your agreement with the customer should be such that nonapproval will
result in a project stoppage. Due diligence will determine who is at fault for
nonapproval and thus be the basis for compensation, if any.
If you did not get this approval early in the project and are now faced with
going into System Test without approval, it is my recommendation that you
stop and sit down with the customer and get approval before proceeding. Unless
you have mutual understanding of all elements of the test, many issues will be
unresolvable and must be run again and again. This will be extremely costly,
probably to you.
For an outline of a System Test Plan, see Data Item Description DI-ATTS-
80005.

60b (NO) The System Test is not traceable to the requirements.

The System Test is not traceable to the requirements when each requirement
is not forward traceable through the unit and the subsystem to the system via
the Requirement Traceability Matrix (RTM) or each requirement is not tested
at least once at the appropriate level (i.e., at the unit level or at the subsystem
level) or system level requirements are not visible and backward traceable to
the requirement through the RTM. The exceptions to this statement are those
requirements that are only visible at the system level. Your RTM should re¬‚ect
this situation by showing the requirement in the leftmost column and the place
where it is tested in the System Test. All columns in between will have no entries
or dashes.

RECOVERY

At this point, you are in for a lot of work. It is not adequate to simply trace
a single requirement back through the RTM; each and every requirement must
be traced back through the RTM. The reason is that if a requirement is visible
at the system level but has not been accommodated at the subsystem or unit
level, there exists a potential point of failure buried within the system.
BLUEPRINT FOR PROJECT RECOVERY
148


Return to Cause Description 60d and accomplish the tasks listed there. Then,
repeat the testing advocated in Cause Descriptions 60a, 60b, and 60c.

60c (NO) The System Test has not tested all elements of the system
concurrently.

The System Test has not tested all elements of the system concurrently when
all elements of the system are not called into play as they will be whenever the
system is operating in its normal mode.
The purpose of the System Test is to test the entire system together. This
process tests the interfaces and the loadings of the system. In many systems, it
is normal that all units or subsystems or modules are not operating at the same
time but rather are operating at some predetermined or commanded sequence.
This is the way the system should be tested, as if it were performing the tasks it
is required to perform. If the system is not tested in its operating mode, it is
not a system test at all but rather a series of subsystem or unit tests.
Generally, the customer will de¬ne the system test coverage and scenarios,
and you will design the Test Plan around these criteria. The best way to design
a System Test is to create and document the System Test as the system is being
designed. This is not necessarily a day-to-day activity but should certainly be
accomplished before each major review so that the design and the testing are
concurrent. Using a documented Con¬guration Management Process, the test
process should require revision of the test after each major revision and follow
the same level of review as the system itself.

RECOVERY

If you are at this point and recovery is necessary, it is clear the System Test
was not created concurrent with the system. You are probably the recovery
project manager because the one that got the project to this point is somewhere
else!
There are four steps that must be taken:


First, a lot of documentation must be reviewed. The original requirement
and all changes to the baseline must be reviewed. The baseline must be updated
(or, if you are really in trouble, created) to re¬‚ect the last documented baseline
of the system. The baseline would now be established, but it must be validated.
This is the tricky part. The only way to validate the current requirements base-
RECOVERING FROM TECHNICAL PROBLEMS 149


line is to negotiate it with your customer. From the customer™s standpoint, this
could be an opportunity to add in all those things they wanted but couldn™t
afford. From your standpoint, you must insist that the customer provide docu-
mentation of the original baseline and each change afterward. Verbal changes
must not be accepted.
Second, the system must be physically and functionally baselined. This will
probably take a lot of time, but it must be done. Keep very close time records
of this activity so the time can be properly allocated during negotiation.
Third, you must negotiate the differences with your customer, and responsi-
bility must be assigned. If the customer has established and documented a valid
baseline with changes and you (your company) has accepted these changes or,
at least, not refused the changes, that™s what you must work to, no mater how
much it hurts. Any change to that statement must be a management decision
because there are all kinds of legal rami¬cations.
Fourth, when the physical and functional baseline has been established and
the requirements negotiated, they must be brought together. Usually, this re-
quires changing the system. At this point, you can write (rewrite) the System
Test with reasonable assurance it will be correct. Sometimes, at this point, there
are other changes the customer sees he wants. This time, keep up with the
changes in the System Test!
60d (NO) The System Test was not performed under appropriate
load(s).2
The System Test was not performed under appropriate load(s) when the
loads on the system are not the loads required by the speci¬cation.
RECOVERY

If the speci¬cation does not stipulate loads, it is best to create ˜˜reasonable™™
loads, as determined by engineering analyses, and perform the system tests
under those loads. Those loads and that fact must be documented and presented
to the customer/client prior to ¬nal acceptance. The best time to present these
issues is in the ¬rst Design Review. If the customer/client has an issue with the
loads, it is a point for negotiation.
60e (NO) The System Test was not performed using the same kind of
personnel that will be used by the customer.
The System Test was not performed using the same kind of personnel that
will be used by the customer with regard to training, education, experience, etc.,
BLUEPRINT FOR PROJECT RECOVERY
150


or the personnel speci¬ed by the customer. To conduct a System Test with
engineers instead of Level X technicians is an invalid test even when you follow
all the procedures in the test. If the speci¬cation does not stipulate operating
personnel level, it is best to assume reasonable operating levels, as determined
by engineering analyses, and perform the system tests using those personnel.
Those operating levels must be documented and presented to the customer/
client prior to ¬nal acceptance. The best time to present these issues is in the
¬rst Design Review. If the customer/client has an issue with the created loads,
that is a point for negotiation.

60f (NO) The System Test was not properly documented and did not
incorporate the test results of all prior-level tests.
The System Test was not properly documented and did not incorporate the
test results of all prior level tests when the results of the unit level tests and the
subsystem level tests are not clearly visible in the construct and conduct of the
system test.
RECOVERY

Assumption: The Unit Tests were properly constructed, and each associated
group of units constituted an appropriate subsystem.
Pull together all the subsystem tests:

’ Check the traceability of each unit and unit test to the appropriate sub-
system.
’ Check the inputs and outputs of each subsystem.
’ Check the interfaces and interface compatibility of each interfacing sub-
system.
’ Check the loads of each subsystem.
’ Make changes as necessary.
’ Modify the system test as necessary and recheck the traceability to the
requirements.
61 CONFIGURATION MANAGEMENT
61a (NO) The Con¬guration Management Plan (CMP) is not
thorough, complete, or authorized.
The Con¬guration Management Plan (CMP) is not thorough, complete, or
authorized unless it follows the required format and maintains the required
RECOVERING FROM TECHNICAL PROBLEMS 151


content as speci¬ed in customer or company con¬guration management policy.
Further, the CMP is not authorized unless it is signed by an authority that is
authorized to sign such documents (usually the vice president or director of
engineering or an equivalent position).
The Con¬guration Management process could easily be looked upon as the
Janus process. Remember the Roman god Janus who looked both backward
and forward? That™s what the con¬guration management process does. It looks
backward to the baseline, as established, and forward to the test process that
will prove the viability of a change.

RECOVERY

If you do not have a CMP, consider the following outline. The detail of what
should be contained in each section can be found in Attachment 5. Modify the
outline as necessary for your purposes:

1. Introduction
2. Reference documents
3. Organization
4. Con¬guration management phasing and milestones
5. Data management
6. Con¬guration identi¬cation
7. Interface management
8. Con¬guration control
9. Con¬guration status accounting
10. Con¬guration audits
11. Subcontractor/vendor control

Con¬guration Management should be treated on at least at two levels. The
generally accepted levels are, appropriately enough, Class I and Class II. Class I
changes are those that affect form, ¬t, or function while Class II changes are
those that affect only documentation. Class I changes require convening a full
Con¬guration Management (or Control) Board, often called the CCB. Class II
changes only require the concurrence of the head of the CCB.
The Con¬guration Management Plan should be prepared for speci¬c project
use and generally follow the requirements of MIL-STD-973, MIL-STD-483,
BLUEPRINT FOR PROJECT RECOVERY
152


MIL-STD-61, EIA-649, or ISO 10007, as determined by the requirements docu-
ment (contract).
The purpose of the Software Con¬guration Management (SCM) plan is to
achieve the ˜˜Repeatable™™ level on the Software Engineering Institute™s (SEI™s)
Capability Maturity Model (CMM) and meeting the ISO/IEC 12207/MIL-STD-
498 and the additional MIL-STD-973 requirements.

Additional Resources:

The following may contribute to developing your plan:

Data Item Descriptions (DIDs)
DI-CMAN-81343
DI-CMAN-80858A

61b (NO) Change requests were not presented and approved by an
appropriate level of the Review Board.

Change requests were not presented and approved by an appropriate level
of the Review Board when the presentations and approvals did not follow the
Con¬guration Management Plan (CMP).

RECOVERY

You should have a CMP (See Attachment 5) containing a Con¬guration
Control Section. A Change Process should be part of the Con¬guration Control
Section.

Additional Resources:

The following may contribute to developing your plan:

Data Item Descriptions (DIDs):

DI-CMAN-81343
DI-CMAN-80858A
RECOVERING FROM TECHNICAL PROBLEMS 153


Standards:

MIL-STD-973
MIL-STD-483
MIL-STD-61
EIA-649
ISO 10007
ISO/IEC 12207
MIL-STD-498

61c (NO) Version controls are not in place and are not re¬‚ected on
(in) the product.

Version controls are not in place and are not re¬‚ected on (in) the product
when the affected product is not appropriately marked with the version which
describes it in the Version Description Document (VDD) (see glossary) and to
which it has not been tested.

RECOVERY

Create a version system and a Version Description Document where:
The general convention for document versions is:

X.YZ
Where: X Major issue or re-issue containing fundamental changes.
Y Minor change containing use changes or additional modules.
Z Minor use changes or documentation clari¬cations.

If you do not have a procedure for Version Description Documents, you can
use DID DI-IPSC-81442 as a guideline. Its contents are as complete as any one
document can get. The outline follows (the headings without content should be
self-evident. If you need further direction, refer to the DID):

1. Scope
1.1 Identi¬cation
1.2 System overview
BLUEPRINT FOR PROJECT RECOVERY
154


1.3 Document overview
2. Referenced documents
3. Version description
3.1 Inventory of materials released
3.2 Inventory of software contents
3.3 Changes installed
3.4 Adaptation data
3.5 Related documents
3.6 Installation instructions
3.7 Possible problems and known errors
4. Notes
A. Appendices

Additional Resources:

MIL-STD-973
DID DI-IPSC-81442

62 SYSTEM EFFECTIVENESS FACTORS

62a (NO) All required System Effectiveness Factors3 have not been
appropriately considered.

All required System Effectiveness Factors have not been appropriately consid-
ered unless all the necessary System Effectiveness Factors have been appropri-
ately considered in both the product and the processes.

RECOVERY

Certainly, not all the System Effectiveness Factors are required for every proj-
ect but they are often overlooked. Table 5-14 groups the System Effectiveness
Factors into their usual primary organizations.
The larger a parent organization, the more likely the second listed organiza-
tion will be a separate organizational element. This is important because, if
System Effectiveness Factor is a distinct organizational element, its function is
more likely to be addressed. When these functions are ˜˜buried™™ in engineering,
RECOVERING FROM TECHNICAL PROBLEMS 155


Ta b l e 5 - 1 4 ” S y s t e m E f f e c t i v e n e s s P a r e n t O r g a n i z a t i o n

System Effectiveness Factor Usual Parent Organization
Reliability Engineering or Reliability & Maintainability
Maintainability Engineering or Reliability & Maintainability
Vulnerability/Susceptability Engineering or Electro Magnetic Interference
Transportability Engineering(1) or Transportation(2)
Supportability Engineering or Logistics
Producibility Engineering or Manufacturing
Quality Quality
When transportability is a function of the product (i.e., a communications shelter).
1

When transportability is a function of delivery of the product to its destination.
2




the likelihood is greater that they will be glossed over or even ignored. These
functions must be addressed even in the smallest organizations. The most eco-
nomical way to accomplish this task is to designate a person to be responsible
Y
for each of the applicable System Effectiveness Factors and to question or defend
FL
that function during reviews. The project manager must ensure that all the
required System Effectiveness Factors are addressed in all processes. This is par-
AM


ticularly true in Design Reviews.

Notes
TE




1. De¬ned as the smallest stand-alone component that produces a de¬nable output
from a de¬nable input. The unit may be hardware or software. In the case of hard-
ware, power can be external (i.e., a separate unit).

2. Loads are stresses placed upon a system. Loads are those stresses in units typical
for the product such as pounds, watts, ergs, number of subsystems, number of users,
number of executions per second, I/O rates, number of queries per second, etc.

3. The System Effectiveness factors refer to Reliability, Availability, Maintainability,
Supportability (including Logistics), Susceptibility, Producibility, Human Engineer-
ing, Safety, and Security.
CHAPTER 6




EXPANDING THE CAUSE
BASE FOR YOUR PROJECT

In developing this book, I had to take a ˜˜middle-of-the-road™™ position with
regard to speci¬c Cause Descriptions and then present a methodology or proc-
esses for deviating from that position. That™s where you are right now.
You may well need to expand your cause database for any number of reasons.
Most likely, you are dealing with an area that wasn™t addressed in the creation
of the Search Tables or Cause Descriptions presented in Chapter 1. Perhaps you
are dealing with a product or service area such as health care or pharmaceuticals
or construction. While the basic precepts presented in Chapter 1 are appropriate
for most product and service areas, the speci¬cs may well be different.
Expanding the cause database is fundamentally a problem-solving process.
As such, it follows the traditional problem-solving steps. While there is any
number of advocates of slightly different steps in the process, I ¬nd that Mary
Ellen Guffey™s1 ¬ve steps are typical. These are:

1. Identify the problem. The ¬rst step in reaching a solution is pinpointing
the problem area.
156




.......................... 9758$$ $CH6 12-09-02 08:30:00 PS
EXPANDING THE CAUSE BASE FOR YOUR PROJECT 157


2. Gather information. Look for possible causes and solutions. Check ¬les,
call suppliers, or brainstorm with fellow workers.
3. Evaluate the evidence. How accurate is the information gathered? Is it fact
or opinion?
4. Consider alternatives and implications. Weigh the advantages and disad-
vantages of each alternative. What solution best serves your goals and
those of your organization?
5. Choose and implement the best alternative. Select an alternative and put
it into action. Then, follow through on your decision by monitoring the
results of implementing your plan.

6.1 General

The purpose of this chapter is to assist you in expanding your database by
creating new causes and Cause Descriptions; these will usually be causes that
are unique to your business area and your products.
The creation technique I recommend using to start the expansion process is
brainstorming. The reason brainstorming is used before the other data collec-
tion and review techniques is so that you approach the issue with an open mind.
That is, you are not biased in favor of the data you will uncover during the data
collection techniques. After brainstorming, you review the processes that are
unique to your market, your customer, your enterprise, and your products.
These are normally referred to as ˜˜benchmarks.™™ The basic Family of Causes
can be expanded to include these business-unique causes. Likely, you will need
the assistance of one or more of your staff organizations to get started on this
task. Then you can research the processes that are standard to your business
area, common to your customer(s), required by your company, and normal for
your kind of project or program.
By following the precepts of this chapter, you will expand the basic Family
of Causes that includes the Search Tables and the Cause Descriptions. You will
eliminate holes and overlaps and you will have tailored the Family of Causes to
your speci¬c needs.
Table 6-1 is a listing of the Expansion Methodologies, together with their
purposes, that you can expect in this chapter. You must be the judge of how
many of these methodologies you need to use for your particular problem set.
Base your choice on the purpose of each of the methodologies as shown in the
table.
BLUEPRINT FOR PROJECT RECOVERY
158


Ta b l e 6 - 1 ” E x p a n s i o n M e t h o d o l o g i e s

Process Purpose

<<

. 5
( 9)



>>