Acceptance Sampling Update

INTRODUCTION An understanding of statistical principles and how to apply them is critical to ensuring compliance with FDA requirements, such as those in the current and July working draft of the good manufacturing practices (GMP) regulation.   Indeed numerous Form 483’s issued by FDA following GMP inspections have cited sampling plans for final, in-process, and receiving inspection of devices and their components for not being “statistically valid”.  What exactly is required for a sampling plan to be statistically valid?   After a brief overview of how sampling plans work, this article will address that question. Several other sampling-related issues currently facing the medical device industry will also be discussed.   In February of this year, the U.S. Department of Defense canceled Mil-Std-105E, which contained a widely used table of sampling plans.   What are the alternatives and how should they be used?   As the industry focuses increasingly on the prevention of defects and statistical process control (SPC), will the need for acceptance sampling disappear?   And finally, how can the cost of acceptance sampling be reduced to help manufacturers remain competitive in today’s marketplace? HOW SAMPLING PLANS WORK Stated simply, sampling plans are used to make product disposition decisions for each lot of product.   With attribute sampling plans, these accept/reject decisions are based on a count of the number of defects and defectives, while variables sampling plans require measurements and calculations, with decisions based on the sample average and standard deviation.  Plans requiring only a single sample set are known as single sampling plans; double and multiple sampling plans may require additional samples sets.  For example, an attribute single sampling plan with a sample size n=50 and an accept number a=1 requires that a sample of  50 units be inspected.  If the number of defectives in that sample is one or zero, the lot is accepted.  Otherwise, it is rejected. Ideally, when a sampling plan is used, all bad lots will be rejected and all good lots accepted.  However, because accept/reject decisions are based on a sample of the lot, there is always a chance of making an incorrect decision.  So what protection does a sampling plan offer?  The behavior of a sampling plan can be described by its operating characteristic (OC) curve, which plots percent defectives versus the corresponding probabilities of acceptance.  Figure 1 shows the OC curve of the attribute single sampling plan described above.  With that plan, if a lot is 3% defective the corresponding probability of acceptance is 0.56.  Similarly, the probability of accepting lots that are 1% defective is 0.91 and the probability of accepting lots that are 7% defective is 0.13.

Figure 1: OC Curve of Single Sampling Plan n=50 and a=1

An OC curve is generally summarized by two points on the curve: the acceptable quality level (AQL) and the lot tolerance percent defective (LTPD).  The AQL describes what the sampling plan generally accepts; formally, it is that percent defective with a 95% percent chance of acceptance.  The LTPD, which describes what the sampling plan generally rejects, is that percent defective with a 10% chance of acceptance.  As shown in Figure 2, the single sampling plan n=50 and a=1 has an AQL of 0.72% defective and an LTPD of 7.6%.  The sampling plan routinely accepts lots that are 0.72% or better and rejects lots that are 7.6% defective or worse.   Lots that are between 0.72% and 7.6% defective are sometimes accepted and sometimes rejected.

Figure 2: AQL and LTPD of Single Sampling Plan n=50 and a=1

Manufacturers must know and document the AQLs and LTPDs of the sampling plans used for their products.  The AQLs and LTPDs of individual sampling plans can be found in Table X of MIL-STD-105E, and Chart XV of ANSI Z1.4 gives the AQLs and LTPDs of entire switching systems (described below).^1,2  Software also can be used to obtain the AQLs and LTPDs of a variety of sampling plans.⁶ SELECTING STATISTICALLY VALID SAMPLING PLANS Documenting the protection provided by your company’s sampling plans is only half the job.  You must also provide justification for the AQLs and LTPDs used.   This requires that the purpose of each inspection be clearly defined.   Depending on past history and other circumstances, sampling plans can be used for a variety of purposes.  In the examples described below, an AQL of 1.0% is specified for inspections for major defects.  The AQL given in this specification is not necessarily equal to the sampling plan AQL, and so it will be referred to as Spec-AQL to make this distinction clear. Spec-AQLs are commonly interpreted as the maximum percent defective for which acceptance is desired.  Lots below the Spec-AQL are best accepted; Lots above the Spec-AQL are best rejected.  The Spec-AQL, therefore, represents the break-even quality between acceptance and rejection.  For lots with percent defectives below the Spec-AQL, the cost of performing a 100% inspection will exceed the benefits of doing so in terms of fewer defects released.  Since this cost is ultimately passed on to the customer, it is not in the customer’s best interest for the manufacturer to spend $1000 to 100% inspect a lot if only one defect is found that otherwise would have cost the customer $100.  Spec-AQLs should not be interpreted as permission to produce defects; however, once lots have been produced, the Spec-AQLs provide guidance on making product disposition decisions. Example 1:  If a process is known to consistently produce lots with percent defectives above the Spec-AQL, all lots should be 100% inspected, but if some lots are below the Spec-AQL, the company could use a sampling plan to screen out lots not requiring 100% inspection.  To ensure that lots worse than the Spec-AQL are rejected, a sampling plan with an LTPD equal to the Spec-AQL can be used, but at the risk of rejecting some acceptable lots.  For a Spec-AQL of 1.0%, the single sampling plan n=230 and a=0, which has an LTPD of 1.0%, would be appropriate.  There is a simple formula for determining the sample size for such studies.  Assuming an accept number of zero and the desired confidence level of 90%, the required sample size is:

n = 230/Spec-AQL

For 95% confidence, the formula is

n = 300/Spec-AQL

Example 2:  The same sampling plan might also be used to validate a process for which there is no prior history.  Before reduced levels of inspection are implemented, it should be demonstrated that the process regularly produces lots below the Spec-AQL.  If the first three lots pass inspections using a sampling plan with an LTPD equal to the Spec-AQL of 1.0%, the manufacturer can state with 90% confidence that each of these lots is <1% defective. However, other sampling plans might be better choices.  Suppose the process is expected to yield lots that are around 0.2% defective.  The sampling plan n=230 and a=0 has an AQL of 0.022% and therefore runs a sizeable risk of failing the validation procedure.  A sampling plan with an AQL of 0.2% and an LTPD of 1% would be a better choice. Using the software cited earlier, the resulting plan is n=667 and a=3.3 Example 3:  Once it has been established that the process consistently produces lots with percent defectives below the Spec-AQL, the objective of future inspections might be to ensure that lots with >=4% defective are not released.  This requires a sampling plan with an LTPD of 4%.  Because the sampling plan should also ensure that lots below the Spec-AQL are released, the plan’s AQL should be equal to the Spec-AQL.  According to Table I from the book Guide to Acceptance Sampling³, which gives a variety of sampling plans indexed by their AQLs and LTPDs, the single sampling plan n=200 and a=4 is the closest match.  It has an LTPD of 3.96% and an AQL of 0.990%, and thus is statistically valid for this purpose. Example 4:  Now suppose that the process has run for 6 months with an average yield of 0.1% defectives and no major problems.  Although the process has a good history, there is still some concern that something could go wrong; as a result, the manufacturer should continue to inspect a small number of samples from each lot.  For example, a sampling plan might be selected that ensures that a major process failure resulting in ≥20% defective will be detected on the first lot.   The sampling data can then be trended to detect smaller changes over extended periods of time. When selecting a sampling plan to detect a major process failure, the nature of the potential failure modes should be considered.  If the primary failure mode of concern is a clogged filter and past failures have resulted in ≥20% defectives, the single sampling plan n=13 and a=0, which has an LTPD of 16.2% and an AQL of 0.4% in Table I³, is statistically valid.  If the potential failure mode of concern is a failure to add detergent to the wash cycle, with a resulting failure rate of 100%, the single sampling plan n=1 and a=0 is valid.

Table I: Single Sampling Plans Indexed by AQL and LTPD³

Example 5:  Finally, one might have a proven process for which procedures are in place that minimize the likelihood of a process failure going undetected.  At that point, acceptance sampling might be limited to the routine collection of data sufficient to plot process-average tends.  There is nothing wrong with simply stating in the written procedures that acceptance sampling is not needed and that the inspections being performed should be considered as process audits. In summary, selecting a statistically valid sampling plan is a two-part process.  First, the purpose of the inspection should be clearly stated and the appropriate AQL and LTPD selected; then, a sampling plan should be selected based on the chosen AQL and LTPD.  Because different sampling plans may be statistically valid at different times, all plans should be periodically reviewed.  If a medical device manufacturer doesn’t know the protection provided by its sampling plan or is unclear as to the purposes of its, it is at risk. MIL-STD-105E AND ANSI Z1.4 The February 1995 cancellation of MIL-STD-105E by the Department of Defense, while directly affecting military purchasing contracts, will also have an impact on the medical device and diagnostic industries.   The cancellation notice indicates that future acquisitions should refer to an acceptable nongovernment standard such as ANSI Z1.4.  To many, this news came as a shock.  However, the change is not nearly as drastic as it first seems.  The differences between ANSI Z1.4 and MIL-STD-105E are minor and the elimination of the later was basically a cost-savings measure to eliminate the duplication of effort associated with maintaining two nearly equivalent standards. Nevertheless, because manufacturers may need to update many specifications as a result of this change,  now is an especially appropriate time to reexamine MIL-STD-105E and Z1.4 and how to use them to select valid sampling plans.   Although the term Z1.4 will be used in the following discussion, all of the ensuing comments apply equally to MIL-STD-105E. Lets us start with what Z1.4 is not.  Used by many industries, Z1.4 is not a table of statistically valid sampling plans.  Instead, it contains a broad array of sampling plans that might be of interest to anyone.  For example, one plan requires two samples and has an accept number of 30.  Such a plan would never be appropriate for a medical device, but is applicable in other industries. Furthermore,  Z1.4 is, in fact, a sampling system.  A user references the sampling plans in Z1.4 by specifying an AQL and a level of inspection, and then following a set of procedures for determining what sampling plan to use based both on lot size and the quality of past lots.  The Z1.4 system includes tightened, normal, and  reduced sampling plans and a set of rules for switching between them.   Although these switching rules are frequently ignored, they are an integral part of the standard.  As Z1.4 states:

This standard is intended to be used as a system employing tightened, normal, and reduced inspection on a continuing series of lots …. Occasionally specific individual plans are selected from the standard and used without the switching rules. This is not the intended application of the ANSI Z1.4 system and its use in this way should not be referred to as inspection under ANSI Z1.4.²

Several companies have received Form 483s from FDA for not using the switching rules, a problem that could have been avoided by having written procedures specifying that the switching rules are not used.  When is the use of switching rules appropriate and when should individual sampling plans be selected instead?  Z1.4 was developed specifically to induce suppliers “to maintain a process average at least as good as the specified AQL while at the same time providing an upper limit on the consideration of the consumer’s risk of accepting occasional poor lots.”²   Thus, the Z1.4 switching system should not be used to inspect isolated lots, nor should they be used to specify the level of protection for individual lots.  In those cases individual plans should be selected instead. One situation warrants special mention.  Acceptance sampling is frequently used for processes that generally produce good product but might on occasion break down and produce high levels of defects.  If protection against isolated bad lots or the first bad lot following a series of good lots is the key concern, the Z1.4 switching rules should not be used or, if they are, the reduced inspection should be omitted.   Because the Z1.4 switching rules are designed to react to gradual shifts in the process average, they frequently fail to detect isolated bad lots and do not react quickly to sudden shifts in the lot quality.   Even when appropriate, the Z1.4 switching rules are complicated to apply.  However, quick switching systems have been developed that are both simpler to use and provide better protection during periods of changing quality.³ Finally, there are two common misconceptions about Z1.4.  Many people believe that the required sample sizes increase for larger lots because more samples are required from such lots to maintain the desired level of protection.  The truth is that the standard specifies larger sample sizes to increase the protection provided for larger lots.  The reason for this increase is based on economics: It is more expensive to make errors classifying large lots; as a result, Z1.4 requires more samples from larger lots to reduce the risk of such errors.  To maintain the same level of protection, one can simply select a sampling plan based on its OC curve and then use this plan for all lots regardless of size.  The single sampling plan n=13 and a=0 provides the same protection for a lot of 200 units as for a lot of 200,000 units.^3,4 The second misconception is that use of Z1.4 ensures that lots worse than the AQL are rejected.  According to this misconception, if the AQL is 1%, lots with >1% defectives are routinely rejected.  The truth is that there is a sizable risk of releasing such lots–one sampling plan with an AQL of 1% accepts lots that are <=16% defective.  The protection provided by the sampling plan is determined by its LTPD, not AQL, which reveals nothing about what a sampling plan will reject.  As a result of this misconception, many manufacturers believe that their sampling plans provide greater protection than they do.  This illusion can lead to the use of inappropriate sampling plans and can provide a false sense of security.  Repeating the advice given earlier, manufacturers should determine and document the actual AQLs and LTPDs of all their sampling plans. While Z1.4 and equivalent standards are widely used by the device industry, rarely are they used in the manner intended.  Most commonly, individuals sampling plans are selected from them.  Other tables are better suited for this purpose, and companies should not be afraid to switch to using those tables.  In addition, using Z1.4 does not ensure valid sampling plans and, in fact, can complicate the selection process. SPC VERSUS ACCEPTANCE SAMPLING? Much has been written about the greater benefits to be achieved by using SPC as opposed to acceptance sampling.  But although preventing defects is certainly more desirable than detecting them through inspection, SPC does not eliminate the need for acceptance sampling.  As indicated in Table II, there are fundamental differences between the two techniques.  In SPC, control charts are used to make process control and process improvement decisions, and actions are taken on the process to ensure that future products are good.  In contrast, sampling plans are used to make product disposition decisions, and actions are taken on previously produced lots to ensure the quality of released product.   Ideally, with SPC in place no defectives will ever be made and acceptance sampling will become unnecessary: in practice, however, all processes have some risk of failure, and thus some procedure for accepting and rejecting product is generally required.

Table II: Differences Between Control Charts and Sampling Plans

Much of the reaction against acceptance sampling is attributed to quality guru W. Edwards Deming, who many believe advocated its elimination.  However, what Deming really called for was ceasing reliance on acceptance sampling.  If more time and resources are spent on acceptance sampling than on process improvement and control, or if a company believes that, no matter what else happens, its sampling plan ensures shipment of only good product, then that company is overly reliant on acceptance sampling.  Instead, its focus should be on defect prevention and continuous process improvement. The real issue is not SPC versus acceptance sampling; it is how to combine the two.  Both techniques require routine product inspections; the trick is to use the same data for both purposes.  When variables sampling is used–that is, when the data consist of actual measurements such as seal strength, fill volume, or flow rate–data can be combined on a single acceptance control chart.  Figure 3 provides an example of such a chart containing fill-volume data.  The inside pair of limits, UCL and LCL, are the control limits.  A point falling outside these limits signals that the process is off target and that corrective action is required.  The outside pair of limits, UAL and LAL, are the acceptance limits.  A lot whose sample falls outside these limits is rejected.  In the figure, lot 13 is outside the control limits but inside the acceptance limits, which indicates that the process has shifted.  Corrective action on the process is required to maximize the chance that future products will be good; however, no action is required on the product lot.  Rejecting whenever a point exceeds the control limits can result in the rejection of perfectly good lots.  Similarly, it is wasteful to wait until the acceptance limits are exceeded before taking corrective action on the process. Therefore, separate limits for process and product actions are required.  Such limits are also frequently called action limits, warning limits, and alert limits.  No matter what the name, however, if the result of exceeding a limit is to act on the process, the limit is serving the purpose of a control limit; if action is instead taken on the product, the limit is serving as an acceptance limit.

Figure 3: Acceptance Control Chart

If attributes sampling is performed, the data must be handled much differently, and care must be taken in implementing SPC so that the resulting change is not illusionary.  Consider, for example, a packing operation that inspects for missing parts using the single sampling plan n=13 and a=0.  Whenever a lot is rejected, an attempt is made to fix the process.  Historically, the process has averaged around 0.2% defective.   When management decides to implement SPC, a p-chart of the inspection data is constructed as shown in Figure 4.  The upper control limit is 3.92%, and samples with one or more defectives exceed this control limit, triggering attempts to fix the process and rejection of recent product.  The company can now state truthfully that SPC is used, but in reality nothing has changed–the same data are collected and the same actions taken.  A better approach is to continue acceptance sampling as before and, because this does not protect against a gradual increase in the process average, to analyze the resulting data for trends.  Figure 5 shows a p-chart of the same data, but with the data from each day combined.  This chart indicates that a change occurred between days 5 and 6; this change is not so apparent in Figure 4. Neither SPC or acceptance testing can detect a problem before defectives are produced.  However, by accumulating data over time, attribute control charts can indicate small changes in the process average that acceptance sampling will not reveal.  Used in combination, sampling plans provide immediately protection against major failures while control charts protect against minor sustained problems.

Figure 4: p-Chart of Inspection Results

Figure 5: Daily p-Chart

REDUCING INSPECTION COSTS Two sampling plans can have the same AQL and LTPD and nearly equivalent OC curves.  When they are followed, the same percentage of good lots will be accepted and the same percentage of bad lots rejected.  The quality of the products released to customers will be the same, as will the reject and scrap rates.  From a regulatory point of view, the two sampling plans are substantially equivalent.  However, one of these plans may be less costly to use. Consider an example.  The ANSI Guideline for Gamma Sterilization⁵ provides procedures for establishing and monitoring radiation dosages.  One procedure is a quarterly audit of the dosage that requires the sterilization of test units at a lower dosage than is actually used for the product.  The test dose is selected to give an expected positive rate of 1%. (A positive is a unit that tests nonsterile.)  For each audit, an initial sample of 100 units is tested.  If two or fewer positives are found, the process has passed the audit; in the event of three or four positives, one retest can be performed.  This quarterly audit procedure has an AQL of 1.50% and an LTPD of 5.55%.  An alternative to this procedure is to test 50 samples, passing on zero positives and failing on four or more positives.  In the event of 1 to 3 positives, a second sample of 100 units is tested.  The audit is considered passed only if the cumulative number of positives in the 150 units is four or less. This double sampling plan has an AQL of 1.36% and LTPD of 5.73%.

Figure 6: OC Curves of the ANSI Quarterly Audit Sampling Plan and an Alternative Plan for Monitoring Sterilization Dosage

The OC curves of both procedures are nearly identical, as shown in Figure 6.  Indeed, these two sampling plans are substantially equivalent procedures, except for the number of units tested.  Figure 7 shows average sample number (ASN) curves for the two plans. If the positive rate is 0.5%, the alternative procedure requires an average of 70 units compared to an average of 102 for the ANSI quarterly audit procedure.  If the alternative plan is used to destructively test expensive medical devices, this difference can mean a sizable savings.

Figure 7: ASN Curves of Audit Sampling Plan and Alternative

CONCLUSION Acceptance sampling is one of the oldest techniques used for quality control, yet it remains poorly understood and misconceptions regarding its procedures and terminology are widely held.  Acceptance sampling does not have to be complicated.   Your company can optimize its procedures by remembering this list of principles:

• The protection level provided by a sampling plan is described by what it accepts — its AQL–and what it rejects– its LTPD.
• Selecting a statistically valid sampling plan requires stating the objective of the inspection, selecting the appropriate AQL and LTPD, and then choosing a sampling plan that provides the desired protection.
• Companies must know the AQLs and LTPDs of all their sampling plans.  It doesn’t matter whether a sampling plan comes from MIL-STD-105E or some other source, and the protection provided by a plan does not depend on the lot size; it’s the AQL and LTPD that reveal what protection the sampling plan provides.
• SPC cannot serve as a replacement for acceptance sampling.  Instead, these two techniques should be combined by using the same data to control the process and to make product disposition decisions.
• Sampling plans with the same AQL and LTPD are substantially equivalent procedures, so costs can sometimes be reduced by using equivalent double, multiple, or variables sampling plans as alternatives to single sampling plans.

REFERENCES

1. Sampling Procedures and Tables for Inspection by Attributes, MIL-STD-105E, Washington D.C. ,  U.S. Government Printing Office, 1989 .
2. Sampling Procedures and Tables for Inspection by Attributes, ANSI/ASQC Z1.4, Milwaukee, WI,  American Society for Quality Control, 1981.
3. Taylor, W A, Guide to Acceptance Sampling, Libertyville, IL, Taylor Enterprises, 1992. (Software is supplied with this book.)
4. Schilling, E G, Acceptance Sampling in Quality Control, New York City, Marcel Dekker, 1982.
5. Guideline for Gamma Radiation Sterilization, ANSI/AAMI ST32-1991, Arlington, VA, Association for the Advancement of Medical Instrumentation, 1992.
6. Sampling Plan Analyzer, Taylor Enterprises, Inc., Variation.com/spa.

Appeared in MDDI (Medical Device & Diagnostic Industry), Oct. 1995,   p. 92-108, Canon Communications

(Used by FDA in new inspector training)

Copyright © 1995 Taylor Enterprises, Inc.

Further information can be found in:

• Book Guide to Acceptance Sampling
• STAT-09, Manufacturing Acceptance Sampling Plans and Inspections, of the book Statistical Procedures for the Medical Device Industry
• STAT-10, Statistical Techniques for Trending Data, of the book Statistical Procedures for the Medical Device Industry
• Software Sampling Plan Analyzer
• Acceptance Sampling Standards page