The FDA takes a new look at Quality Metrics

The FDA has proposed a new approach to the collection of quality metrics from pharmaceutical companies – the third major proposal by the FDA since the initial rollout of a 2015 proposal which received significant pushback from the pharmaceutical industry. But as AgencyIQ has found, FDA has been quietly working on a revamp of this proposal for years, and the latest effort is emphasizing the importance of industry feedback before any proposal is officially put forth.

Regulatory Background

• The FDA regulates thousands of facilities located around the world. Given the breadth of the footprint of the FDA’s monitoring, its current reliance on onsite inspections to gather drug quality data can make it difficult to inspect facilities as often as it would like. This often results in the FDA feeling as if it has gaps in its inspection oversight capabilities.
• In an ideal world, the FDA would be able to regularly receive information related to the quality manufacture of pharmaceuticals and medical products, which it could then use to better understand the risk profiles of various manufacturers. This, in turn, would permit the FDA to conduct risk-based inspections.
• This need for real-time quality information was accentuated during the pandemic, when the FDA found itself unable to physically inspect many facilities. As a result, it conducted remote inspections for some facilities, relying on authority it received in 2012 under the FDA Safety and Innovation Act (FDASIA). This authority allowed the agency to collect records from pharmaceutical and biopharmaceutical facilities “in advance of” or “in lieu of” a physical inspection. The authority was originally intended to support FDA’s efforts to conduct efficient inspections, such as by allowing it to see a facility’s records prior to an inspection, thereby allowing it to conduct a more targeted inspection once its employees entered a facility.
• However, after receiving this authority, the FDA began to think creatively about how it might use that authority in other, more advanced ways. The biggest of these ways began in 2015 when the agency issued a draft guidance document proposing that drug manufacturers submit so-called drug quality metrics to the agency on a regular basis. That original proposal floated the drug quality metrics approach as a requirement, and proposed that the FDA collect several specific metrics from companies.
• As explained by the FDA at the time: “In order to both inform FDA’s risk-based drug inspection scheduling and to better detect manufacturing conditions that may lead to a shortage, FDA intends to collect and use quantitative quality data to calculate certain quality metrics, as further described in [the guidance]. FDA intends to use these quality metrics, in part, as a tool to identify risk-based factors that could increase or decrease inspection frequency and that could potentially be predictive of drug supply disruption.”
• In its guidance, the FDA proposed to collect four specific metrics it believed was correlated with drug quality: Lot acceptance rate, product quality complain rate, invalidated out-of-specification rate, and the annual product review/product quality review on-time rate. In addition, the FDA requested that companies consider whether additional quality metrics might be suitable, such as “Senior Management Engagement,” “CAPA effectiveness,” and others.
• Industry response to the guidance was overwhelmingly negative and raised significant operational and legal concerns. Among the most serious points of contention: A guidance document is not intended to be a legally binding document. The FDA cannot establish requirements in a guidance document – only recommendations. As such, and requirement that companies submit quality metrics data would be out-of-scope for the guidance.
• In response to industry pushback, the FDA published a subsequent guidance in November 2016 that framed the quality metrics reporting as voluntary for a limited time. Specifically, the FDA revised the language to provide for a “voluntary phase of the quality metrics reporting program” under which the FDA would not “take enforcement action based on errors in a quality metrics data submission” for data submitted in good faith.
• In response to both the draft and revised draft guidance, however, industry also raised significant concerns about the specific quality metrics sought by the FDA.  The requested metrics—lot acceptance rate, product quality complaint rate and an out-of-specification metric—were not going to give the agency much meaningful insight into manufacturing quality, industry argued.
• Following industry’s response to the revised draft guidance, the FDA retreated somewhat and tried to build a foundation to better support its efforts. To do so, it quietly began working with a group associated with St. Gallen’s Institute of Technology Management. This Swiss university-based operation maintained a multi-year association with the FDA to investigate pharmaceutical manufacturing performance metrics with the goal of assessing quality; and to forecast the risk of quality failures and downstream drug shortages. Beginning in 2016 and spanning the course of several years, the project has tapped St. Gallen’s Operational Excellence Benchmarking Database to ascertain which quality metrics are most predictive, with the aim of building a risk-based surveillance model.
• There have been other hints that the FDA has been continuing work on its drug quality metrics effort, too. In May 2020, the FDA sent out a survey asking pharmaceutical manufacturers to participate in another year of drug quality benchmarking research with St. Gallen, with the support of commercial data powerhouse Dun & Bradstreet. In August 2020, FDA issued a request for quotations (RFQ) seeking a third party to develop an assessment for measuring Quality Management Maturity (QMM) and train its staff on how to use it. While the FDA didn’t outline strict requirements for what the assessment should look like, the RFQ noted interest in things like response to shortages, regulatory compliance, inventory management, consistency across sites, planning and quality culture. In October 2020, FDA began two new pilot programs to assess companies’ QMM, with measures of maturity bearing striking resemblance to the agency’s original quality metrics proposal.
• The pandemic has also played a key role in the FDA’s efforts. In addition to the FDA’s focus on remote inspection capacity and risk-based inspections, agency officials have frequently spoken about the need to anticipate and mitigate drug shortages, the need to understand which facilities might be at risk of disruptions in emergencies, and even supply chain redundancies. Consider this Tweet from the FDA’s top drug quality official, Michael Kopcha, wondering aloud if “there were public ratings of a manufacturing facility’s quality management maturity that patients and purchasers could consider when making buying decisions about medicines,” would they be able to avoid purchasing products from entities most likely to be at risk of drug shortages?
• Also consider this opinion piece from former FDA Commissioner Stephen Hahn and former Deputy Commissioner for Medical and Scientific Affairs Anand Shah, who argued that a quality rating system was a matter of national security. “Action to onshore America’s critical medical supply chain and modernize infrastructure for life sciences manufacturing is long overdue.” The pair went on to say that the country needs “enduring solutions that address the root causes of America’s medical manufacturing crisis,” including the “creation of a quality rating system” and investment into continuous manufacturing.
• In fact, the FDA already has a quality rating system—just not a public one. A substantial report issued by the FDA’s Office of Pharmaceutical Quality in June 2020 on the state of pharmaceutical quality in the life sciences industry indicated that the FDA generates “site inspection scores” for facilities. Those scores are not publicly available. The FDA has also used other metrics in its public-facing research. In August 2020 FDA quality officials published a study on the quality of difficult-to-make prescription pharmaceutical products marketed in the US. For that study, the FDA developed a metric called Process Performance Index based on dosage unit uniformity and dissolution.

Despite industry pushback, there FDA has been able to learn quite a bit from its voluntary Quality Metrics Pilot Program, it said this week in a new Federal Register announcement. And what it learned is going to affect what it does next.

• The Quality Metrics Pilot Programkicked off in 2018, piloting site visits that gave the 14 organizations who volunteered to participate “an opportunity to explain the advantages and challenges associated with implementing and managing a Quality Metrics Program.” Examples of challenges cited by the FDA included a poor match between how the FDA defined the metrics it was proposing to solicit, and how industry defined them. Through the on-site facility tours and meetings, the FDA said it learned more about how quality metrics are defined, obtained, and used to effect quality improvement and, ultimately, avert shortages.
• The Quality Metrics Feedback Program saw participating volunteers present their own quality metrics programs to the FDA. Ensuing discussions dove deeply into questions of data use and statistical and analytical techniques, according to the Federal Register announcement.
• For its part, the FDA “plans to analyze the data using advanced statistical techniques” such as data and text mining, as well as through statistical techniques like time series analysis and machine learning processes such as natural language processing (NLP).
• From these pilots, the FDA was able to dial in a Quality Metrics approach the agency hopes will work for industry. A key learning was that a one-size-fits-all approach, no matter how simple, will not work. “Different industry sectors prefer different metrics due to their individual operations and business needs,” according to the announcement, so a flexible approach that allows industry to choose from a menu of options is key.
• Metrics should be meaningful to the given practice area, said the agency. This concept follows on from the need for flexibility; the metrics should be useful in driving decision-making for the organization. And a combination of metrics might be more useful than a single data point, the FDA noted.
• Most pilot participants wanted to report at the establishment level, segmenting by product, though there were exceptions for vertically integrated firms, for example.
• Some of the agency’s proposals for metrics in the first two draft guidance documents represented a swing and miss, the FDA conceded. There were three key metrics the agency had zeroed in on for the 2016 draft guidance: Lot Acceptance Rate (LAR), which shows how effectively the manufacturing process is working; Product Quality Complaint Rate (PQCR), which gives insights into end user feedback; and Invalidated Out-of-Specification (OOS) Rate (IOOSR), which helps provide insights into lab quality.
• Technically, the definitions of LAR and PQCR set forth in the 2016 draft could result in impossible figures, such as rates exceeding 100%, or a calculation that would require using zero for the denominator. The definitions didn’t account for real-time variabilities in pharmaceutical operations where processes might span across more than one reporting quarter, for example.
• LAR and IOOSR are tracked by organizations, but they are “not discerning metrics” and can lead quality teams on a wild goose chase “by highlighting nonexistent reporting issues” in some circumstances, acknowledged the FDA. “Other metrics should be identified as surrogates for manufacturing process performance and laboratory robustness.”
• A quality system’s effectiveness is “a critical component” of Quality Metrics reporting. However, an organization’s Quality Culture is also vital to robust performance and reliability. Though some numerical key performance indicators (KPIs) may help with ascertaining the culture of quality, “It is difficult to capture quality culture at an establishment based on numerical metrics alone,” observed the FDA.

Now, the FDA has put out a new call for comments from industry in a bid to launch a new Quality Metrics approach.

• This time, the FDA has proposed a completely revamped way of looking at performance metrics that is, at first blush, a much more complicated system. But the agency, in receiving feedback on its first two rounds of quality metrics guidance, appears to have constructed a system that allows manufacturers to take an a la carte approach to metrics reporting, selecting the metrics that best suit their industry and needs.
• What will stay the same? “The entities responsible for collecting and submitting quality metrics data,” according to the FDA’s roadmap for the new Quality Metrics proposal. In other words, all the pharmaceutical companies, biopharmaceutical companies and compounders originally covered by the 2016 proposal will still be covered.
• What will change? Almost everything else. With a new emphasis on flexibility gleaned from the pilots (and, perhaps, industry’s panning the two prior draft guidance documents), the FDA has chosen four general “practice areas” for the Quality Management reporting program.
• The first area, manufacturing process performance, will be evaluated through process capability/performance indices (Cpk/Ppk), which compare process outputs to the specification limits for that process. This figure is calculated and reported as a proportion. Lot Acceptance Rate (LAR), with a revamped definition, will be another metric in this area, as well the right-first-time-rate, which measured the proportion of lots that are manufactured without non-conformance. Finally, lot release cycle time will be tracked for the manufacturing process area practice area.
• Pharmaceutical quality system (PQS) effectiveness, the second practice area, will be tracked through capturing CAPA effectiveness, as measured by such inputs as how many CAPAs are initiated, closed on time, and deemed “effective, among others. Another metric for this practice area is the effectiveness of change controls; repeat deviation rate and overall equipment effectiveness, as well as unplanned maintenance round out the proposed PQS metrics.
• Laboratory performance, the third area,can be tracked by testing right-first-time-rate, timeliness of test equipment calibrations, Invalidated out-of-specification rate (IOOSR), and adherence to lead time for testing.
• The last practice area the FDA is proposing is “supply chain robustness.” Here, on-time-in-full and on-time disposition are important metrics, as is fill rate, or the proportion of the total demand that is filled by shipped orders. The agency is also seeking the metric of days of inventory on-hand.
• It’s worth noting that this is only a “potential direction” for the quality metrics program, and nothing is yet set in stone. That’s likely a reflection of the prior difficulties in getting the program set up in the first place.

The FDA has some specific questions it wants industry to address about this new proposed scheme.

• First, it wants to know about reporting levels, asking whether reporting at the establishment level is appropriate, and whether doing so would make it easy for contract manufacturers to submit their quality metrics data. The agency also wants to know how “product family” is defined within the context of an establishment where more than one “family” is produced.
• Next, the agency has questions about “practice areas” as they relate to quality metrics. The lengthy list of queries includes whether the FDA has gotten it right in setting forth the general practice areas outlined in the proposed reporting metrics, and whether other areas should be considered. This section of questions also asks whether the proposed metrics are appropriate for each designated practice area, as well as how much experience an establishment has with implementing both performance metrics and those related to process capability.
• In this area, the FDA also asks how to know when change is needed in terms of quality metrics, asking which criteria “should be applied to justify changing or modifying a quality metric,” either on the agency side or by industry. And, the FDA asks how frequently these changes would be likely to occur.
• A key question relates to Quality Culture. “If FDA were to consider Quality Culture as one of the general practice areas, what are the critical components of a robust quality culture and can any of these components be measured quantitatively?” asks the agency. The question suggests some potential metrics such as near misses and binary responses to a Quality Culture Survey, among others. The agency’s probing reflects a fundamental uncertainty: Can a company’s culture, and its approach to quality, be captured in a fully quantitative fashion?
• In some cases, metrics may be considered in tandem “because one metric influences the other,” observed the agency, in asking when multiple metrics might be appropriate. “What combination of metrics have been meaningful and useful?” asks the FDA.
• A catch-all “Other Considerations” category of questions asks whether some specific drug products, like generics or OTCs, might have unique considerations that should be reflected in the reporting program. The FDA also asks about the optimal reporting frequency.
• The agency is also soliciting input about what to do when a submission contains U.S. and foreign data and “a manufacturer is not able to extract domestic data.” The goal would be for the submission to be “informative” for the FDA, even if U.S. data can’t be pulled out and separated.

Potential pitfalls and possible implications of the FDA’s proposed restructure of quality metrics reporting

• FDA’s proposal was expected. AgencyIQ has made extensive predictions (see this September 2020 Twitter thread, and Kopcha’s response that highlights many of the points included in today’s proposal) about the likelihood that a quality metrics proposal might return as the result of the pandemic – or just the FDA’s ongoing efforts following the challenging rollout of its 2016 voluntary pilot program. As a result, much of what’s included in this call for comments isn’t a surprise, even if the specific details will still require time for companies to digest and analyze.
• Will an expanded suite of data fields yield sufficient data, and data of sufficient quality, for FDA’s intended analyses? The FDA says that it still “intends to analyze the information with statistical and machine learning methods to provide useful insights for inspection resource allocation,”but setting up these analyses will be more complicated, and the datasets will be smaller.
• What are the costs of collecting this data? Among the concerns raised by industry in response to the 2016 proposal was that the costs of setting up systems to track this data in real time would be considerable, and that not all facilities (and in particular older ones maintained by generic drug companies) may not be able to accommodate the added cost of technological complexities of collecting and reporting this data. While it’s possible the data environment at many companies has improved since the FDA first proposed to collect quality metrics data in 2015, we expect cost considerations to be among the things industry comments on.
• Metrics reported under this new proposal may generate a more heterogeneous data set, complicating analysis; on the other hand, data that better fit particular practice areas may be more useful both for the manufacturer, and for the regulation of each practice area. It’s also possible that collecting more granular metrics could yield new insight into early signals for supply chain disruption; small upstream perturbations may wind up being linked with bigger downstream troubles.
• Will these metrics really measure quality, though? It may be a big lift to control for variability in how each firm, and even each facility, captures data, or even assigns thresholds for certain CAPA actions. Further, if other entities ultimately make use of these quality metrics, as FDA has previously proposed, that might encourage companies to adopt the metrics most favorable to their facilities and “game” the metrics reporting to make their facilities look better than they are.
• However, COVID-19 has shown us that lots of quality data can be garnered remotely. The FDA has used its authority under 21 USC 704 for remote inspections during the public health emergency of the pandemic. If quality metrics reporting winds up substituting for some evaluation that has traditionally been conducted in-person, both the agency and industry could stand to benefit. And, a system grounded more in metrics might alleviate some perceived inequities between domestic firms, which are inspected unannounced, and foreign firms, which often have ample lead time – and, sometimes, less frequent inspections.
• Whether a firm’s culture around quality issues will ever be captured adequately through metrics alone remains an open question, although the FDA proposes some quantitative indicators. Though there are well-established markers for mature quality cultures, it’s hard to imagine that FDA inspectors do not factor in a “gut check” on a firm’s quality culture when their boots hit the plant floor.
• Taking the discussion to a higher level, could drug purchasers make use of a metrics-driven quality rating system in making their decisions? The FDA’s Principal Deputy Commissioner Janet Woodcock, floated that idea in a 2019 FDA Voices piece penned when she was the director of CDER. While reassuring consumers of the quality of FDA-approved products, Woodcock added a caveat: “But if purchasers also had the option to pay for reliable production, they might be willing to pay more for a medicine from a manufacturer with a proven track record of prompt and regular delivery of product, in order to avoid shortages.” The challenge, though, is that if the FDA doesn’t make use of the same metrics for all program participants, then purchasers may find it difficult to evaluate which metrics they should care most about. AgencyIQ will be interested to see which non-pharmaceutical industry stakeholders, including hospital groups and purchasers, might weight in on the proposal with respect to these points.

To contact the author of this item, please email Kari Oakes.
To contact the editor of this item, please email Alexander Gaffney.

Key Documents and Dates

• Food and Drug Administration Quality Metrics Reporting Program; Establishment of a Public Docket; Request for Comments