Trustworthy Autonomous Pharmaceutical Manufacturing Systems: Human Oversight, Model Drift, and Quality Accountability

Pharmaceutical manufacturing is moving from automated equipment and digitally assisted control toward more autonomous systems capable of interpreting process data, adjusting operating conditions, and supporting quality decisions. This trajectory promises faster response, improved consistency, and more adaptive control across complex production environments. Yet autonomy also changes the nature of manufacturing responsibility because technical decisions increasingly occur inside algorithmic systems rather than through visible human judgement alone. The central problem addressed in this article is the trust deficit created by autonomous pharmaceutical manufacturing. When an algorithm modifies a critical process parameter, detects an anomaly, recommends batch continuation, or contributes to a quality disposition, regulators, operators, quality units, and patients require confidence that the decision remains safe, explainable, reversible, and accountable. Trust cannot be assumed simply because the system performs well during validation; it must be sustained over time as processes, materials, sensors, models, and organisational practices evolve. This article develops an original theory-driven framework for trustworthy autonomous pharmaceutical manufacturing. The framework is structured around three interdependent pillars: human oversight, model drift management, and quality accountability. These pillars are treated not as separate compliance add-ons but as mutually reinforcing design requirements for autonomous manufacturing systems operating in Good Manufacturing Practice environments. The article draws on a theoretical synthesis of peer-reviewed literature on pharmaceutical manufacturing automation, process analytical technology, machine learning, trust in automation, human–autonomy teaming, resilience engineering, socio-technical systems, and model drift. It reframes autonomous manufacturing as a socio-technical trust problem rather than a purely technical optimisation problem. Four tables map the theoretical foundations, oversight architectures, drift-management logic, and integrated Trustworthy System Framework. The proposed framework argues that trustworthiness in autonomous pharmaceutical manufacturing is not a property of an algorithm alone. It emerges from the designed relationship among people, models, process controls, quality systems, audit trails, and governance responsibilities. Autonomous manufacturing will become viable only when the system can remain technically reliable, humanly overseen, and institutionally accountable throughout its lifecycle.

Pharmaceutical manufacturing is entering a phase in which automation is no longer limited to isolated equipment control or conventional process monitoring. Industry 4.0 visions for pharmaceutical production increasingly emphasise connected data infrastructures, continuous manufacturing, process analytical technology, and intelligent control systems that can support real-time process understanding and adaptive decision-making [1]. These developments are visible in continuous manufacturing research, where process models, soft sensors, and control strategies are being designed to link material attributes, unit operations, and quality outcomes more tightly than in traditional batch paradigms [2, 3].

This movement toward autonomy is also being accelerated by machine learning applications in formulation development, process monitoring, quality prediction, and manufacturing decision support. Artificial neural networks and other data-driven methods are being positioned as tools for extracting patterns from complex pharmaceutical process data, while model predictive control and data-driven control architectures are being explored for continuous production environments [4, 5]. The resulting trajectory points toward manufacturing systems in which algorithms may not only monitor process states but also recommend, initiate, or execute corrective actions within predefined control spaces [6].

The trust challenge emerges because pharmaceutical manufacturing decisions have direct implications for product quality and patient safety. If an autonomous system changes a blending endpoint, adjusts a drying trajectory, modifies a feed rate, or supports a release-relevant decision, the manufacturer must still be able to justify how that decision was made and why it remained within an acceptable quality and risk framework [7]. Regulatory and quality expectations therefore make pharmaceutical autonomy fundamentally different from domains where algorithmic optimisation may proceed with less explicit accountability for each operational decision [8].

The aim of this article is to construct a theory-driven framework that makes autonomous pharmaceutical manufacturing trustworthy by design. The framework treats trustworthiness as a system property produced through three pillars: human oversight as the human assurance layer, model drift management as the technical integrity layer, and quality accountability as the governance layer. By integrating pharmaceutical manufacturing literature with trust theory, human factors, resilience engineering, and socio-technical systems thinking, the article proposes a conceptual architecture for autonomy that can remain reliable, governable, and credible over time [9, 10].

Trust theory in automation shows that trust is neither simple acceptance nor blind reliance. Trust must be calibrated so that human users rely on automated systems when they are competent, question them when uncertainty is high, and intervene when their behaviour becomes unsafe or inappropriate [11]. In autonomous pharmaceutical manufacturing, this means that trust must be grounded in evidence of system competence, predictability, transparency, and recoverability rather than in the mere presence of advanced digital technology [12].

Human–autonomy teaming extends this logic by emphasising that autonomous systems and human actors form joint decision systems rather than separate layers of action. In manufacturing, operators, engineers, quality personnel, and algorithmic control systems must share a coherent understanding of process state, system confidence, intervention boundaries, and escalation triggers [13]. Meaningful human control becomes particularly important where autonomous action may affect critical quality attributes, because humans must retain the ability to understand, contest, override, or approve consequential system decisions [14].

Resilience engineering adds a further theoretical dimension by shifting attention from static compliance toward the capacity of systems to monitor, respond, learn, and adapt under changing conditions. Pharmaceutical manufacturing systems that use AI-driven control must remain resilient when sensors degrade, materials vary, process dynamics shift, or model assumptions become outdated [15]. A resilience perspective therefore supports the argument that trustworthy autonomy requires continuous verification and adaptive governance rather than one-time validation alone [16].

Socio-technical systems theory provides the integrating logic for this article because autonomous manufacturing cannot be understood as a purely technical artefact. The performance of an autonomous pharmaceutical line depends on the joint optimisation of algorithms, equipment, data pipelines, human roles, quality procedures, and organisational accountability structures [17]. Table 1 maps key theoretical constructs to the pharmaceutical autonomous manufacturing context.

Autonomous pharmaceutical manufacturing is often framed as a capability problem: whether equipment, sensors, models, and control algorithms can achieve sufficient accuracy and responsiveness. This framing is incomplete because the decisive question is not only whether an autonomous system can optimise production, but whether its decisions can be trusted in a regulated quality environment [18]. A system may be technically sophisticated and still be untrustworthy if its decision logic is opaque, its failure modes are poorly monitored, or its accountability chain is ambiguous [19].

The pharmaceutical setting places unusually high demands on trust because manufacturing failures can compromise identity, strength, purity, safety, efficacy, or supply continuity. Continuous manufacturing and process analytical technology can increase process visibility, but they also create dense streams of data that require disciplined interpretation and validated control strategies [20]. As systems become more autonomous, the central risk shifts from whether humans can manually detect every deviation to whether the combined human–algorithm system can detect, explain, and correct emerging quality threats [21].

This article therefore reframes autonomous pharmaceutical manufacturing as a trustworthiness problem. Trustworthiness is not equivalent to automation, digitalisation, or artificial intelligence adoption; it refers to the demonstrable capacity of the system to remain safe, reliable, understandable, controllable, and accountable under expected and changing conditions [22]. In this view, a lights-out or minimally staffed facility would not be trustworthy merely because it reduces human intervention; it would be trustworthy only if it preserves adequate human assurance, technical integrity, and quality governance [23].

The proposed reframing decomposes trustworthiness into three pillars that correspond to distinct but interdependent system functions. Human oversight provides the human assurance layer by ensuring that autonomous decisions remain visible, interpretable, and subject to escalation or override. Model drift management provides the technical integrity layer by detecting when algorithmic performance may degrade as data, equipment, materials, or processes change [24]. Quality accountability provides the governance layer by assigning responsibility, preserving auditability, and ensuring that autonomous decisions remain embedded within the pharmaceutical quality system [25].

Figure 1 reframes autonomous pharmaceutical manufacturing as a trustworthiness problem structured around human oversight, model drift management, and quality accountability.

The first assumption is that full autonomy without human oversight is unacceptable for quality-critical decisions under the current pharmaceutical regulatory paradigm. Even when advanced analytics or machine learning systems perform reliably, human actors must retain meaningful authority over decisions that may affect batch disposition, critical process parameters, or quality-risk escalation [14]. This does not require constant manual control, but it does require designed oversight architectures that specify when humans are informed, when they approve, and when they must intervene [26].

The second assumption is that AI models used in manufacturing will inevitably drift and must be managed as lifecycle objects rather than treated as static validated tools. Concept drift research shows that model performance can deteriorate when the statistical relationship between inputs and outputs changes, and manufacturing environments are particularly exposed to such change through raw-material variation, equipment wear, sensor ageing, environmental shifts, and process learning [24, 27]. Trustworthy autonomy therefore requires continuous model monitoring, predefined drift thresholds, controlled retraining, and validated fallback strategies [28].

The third assumption is that accountability rests with the manufacturer and the pharmaceutical quality system, not with the algorithm. Autonomous systems may generate recommendations or execute actions, but legal, ethical, and quality responsibility must remain traceable to accountable organisational roles, approved procedures, validated model versions, and auditable decision records [7]. The framework is therefore delimited to manufacturing and quality-control contexts, including process monitoring, control, deviation response, and quality decision support, while excluding early discovery, molecular design, and non-GMP research activities [8].

Human oversight is the first pillar of trustworthy autonomous pharmaceutical manufacturing because it preserves human authority over decisions that may affect product quality, patient safety, or regulatory confidence. In a mature autonomous line, the human role should not disappear but should shift from routine manipulation to supervision, interpretation, escalation, and quality assurance. Human–autonomy teaming research indicates that such systems require explicit role allocation, shared situational awareness, and clear boundaries between machine recommendation, machine execution, and human approval [13]. For pharmaceutical manufacturing, this means that operators, automation engineers, and quality personnel must understand not only what the autonomous system is doing but also why it is acting and when its confidence is insufficient [29].

The oversight architecture should be dynamic rather than fixed. Low-risk, well-characterised adjustments within a validated design space may be suitable for human-on-the-loop supervision, whereas novel disturbances, low-confidence model outputs, unexplained deviations, or quality-critical interventions should trigger human-in-the-loop approval [26]. This risk-adaptive approach is consistent with meaningful human control because it does not require humans to manually approve every routine action, but it does require them to retain informed authority over consequential decisions [14]. Table 2 defines the levels and types of human oversight in autonomous pharmaceutical manufacturing.

The main design problem is that oversight can become symbolic if the human is technically present but cognitively disengaged. Trust measurement research shows that overtrust and undertrust both impair performance because users may either accept unreliable automation or reject useful support [11, 12]. In pharmaceutical manufacturing, automation complacency may be especially dangerous because operators may become accustomed to stable autonomous performance and fail to detect subtle degradation, while excessive distrust may lead to unnecessary manual interventions that destabilise a controlled process. Oversight design must therefore include training, explainable interfaces, uncertainty displays, alarm prioritisation, and periodic challenge scenarios that keep human expertise active [30].

Human oversight also has an organisational dimension because no single operator can carry the full burden of autonomous system assurance. Quality units must define which decisions require approval, engineers must define validated control boundaries, and manufacturing supervisors must ensure that daily operations remain aligned with approved procedures [1]. A trustworthy oversight system should therefore distribute responsibility across operational, technical, and quality roles while maintaining a clear escalation chain. In this sense, oversight is not a human substitute for technical reliability but a socio-technical assurance function that keeps autonomous manufacturing aligned with GMP expectations [17].

Model drift is the second pillar because autonomous manufacturing systems depend on models whose validity can weaken as production conditions change. In pharmaceutical manufacturing, drift may arise from raw-material variability, sensor fouling, equipment ageing, environmental shifts, changes in upstream processing, maintenance interventions, or gradual process learning. Concept drift literature shows that predictive performance can deteriorate when the relationship between input data and target outcomes changes, even if the model once performed well during development or validation [24]. This is especially important for AI-supported continuous manufacturing because control decisions may depend on real-time predictions generated from evolving process data [4].

Drift in pharmaceutical manufacturing can be understood through several overlapping categories. Covariate shift may occur when input distributions change, such as when excipient lots, particle-size distributions, moisture content, or feed properties differ from historical training data. Concept drift may occur when the relationship between process variables and quality attributes changes because of process ageing, equipment wear, or altered operating regimes [27]. Sensor degradation is a related but distinct threat because the data stream may appear stable while the measurement system gradually loses accuracy, creating false confidence in model predictions [19].

Detection methods must therefore combine statistical process monitoring, model-performance analysis, sensor-health checks, and out-of-distribution detection. Soft-sensor research and PAT-enabled continuous manufacturing demonstrate the value of linking process signals to quality predictions, but these tools require ongoing verification that their assumptions remain valid [22, 23]. Drift detectors can support early warning, yet they must be adapted to GMP expectations because false alarms, missed alarms, and uncontrolled model changes can all create quality risk [28]. Table 3 categorises model drift types, detection methods, and mitigation strategies.

Figure 2 illustrates how model drift management operates as a continuous lifecycle control loop within trustworthy autonomous pharmaceutical manufacturing.

Mitigation must be controlled because an autonomous system that can freely retrain itself may create unapproved changes to the manufacturing control strategy. Scheduled recalibration, predefined retraining protocols, locked model versions, rollback procedures, and independent quality review are therefore essential elements of drift governance [7]. When drift is detected, the system should not automatically assume that adaptation is safe; instead, it should evaluate whether continued operation is acceptable, whether human review is required, and whether fallback to proven acceptable ranges is necessary [3]. Drift management is thus not only a data-science activity but a pharmaceutical quality function embedded in validation, change control, and continued process verification [2].

Quality accountability is the third pillar because autonomous decisions must remain traceable to responsible persons, approved procedures, and validated systems. In pharmaceutical manufacturing, accountability cannot be transferred to an algorithm, even when the algorithm recommends or executes a process action. The manufacturer must be able to reconstruct the data, model version, decision logic, control context, and human approvals associated with each quality-relevant action [7]. This requires audit trails designed for algorithmic decision-making rather than only for conventional equipment events or manual entries [8].

A trustworthy autonomous system should assign ownership for each class of algorithmic decision before deployment. Process-control decisions may be owned operationally by manufacturing and automation teams, while quality-impacting decisions require defined quality-unit responsibility and documented review pathways [25]. Where AI systems support release, deviation triage, or critical parameter adjustment, the decision record should capture whether the system acted autonomously, recommended action, escalated uncertainty, or requested human approval. This makes accountability operational rather than rhetorical because each autonomous action becomes reviewable within the pharmaceutical quality system [10].

Risk governance must also expand to include algorithmic risk assessment. Traditional process risk tools focus on materials, equipment, process parameters, contamination, human error, and specification failures, but autonomous systems introduce additional hazards such as training-data bias, model opacity, drift, automation complacency, and unvalidated adaptation [18]. Reliability and systems-safety research shows that complex automated systems can fail through interactions among components, procedures, human expectations, and organisational assumptions rather than through a single component breakdown [31]. Pharmaceutical risk governance must therefore examine not only whether the model is accurate but also how the model can fail within the wider manufacturing system [16].

The quality system should integrate autonomous decisions into deviation management, corrective and preventive action, change control, periodic review, and management review. If drift is detected, a model produces unexplained recommendations, or an operator overrides an autonomous action, the event should be evaluated not only as a local anomaly but also as evidence about the system’s trustworthiness. Explainable AI and situational-awareness research suggests that decision support becomes more useful when it helps humans understand the system state, relevant evidence, uncertainty, and action consequences [29]. Accountability therefore requires both recordkeeping and interpretability, because a decision that cannot be meaningfully reconstructed cannot be responsibly governed [30].

The proposed Trustworthy System Framework integrates the three pillars into a single socio-technical architecture for autonomous pharmaceutical manufacturing. Human oversight provides the outer assurance loop by ensuring that autonomous decisions remain visible, interpretable, contestable, and reversible when risk increases [13]. Model drift management provides the technical integrity loop by continuously testing whether data streams, model assumptions, and predictive relationships remain within the validated applicability domain [24]. Quality accountability provides the governance loop by ensuring that decisions, interventions, model updates, and failures are assigned, recorded, reviewed, and improved through the pharmaceutical quality system [25].

The framework’s central claim is that trustworthiness emerges from interactions among these pillars rather than from any one pillar alone. Drift monitoring should trigger human review when uncertainty, novelty, or quality risk exceeds predefined thresholds, while human overrides and operator concerns should feed back into model-performance review and quality-risk assessment [28]. Accountability structures should ensure that both technical failures and oversight failures are captured through deviation, CAPA, and change-control processes rather than treated as isolated anomalies [31]. Table 4 presents the proposed Trustworthy System Framework integrating the three trustworthiness pillars.

Figure 3 presents the integrated Trustworthy System Framework showing how oversight, drift management, and accountability interact to produce calibrated trust, sustained reliability, and regulatory confidence.

The framework generates three emergent system-level properties. The first is calibrated trust, where humans neither blindly defer to the system nor unnecessarily resist it because they receive meaningful evidence about system capability and uncertainty [11]. The second is sustained reliability, where model performance is continuously protected against drift, sensor degradation, and process change through validated monitoring and recovery mechanisms [27]. The third is regulatory confidence, where autonomous manufacturing can be inspected, explained, and governed because decision authority, technical evidence, and quality responsibility remain traceable throughout the system lifecycle [8].

Implementation should proceed through phased autonomy rather than an abrupt transition from conventional automation to self-governing manufacturing. The first phase should use advisory AI systems that generate recommendations while humans retain approval authority, allowing organisations to evaluate model behaviour, operator trust, alarm quality, and audit-trail sufficiency before allowing autonomous execution [5]. The second phase can permit conditional autonomy within tightly validated operating boundaries, where the system may adjust parameters but must escalate uncertainty, novelty, or quality-impacting deviations [3]. The third phase may support broader autonomous operation only when accumulated evidence demonstrates reliable oversight, drift control, and accountability performance [1].

Validation strategies should explicitly test the oversight–drift–accountability triad. Simulation-based stress testing can expose the autonomous system to raw-material shifts, sensor faults, equipment disturbances, missing data, conflicting signals, and low-confidence predictions without endangering commercial product quality [23]. Human-factors validation should evaluate whether operators correctly interpret system confidence, detect abnormal behaviour, and use escalation pathways under realistic workload conditions [12]. Quality-system validation should confirm that every autonomous decision can be reconstructed with its data inputs, model version, control action, human interaction, and governance outcome [7].

Regulatory engagement and industry standardisation will be essential because many organisations will otherwise build incompatible approaches to model lifecycle management in GMP. Emerging digital manufacturing practices already point toward greater use of integrated data systems, predictive analytics, and AI-enabled process control, but trustworthy autonomy requires shared expectations for model updates, audit trails, fallback modes, human oversight, and algorithmic risk assessment [9]. Cross-sector learning from reliability engineering and high-hazard industries can help pharmaceutical manufacturers avoid treating autonomy as a software upgrade rather than a system transformation [15]. The practical pathway is therefore not merely to automate more functions but to generate progressively stronger evidence that autonomy remains humanly overseen, technically stable, and quality-accountable [32].

Trustworthy autonomous pharmaceutical manufacturing cannot be achieved by adding artificial intelligence to existing production systems and assuming that technical performance alone will create acceptance. As algorithms begin to support or execute control and quality decisions, the central challenge becomes the design of a trustworthy socio-technical system. Autonomy will advance sustainably only when it is paired with explicit mechanisms for oversight, drift management, and accountability.

The framework developed in this article decomposes trustworthiness into three mutually reinforcing pillars. Human oversight provides the assurance that consequential decisions remain interpretable, contestable, and subject to responsible intervention. Model drift management protects the technical integrity of autonomous systems as materials, sensors, equipment, processes, and data relationships change over time, while quality accountability ensures that autonomous decisions remain traceable within the pharmaceutical quality system.

The future of autonomous pharmaceutical manufacturing will depend on coordinated action by manufacturers, technology providers, regulators, quality professionals, automation engineers, and human-factors specialists. The next step is to translate trustworthiness from a broad aspiration into standards, validation practices, pilot programmes, and inspection-ready evidence models. Only then can autonomous manufacturing become not merely more efficient, but genuinely reliable, governable, and worthy of confidence.

None

None

None

None

Siti Rahman, Ahmad Zaki, Nurul Huda, Amir Faisal & Lim Wei contributed to this work.

Authors and affiliations

Department of Pharmaceutical Manufacturing Systems, Faculty of Pharmacy, University of Malaya, Kuala Lumpur, Malaysia
Siti Rahman, Ahmad Zaki & Amir Faisal

Department of Therapeutic Technologies and Engineering, Faculty of Health Sciences, Universiti Teknologi Malaysia, Johor Bahru, Malaysia
Nurul Huda & Lim Wei

Corresponding author

Correspondence to Siti Rahman

Open Access The author(s) retain copyright. This article is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. It may be shared and adapted for non-commercial purposes with appropriate attribution, an indication of changes, and distribution of adaptations under the same license. Third-party material may be subject to separate terms identified in its credit line. View the license at https://creativecommons.org/licenses/by-nc-sa/4.0/.