Informing Industry End-Users on the Credibility of Model Predictions for Design Decisions

Jakob Hartl, Purdue University

Abstract

Many industrial organizations invest heavily in modeling and simulation (M&S) to support the design process. The primary business motivation for M&S is as a cheaper and faster alternative for obtaining information towards a better understanding of system behavior or to help with decision making. However, M&S predictions are known to be inexact because models and simulations are mathematical approximations of reality. To ensure that models are applicable for their intended use, organizations must collect evidence that the M&S is credible. Verification, validation, and uncertainty quantification (VVUQ) are the established methods for collecting this evidence. Structured frameworks for building credibility in M&S through VVUQ methods exist in the scientific literature but these frameworks and methods are generally not well developed, nor well implemented in industrial environments. The core motivation of this work is to help make existing VVUQ frameworks more suitable for industry. As part of this objective, this work proposes a new credibility assessment that turns VVUQ results into an intuitive, numerical decision-making metric. This credibility assessment, called the Credibility Index, identifies the important aspects of credibility, extracts the relevant VVUQ results, and converts the results into an overall Credibility Index score (CRED). This CRED score is unique for each specific prediction scenario and serves as an easy-to-digest measure of credibility. The Credibility Index builds upon widely accepted definitions of credibility, well-established VVUQ frameworks, and decision theory. The Credibility Index has been applied to several prediction scenarios for two publicly available benchmark problems and one Rolls-Royce funded subsystem case; all examples relate to the aerodynamic design of turbine-engine compressors. The results from these studies show how the Credibility Index serves as a decision-making metric, supplements traditional M&S outputs, and guides VVUQ efforts. A product feedback study, involving model end-users in industry, compared the Credibility Index to three other established credibility assessments; the study provides evidence that CRED consistently captures all key aspects of information quality when informing end-users on the credibility of model predictions. Due to the industry partnership, this research already has multiple avenues of practical impact, including implementation of the structured VVUQ and credibility framework in an industrial toolkit and workflow.

Degree

Ph.D.

Advisors

Karl, Purdue University.

Subject Area

Epistemology|Physics|Aerospace engineering|Design|Fluid mechanics|Marketing|Mechanics

COinS