You are on page 1of 26

Data Reconciliation
Data reconciliation technique allows the
adjustment of the measurements so that
the corrected measurements are
consistent with the corresponding
balances. The information from the
reconciled data can be used for different
purposes such as:
• Monitoring
• Optimization
• Simulation
• Instrument maintenance
• Management
• Modelling
• Equipment Analysis
This is especially true with the
implementation of a DCS.

Conservation of Mass Balance 2. Conservation of Energy Balance . The process models used in the reconciliation are 1.Data Reconciliation Data reconciliation is the estimation of process variables based on information contained in the process measurements and models.

Data Reconciliation Data reconciliation is the estimation of process variables Data Reconciliation in steady state systems Extensions for nonlinear systems and for dynamics Observability and redundancy in process data estimation Online data reconciliation for process control Gross error detection / fault diagnosis .

• Gross Errors: Can caused by measurement device failure & nonrandom events affecting process. x is the true value of the process variable. and e is the measurement error. It is assumed that any observation is composed of a true value plus some error value. It follows a Gaussian distribution. since measurements are obtained with imperfect instruments which have their own accuracy.Basic Concepts in DVR • • • Other issues that must be addressed in practical implementations also include measurement filtering before reconciliation to eliminate high frequency noise and estimating the typical variability (variance) of sensors. such as process leak. This indicates that a measurement can be modeled as: y=x+e where y is the observed value of the raw measurement. . Process Measurement Measured process data inevitably contain some inaccurate information. Errors can be divided into 2 types: • Random Error: Random error is caused by one or more factors that randomly affect measurement of a variable.

• zˆ is an N×1 vector of estimates for unmeasured process variables.zˆ ) = (y − yˆ )TV−1(y − yˆ ) subject to f (yˆ. At process steady state. • yˆ is an M×1 vector of estimates (reconciled values) for the M process variables. the optimal estimates for process variables by DR are solutions to a constrained least-squares or maximum likelihood objective function.zˆ ) ≥ 0 • y is an M×1 vector of raw measurements for M process variables. • f is a C×1 vector describing the functional form of model equality constraints. z. the reconciled data are obtained by: Minimizing J(yˆ. • V is an M ×M covariance matrix of the measurements.zˆ ) = 0 and g (yˆ. where the measurement errors are minimized with process model constraints • With the assumption of normally distributed measurements. . a least-squares objective function is conventionally formulated for the data reconciliation problem.Basic Concepts in DVR • In general. • g is a D×1 vector describing the functional form of model inequality constraints which include simple upper and lower bounds.

• • • • Process Variable Classification A redundant variable is a measured variable that can be estimated by other measured variables via process models. A nonredundant variable is a measured variable that cannot be estimated other than by its own measurement. . in addition to its measurement. A nonobservable variable is a variable for which no information is available. An observable variable is an unmeasured variable that can be estimated from measured variables through physical models.

Applying the general data reconciliation algorithm formulated by above equation (1. For this example. the vector of the raw flow measurements can be written as: The mass balances around each node can be written as: The mass balances around each node can be written as: .Linear Steady-State Data Reconciliation with All Variables Measured The simplest data reconciliation problem occurs in reconciling process flow rates in a plant as illustrated below.3). all the flows are measured in the network.

where each row represents each node and each column represents each flow stream.Linear Steady-State Data Reconciliation with All Variables Measured • The vector of the raw flow measurements can be written as: • The Variance matrix is • The matrix A is called the incidence matrix. • Ayˆ = 0 . respectively.

The reconciled flow rates are obtained by Minimizing J(yˆ ) = (y − yˆ )TV−1(y − yˆ ) − 2λ TAy After rearranging below is the basic solution for a linear steady-state data reconciliation problem. yˆ = y − VAT (AVAT )−1Ay • Results: The calculation results of the reconciled values for each measurement are listed in below table. It shows that the reconciled values satisfy mass balances. .Linear Steady-State Data Reconciliation with All Variables Measured Thus the data reconciliation problem for the network becomes: Minimizing J(yˆ ) = (y − yˆ )TV−1(y − yˆ ) subject to Ayˆ = 0 The optimization problem can be solved using Lagrange multipliers.

8) into (2.9) gives: E(yˆ ) = E(x + ε) −E[VAT (AVAT )−1A(x + ε)] Expanding the above eq & Since E(ε) = 0.7) gives: yˆ = (x + ε) − VAT (AVAT ) −1A(x + ε) Taking the expected value of (2. x. should be equal to the true values of process variables.11) shows that the reconciled values are unbiased estimates for the linear steady-state reconciliation problem. This is to say that the expected values of the reconciled data yˆ . VAT(AVAT)-1Ax = 0. and x is a deterministic variable. . E(x) = x. thus E(yˆ ) = x − VAT (AVAT )−1Ax And since Ax = 0 (the true values of the flows satisfy mass balances).Statistical Properties of Reconciled Values It is very important that the reconciled values be unbiased. The raw measurements can be written as the additive noise model: y = x + ε Putting (2. Therefore: E(yˆ ) = x Equation (2.

Rewrite Equation (2.7) as: yˆ = [I − VAT (AVAT )−1A]y yˆ = Wy From above the covariance matrix of the reconciled data can be given as: Cov(yˆ ) = WCov(y)WT = WVWT The standard deviation of the reconciled flows along with the standard deviation of the raw measurements are listed below.Statistical Properties of Reconciled Values The covariance matrix of the reconciled data can also be obtained. . It is clear that the reconciled flows have smaller standard deviations and are therefore more precise.

In this case. we need to develop a DR technique to reconcile the measurements and to estimate unmeasured flow rates as well. not all flows are measured in a plant due to physical or economical reasons. The problem of data reconciliation with both measured and unmeasured flows can be efficiently solved by the method of Projection Matrix as described in the following slides. we can partition the incidence matrix of the mass balances in terms of measured and unmeasured flows: A yˆ + A zˆ = 0 Ay correspond to the measured flows Az correspond to the unmeasured flows .Linear Steady-State Data Reconciliation with Both Measured and Unmeasured Variables In practice. First of all.

Linear Steady-State Data Reconciliation with Both Measured and Unmeasured Variables yˆ is the vector of reconciled values for measured flows zˆ is the vector of estimates for unmeasured flows Now we can rewrite the data reconciliation problem as Minimizing J(yˆ.zˆ ) = (y − yˆ )TV−1(y − yˆ ) subject to A yˆ + A zˆ = 0 .

P. can be performed efficiently using Q-R factorization of matrix Az.2) can be solved by first eliminating the unmeasured flows.Linear Steady-State Data Reconciliation with Both Measured and Unmeasured Variables The solution to the data reconciliation problem (3. P. in the constraint equations by premultiplying both sides by a projection matrix . P = Q2T = [0 0 1 0] . such that PAz = 0. . Then the data reconciliation problem becomes: Minimizing J(yˆ ) = (y − yˆ )TV−1(y − yˆ ) subject to PAyyˆ = 0 Then the final form of the equation becomes The construction of the projection matrix.

Now. This means it can only be evaluated by its measurement. F3 and F5. The measurement F1 is nonredundant. Rewrite the problem as: yˆ = y − VAT (AVAT )−1Ay The reconciled values for F3 and F5 are: . This indicates that the measurement F1 will disappear in the mass balance of the constraint equations. the data reconciliation becomes to reconcile the two redundant measurements.Linear Steady-State Data Reconciliation with Both Measured and Unmeasured Variables Note that the first element in the matrix PAy is zero.

using the information provided by the measured flows and the process mode A zˆ = −A y yˆ The quantities on the right side of eq are known. zˆ. so now the problem is to solve the linear equations on the left side. the number of equations is greater than the number of unmeasured flows. yˆ.Linear Steady-State Data Reconciliation with Both Measured and Unmeasured Variables Solutions to Estimates of Unmeasured Variables: After we obtain the reconciled values (estimates) of the measured flows. Usually. the next step is to estimate the unmeasured flows. The least-squares technique can then be applied and give the solution of the observable unmeasured flows as: .

A. suppose only flows F1 and F6 are measured and the other flows are unmeasured. the cooling water network here is cited again. For any process network. For simplicity. which is the incidence matrix. the analysis of the observability and redundancy of flow variables can be performed by analyzing the system matrix. while unmeasured variables are either observable or nonobservable.Observabilty and Redundancy Analysis Measured variables are either redundant or nonredundant. In this example. .

zˆ ) = (y − yˆ )TV−1(y − yˆ ) subject to A yˆ + A zˆ = 0 . we write the data reconciliation problem as: Minimizing J(yˆ.Observabilty and Redundancy Analysis For convenience.

Observabilty and Redundancy Analysis .

.Data Reconciliation in steady state systems • The simplest data reconciliation problem deals with steady state mass balances. assuming all variables are measured. and results in a linear problem.

provided that they are observable There are 3 techniques for data reconciliation: 1. Estimate model parameters 3. Reconcile raw measurements 2.Data reconciliation The algorithm of Data reconciliation indicates that the reconciliation techniques not only reconcile the raw measurements. Estimate unmeasured varaibles . but also estimate unmeasured process variables or model parameters.

2. Unmeasured variables are classified as observable and non-observable • A redundant variable is a measured variable that can be estimated by other measured variables via process models.Data Reconciliation Process Variables Classification: It is also important to clarify some concepts in data reconciliation techniques. . 1. • An observable variable is an unmeasured variable that can be estimated from measured variables through physical models • A non-observable variable is a variable from which no information is available. Measured variables are classified as redundant and non-redundant. • A non-redundant variable is a measured variable that cannot be estimated other than by its own measurement. in addition to its measurement.

Instrumentation performance o Measurement bias. drifting.Gross Error Detection Two major types of gross errors 1. Constraint Model-related o o Unaccounted loss of material and energy resulting from leaks process equipment Model inaccuracies due to inaccurate parameters . total instrument failure. miscalibration . 2.

KPIs etc.Applications • Data reconciliation has become a standard method in a number of areas that have direct influence on economy and process safety: – Efficiency monitoring (specific consumption of energy.) – Integration of industrial measurements into plant accounting – Instrumentation maintenance – Advanced Process Control – Virtual/Electronic flow metering – Back tracking .

Facilitate regulatory compliance by creating a single set of traceable data. Improve productivity by reducing manual identification and correction of inefficient production processes. . Reduce production losses through automated monitoring that immediately identifies faulty processes and equipment.Benefits • • • • • • Improve data accuracy by reconciling and validating process information before it reaches an ERP system. reliable. Speed the delivery of reconciled information that is consistent. Gross error detection takes place through a consistency check before the data reconciliation engine fires. and accurate to business systems.