Detection of steady state in discrete event dynamic systems : an analysis of heuristics

The choice of initial conditions is a fundamental issue in the experimental analysis of non-terminating, stochastic, discrete-event dynamic systems (DEDS). Because output observations are sequentially correlated, cumulative performance estimators are biased when initial observations are not representative of steady-state operation. A poor choice of initial conditions, at best, requires a larger sample of steady-state operations so as to dilute the initialization bias. At worst, the bias goes undetected, and the result is that insufficient data are collected to ensure accurate statistical performance estimates.

Initialization bias is most often associated with discrete-event simulation, in which initial conditions are generally selected to convenience the analyst. In the simulation literature, this issue is called the "initial transient" or "start-up" problem. While various solutions to this problem have been proposed, by far the most common are those based on truncation of the output sequence. An observation in the sequence is identified as the first point representing steady-state operation. All prior observations are then deleted and the truncated sequence is retained for output analysis.

This research contributes a new and superior method for quantifying the performance of steady-state detection heuistics used to determine truncation points. Prior studies have concluded that existing heuristics are either ineffective, because the implied number of observations truncated is too few, or inefficient, because the implied number of observations truncated is too great. These prior studies are reviewed and several methodological deficiencies are identified. The method developed and applied here corrects these deficiencies by posing a more appropriate set of evaluation criteria and a more representative set of benchmark models.

This research also contributes a new and superior steady-state detection heuristic. The Confidence Maximization Rule (CMR) compares sample confidence intervals across candidate truncation points and identifies the onset of steady state as the point that minimizes the halfwidth of the truncated sample. The CMR was suggested by and tested using the method previously developed.

The CMR was evaluated against two existing heuristics that do not require pilot runs. The implied truncation point was computed using each heuristic for each of ten fixed-length runs of five different DEDS benchmark models. In most cases, the performance of the CMR was superior in terms of coverage and consistency. In all cases, the performance of the CMR was at least as good as the next best alternative. For the benchmark problems with significant initialization bias, the CMR also was shown to yield estimates comparable to those derived from untruncated samples with six times the number of observations.