4 |
Systems Analysis |
|
4.2.1 Setting of objectives and preliminary synthesis |
|
|
4.2.2 Experimentation |
|
|
4.2.3 Management |
|
|
4.2.4 Evaluation |
|
|
4.2.5 Final synthesis |
|
In the previous chapter, it was emphasized that mathematical models should be incorporated into a broad framework of systems analysis. In that chapter, models were defined as formal expressions of the essential elements of a problem in either physical or mathematical terms. Similarly, systems analysis was defined as the orderly and logical organization of data and information into models, followed by the rigorous testing and exploration of these models necessary for their validation and improvement. It is on this orderly and logical organization, and the subsequent validation and testing of models, that we shall concentrate in this chapter. We will also review the role of system analysis in:
integrating research on complex problems;
providing a link between research and the application of research results .
The earliest detailed consideration of systems analysis and modelling in the Man and the Biosphere Programme of UNESCO was at a meeting of an Expert Panel on Systems Analysis and Modelling Approaches, which took place in Paris in April 1972 (UNESCO, 1972).
After considerable discussion by the experts on this Panel, it was concluded that integration and co-ordination of scientific activity are needed to bring about greater coherence of ecological research, and that systems analysis could help to bring about this coherence, and could increase the significance of the research for the practical management of resources.
Systems analysis was not formally defined by the Expert Panel, but the members' interpretation of systems analysis may be derived from the following considerations. First, there was a need to develop a predictive understanding of the functioning of the complex natural systems upon which man depends. When faced with complex and highly interactive systems, human judgement and intuition may lead to wrong decisions, sometimes with results that cannot be reversed. Numerous examples of these wrong decisions exist in the history of man's management of his natural resources, and, if anything, the number and the gravity of these errors have increased in the recent past. Mere increase of knowledge, without predictive understanding of the functioning of complex systems, is not sufficient for the management of such systems. There has been extensive study of the behaviour of complex interacting systems in such fields as engineering, physiology and economics. Resulting from this study has been the development of methods for understanding the dynamics of systems, and the impact of stresses upon them. Such methods can be adapted to systems, ecology, with the assumptions that the state of an ecosystem at any particular time can be expressed quantitatively, and that changes in the system can be described by mathematical expressions. Understanding of any system depends on translation of its variables and properties into a generalized form so that is becomes an abstract, or model. Such models are essentially simplifications of the system, but are nevertheless more comprehensive and more precise than the mental models of the field scientist or the resource manager.
Figure 2 shows how mathematical models of different types can help bridge the gap between ecological theory, management experience, experiments, and between strategic and tactical management prescriptions. The analysis incorporates data and information from a wide array of studies into a single interrelated system. Analytical procedures are used to select the combination of tactics required to give maximum output from a system consistent with a variety of constraints on management alternatives.
It is possible to recognize some common stages in the use of systems analysis for the solution of problems concerned with dynamic change in ecosystems.
Such studies necessarily begin with the establishment of objectives and the creation of an initial synthesis of existing information. The objectives should specify: (i) the range of subject matter; (ii) the types of manipulation, modification or disturbance to be included in the prediction; and (iii) the variables which it is intended to measure and to predict. The initial synthesis requires the assembly of relevant existing knowledge from publications, unpublished reports and field data, and discussions with experienced research and management personnel. Assumptions will necessarily have to be made as to the relevance of particular kinds of knowledge, particularly when such knowledge exists at the periphery of the principal problem. For this reason, it is frequently useful to re-examine the objectives after the preliminary synthesis has been completed, as the act of assembling data will have helped to clarify ideas in the minds of the research scientist. Whenever possible, the existing knowledge should be incorporated into predictive models as a basis for the design of the next phase of the investigation, which is concerned. with direct experimentation.
Figure 2. Interrelations of theory, experience and data with the tasks of statistical, simulation and optimization modelling towards improved recommendations for natural resource management. Solid lines represent direct flows of information, dashed lines represent feedbacks. The central (shaded) portion of the figure includes many of the kinds of models used in a systems analysis and operational research approach. The elements of the more conventional approach to research management decision-making lie outside the dotted portion.
Once the objectives of the project and the preliminary synthesis have been completed, it will usually be necessary to proceed to direct experiments in the field and in the laboratory, as well as on any preliminary models which may have been developed. The type of the experimentation depends principally on the level of sophistication of the project. Validation data should therefore be gathered to test the output of, or the functions in, the predictive models, so that these models are themselves helping to define the kinds of experiments which need to be performed.
Adjustment or revision of models is greatly accelerated by formulating procedures for the rapid transmission of information on the structure and operation of the models which have been developed. Similarly, experiments conducted on models leads to new ideas on the management of the system being examined. It is therefore necessary to ensure a three-way communication and feedback between research scientists, modellers and managers or planners. Ideally, this communication should be initiated earlier during the phases of synthesis, planning and setting of objectives, so that all three groups of personnel have an opportunity to participate in the development of the research from the beginning. To aid to this communication, it is also a useful procedure to develop simulation and optimization models of natural resource systems for use as models in management games.
The communication and dialogue between scientists, managers and planners during the examination of the output from models and experiments should result in the planning of pilot-scale management studies. After appropriate testing at the pilot scale, larger-scale management schemes can then usually be undertaken, again with consultation and feedback between the scientific groups developing the synthesis, experiments and models.
As the research proceeds, it is possible to evaluate the effects of the management changes proposed on the structure, function and stability of the system. In these terms, evaluation is a continuous process, starting with the setting of objectives and priorities, but, at an appropriate stage, it may be desirable to summarize the results of all the ideas developed in the earlier stages of the project. Such an evaluation may lead to a return to one of the earlier phases in order to correct for faulty assumptions or design failures in the model.
Ultimately, it is necessary that the information derived from the research should be summarized, evaluated and published. However, no model is ever likely to be regarded as 'final' so that the models which are derived from one research project are very likely to form the preliminary synthesis for further research.
With the development of computers and computer languages, new approaches and methodologies have been designed to handle complex biological systems, so that it is now possible to suggest new research and policy strategies for situations with a large number of interacting components. Nevertheless, the Expert Panel suggested that inadequate integration might be expected to occur in the systems analysis of ecological research at the following junctions:
between data experimentation and model development;
between simulation model operations, the overall systems analysis and the implementation of models in field testing;
between the examination of predictions from systems analysis and the implementation of management procedures;
between the testing of management techniques and the development of new hypotheses;
between the implementation of results from pilot studies and the development of new hypotheses.
The Expert Panel also presented the following conclusions.
Data quality and the understanding of causal pathways in ecology are generally unreal, and this lack of reality is likely to hinder systems analysis of natural systems.
Systems analysts and data collectors can often develop a mutually beneficial relationship from which a decision-maker himself eventually derives the maximum benefits.
Systems training is valuable for stressing a broad interdisciplinary, problem-orientated philosophy of research, and such a philosophy is urgently needed for the kinds of ecological research which is necessary in the MAB programme.
Systems models can be improved only by building them, and by striving to correct the weaknesses of the early versions of these models. An adequate model does not spring fully fashioned from an initial research project.
The scientists contributing to systems analysis must be broadly inter- disciplinary, and the contributions of these many disciplines are essential for the development of the models of dynamic change in ecosystems.
Systems models often demand a large quantity of high-quality data, and they can therefore be very expensive. Nevertheless, systems analysis provides a powerful tool for prediction and planning, a set of procedures for the formal application of logical processes, and a means of communicating with research scientists and managers.
Since the meeting of the Expert Panel in 1972, there has been considerable development of systems analysis and modelling within the scientific community. In particular, the studies of the International Institute for Applied Systems Analysis in Laxenburg, Vienna, have further helped .to define the role and broad functions of systems analysis. From this development, the role of systems analysis has come to be seen as an explicit formal enquiry, designed to assist in the making of decisions for the forming of policy. The analysis determines a preferred action or policy of identifying and examining the alternatives, and by a comparison of alternatives in terms of their consequences. Explicit formal enquiry usually involves the use of mathematical models, but such models are not strictly necessary.
From this definition, the broad functions of systems analysis may be defined. First, systems analysis provides an objective basis for assessing and assimilating available information about the system. Second, it directs research into areas for which, relative to the understanding of the whole system, present knowledge is uncertain. Third, it provides a means of assessing and applying the results of this research. Fourth, it assists in the management, control or development of the ecological system.
Figure 3 shows, in diagrammatic form, the seven steps which have been identified in the application of systems analysis to practical research. The process begins with the recognition of the existence of a problem, or of a constellation of interconnected problems. Such problems must be amenable to analysis and sufficiently important for detailed investigation, so that recognition is a critical step which may determine the success or failure of the subsequent research. The second step is the definition and bounding of the extent of the problem. This definition and bounding seeks a simplification of the problem in order to make it capable of analytical solution, while preserving all the elements which are of sufficient interest for practical research. Again, this is a critical stage in any analysis, and specialist experience is often valuable in helping to determine the relative importance of the inclusion or exclusion of elements of the problem. Such experience also facilitates the balancing of the relevance of elements of the analytical solution against complications which will make the solution unmanageable. The third step is the identification of the hierarchy of goals and objectives, in which the major objectives set in the earlier stages are progressively subdivided into a series of minor objectives. Each of these minor objectives will frequently represent a 'milestone' in the conduct of the investigation. For this reason, it is necessary to ensure that the determination of the priorities is relative to the amount of effort required to meet the objectives. There is little point in placing a great deal of emphasis and effort on an objective which is only of minor importance. Conversely, most of the effort in the investigation should be directed towards the major objectives.
Figure 3. The steps in system analysis
When these earlier stages have been completed, it is possible to proceed to the generation of the solutions to the problem. The aim is to generate a series of possible solutions, ideally selected as members of broad families of models, as described in the outline of modelling in Chapter 3. In generating solutions in this way, it is desirable to seek analytical models of the greatest possible generality. Such models will often make the best use of previous work on similar problems and, at the same time, will attempt to keep the underlying mathematics as simple as possible. It's seldom possible to predict the modelling strategy which is most likely to be successful in finding a solution to a particular practical problem, and as many models as possible should be developed in the early stages of the systems analysis. Only when some of the preliminary models have been tested fairly extensively will it be possible to decide on the best strategies for solving the problem.
Modelling of the complex, dynamic interrelationships of the various facets of a problem may be attempted for many of the alternative strategies. A full awareness of the inherent uncertainities in the various processes to be modelled, or of feedback mechanisms which may complicate both understanding and practicability of the model, can often be gained by practical experience. An experienced analyst may be able to contribute much to the shortening of this phase of systems analysis. There may, too, be a complex series of rules which will be used to reach a decision about appropriate courses of action. These rules must also be incorporated into the model.
Evaluation of potential courses of action for solving a practical problem will begin by investigating the sensitivity of the results of the modelling to assumptions made by the model, a process which is known to analysts as sensitivity analysis. In this analysis, previously unexpected weaknesses in the assumptions, and in the model formulation, may be revealed. Discovery of an important flaw in a major assumption may lead to a return to the modelling phase, but, sometimes, relatively simple modifications of the original model may be sufficient. Similarly, investigations of the sensitivity of the model to facets of the problem which were excluded from the formal analysis when the problem was defined and bounded will frequently play an important part in this evaluation. The problems of sensitivity analysis are discussed further below. Finally, however, systems analysis is incomplete unless the whole analysis is moved to the implementation of results. Having investigated and modelled the practical process, implementation may itself demonstrate that various phases of the analysis were incomplete or need to be revised.
Because systems analysis of a framework of thought rather than a defined prescription, the list of steps above needs to be understood in a qualified sense. Not all of the steps need to be included in every example of systems analysis. Similarly, the order in which the phases are undertaken may be varied, or it may be necessary to work through them in various patterns. For example, the importance of excluded factors may be reassessed repeatedly, necessitating several cycles of the modelling and evaluating phases. The relevance of the objective structure of the analysis may have to be examined periodically, sometimes requiring a return to one of the earlier phases, even after a considerable amount of work has been done on some of the middle and later phases. The most useful models will mimic reality with sufficient precision to serve a broad spectrum of decision and decision-makers. The decision phase may, therefore, be diffuse and broad, and follow the completion of the formal scientific analysis. Some of the possible points of return to earlier phases are shown in Figure 3.
As described above, sensitivity analysis involves the investigation of the effects of changes in the input variables and parameters on the results obtained from the models. Large or small variations resulting from such changes in performance of the model help to determine the 'sensitivity' of the particular variables and parameters, and so provide a basis of comparison for the importance of such elements. Ideally, sensitivity analysis begins during the modelling phase and continues to the end of the research and the implementation of results. Those parameters to which the model behaviour is especially sensitive can often be made the subject of close scrutiny and subsequent modification, with further experimental work or data analysis to ensure that those processes are more precisely modelled. Indeed, sensitivity analysis may itself aid decisions about allocations of resources to various parts of the research programme.
Sensitivity analysis may be expected to concentrate on four major aspects of model performance, and, specifically, to investigate uncertainty which has been introduced during the modelling phase. First, such an analysis will concentrate on the relative uncertainty of parameter values used in the model. Few of the ecological parameters necessary for the modelling of dynamic systems are known exactly. They frequently have to be estimated from the results of earlier research, or as a result of special surveys and experiments. At best, there will be sample determinations which are assumed to be representative of some population parameter. Second, parameter values may be expected to be subject to experimental variation, partly reflecting the inherent variability of the biological material modelling in the dynamic system, and partly dependent on unavoidable differences in the ways in which successive trials and experiments are carried out. Experimental variation of this kind may have a significant influence upon the range of values within which the true values of the parameters lie, especially where the values are themselves not independent from one variable or parameter to another. Third, the importance of interactions between parameter values and the effects of significant variables within the dynamic system can seldom by overemphasized. Few, if any, models of dynamic systems can be reduced to such simplicity that the behaviour of the model systems is adequately defined by models with completely independent variables. The investigation of interactions in systems analysis makes it essential to perform such analyses by varying more than one variable at a time in the model system. Fourth, it is frequently necessary to extend the range of input state variables to ensure that the model behaves consistently for the full range of values for which predictions are to be made. Most model systems represent an extrapolation from the range of conditions and processes for which information is available experimentally, even when the scientists constructing the model have been careful to stress the dangers of extrapolation. It is important, therefore, to know the range of variables for which the model behaves in a reasonable manner, so as to ensure that those subsequently using the model do not extrapolate beyond these points.
As will be apparent, there are important implications of sensitivity analysis in the whole process of modelling. For example, sensitive regions of the model may indicate sensitive regions of the real system, and such regions may need to be stressed in terms of closer control in the implementation of results from systems analysis, and in the preparation of guidelines of management. Similarly, the process of sensitivity analysis may indicate the need for close validation of specific sub-systems, relationships or individual parameters. In particular, the analysis may reveal overemphasis of particular processes where these processes are themselves of relatively little importance in the overall system. In this way, it may be possible to achieve modification of the model required to give a more balanced representation of the real system. Again, as has previously been suggested, the analysis may indicate the sensitive areas or elements which have been isolated by sensitivity analysis and which need to be better understood and properly represented. Such a process again helps to establish research priorities.
Although many texts do not make a distinction between verification and validation, it is often helpful to do so, although the usage of these terms is certainly not consistent. Verification may be regarded as the process of testing whether the general behaviour of a model is a 'reasonable' representation of that part of the real-life system which is being investigated, and whether the mechanisms incorporated in the model coincide with the known mechanisms of the system. Verification is, therefore, a largely subjective assessment of the success of the modelling, rather than an explicit test of the hypothesis underlying the model. Some verification will inevitably have been going on during the hectic phase of mathematical activity, as the 'reasonableness' of the results will be one of the criteria by which the modeller will have judged the success or failure of his efforts. Nevertheless, what is reasonable in small parts of the model may be less so when the parts are put together into a composite of the individual parts. Interactions between responses and impacts may need to be explored sequentially and factorially to ensure that the full range of possible conditions have been covered, and that, within the limits bounding and defining the problem and the ecosystem, and model behaves, for the defining purposes, in much the same way as the real system. We must, of course, be careful that we do not reject a model simply because it behaves in a counter-intuitive fashion. There are plenty of examples of solutions which are contrary to what is usually regarded as common sense. No model should therefore be rejected simply because the results are unexpected. Where, however, the model behaves in a completely different fashion from the real system which is being investigated, some explanation has to be sought, at the very least, for the inconsistency. This is the role of verification.
Validation, in contrast, is the quantitative expression of the extent to which the output of the model agrees with the behaviour of the real-life system, and is an explicit and objective test of the basic hypotheses, made by means of a delineation of test procedures, primarily statistical, which are applicable to the determination of the adequacy of the model. In most ecological applications of systems analysis, this process of validation has hardly been attempted, mainly because of inadequate definition and bounding of the initial problem. Typically, validation, where it is attempted at all, is approached in a direct and obvious manner, mainly by observing the behaviour of the model systems under a set of controlled or measureable loading and other conditions and then comparing the observation with corresponding predictions of the simulator. When the observations and the predictions agree within the required limits for all conditions treated, the simulation is considered to be validated.
The procedure of validation has several recognized difficulties, not the least of which is the uncertainty associated with the drawing of general conclusions from a finite, and typically small, number of experiments. This uncertainty is of particular concern in the validation of systems analysis models where one may be attempting to predict effects which are of the same order of magnitude as the random fluctuation or 'noise' inherent in real system measurements. In such cases, it is advantageous to use techniques for statistical design and analysis of experiments, both to reduce the number of experiments needed for a given level of confidence and to indicate the statistical significance of measured and simulated effects. Fortunately, effective techniques for experimental design have been developed during the last fifty years, and techniques which were originally intended for use in experiments on real-life systems are now proving valuable in testing the behaviour of simulations of those systems. A full account of these techniques is given by Schatzoff and Tillman (1975), Kleijnen (1975), and Dent and Blackie (1979).
Systems analysis does not merely help to define priorities. The modelling process itself indicates the exact form in which data can be most readily used, so that simulation can assist in directing the tactics, as well as the strategy, of research planning. Similarly, while the results from the experiments may be published and disseminated by traditional forms of publication, they may also be assimilated and assessed within the models of systems analysis. As these models may eventually by incorporated into higher models as sub-models, they may be expected to have two major tasks, namely that of guiding the establishment of research priorities, and as a medium for the accumulation of research findings.
The introduction of systems analysis and systems modelling in ecological research also has a major effect upon the direction of research. Ecological research is costly, and the results are sometimes uncertain. Management of such research therefore requires objective guidance in establishing programmes for maximum effectiveness, and such guidance may be provided by systems analysis. If systems analysis is directed towards the ecological system, the objective of the better understanding of the real-life system so that is can be more effectively monitored and controlled helps to provide a linkage between research and the applied system. Such a linkage is illustrated in Figure 4.
Figure 4. A framework for linking applied research and modelling through systems analysis (after Dent and Blackie, 1979)
Following Rowen (1976), good systems analysis has the following characteristics.
The analysis uses methods which fit the character of the problem and the nature of the available data, while treating all data sceptically.
Systems analysis defines, explores and reformulates objectives, while recognizing that there may be several objectives capable of being arranged in a hierarchy.
Good systems analysis uses criteria sensitively and with caution, giving weight to qualitative as well as quantitative factors.
Effective analysis emphasizes design and creation of alternative solutions and options, and avoids concentration on too narrow a set of options.
Modelling within systems analysis should handle uncertainty and stochastic variables explicitly.
The approach to the problem should indicate that the analyst understands the essential nature of the practical problems.
It is important to use simple models to simulate the essential aspects of the problem, and to avoid large and complex models that attempt to mimic reality while concealing the basic structure of the problem and the uncertainties of the estimation of model parameters.
The results should display honesty in the labelling of assumptions, values, uncertainties, hypotheses and conjectures.
The whole process of systems analysis should demonstrate understanding. The task is not merely to indicate the 'best' solution, but also to develop a range of alternatives.
The analysis and its results should also show that an effort has been made to understand the practical problems and constraints of management and administration, especially if the analysis suggests a radical reformulation of the problem.
The solutions should take into account the organizational factors that affect the alternatives generated and influence the decisions.
The whole procedure of systems analysis should exhibit awareness of partial analysis, and the limits of analysis generally.
These additional precepts have been suggested as a result of experience in IIASA.
Good systems analysis makes as certain as possible that the suggested alternatives are feasible.
The analysis should consider the difficulties of the implementation of solutions and the costs of achieving them.
The analysis should recognize that an approximate solution before any decision has to be made is better than an exact solution long after the decision has been made.
|
|
|
The electronic version of this publication has been
prepared at |