Analysing Uncertainties of Interval Judgment Data in Multiple-Criteria Evaluation of Forest Plans

The use of interval judgments instead of accurate pairwise comparisons has been proposed as a solution to facilitate the analysis of uncertainties in the widely applied pairwise comparisons technique. A method is presented for deriving probability distributions for the pairwise comparisons and for utilizing the distributions in the analysis of uncertainties in the evaluation process. The first step is that the expert or the decisionmaker is queried as to the best guess of the priority ratio of the attributes compared. This is followed by an adjusting query concerning the uncertainty in the comparison: what is the probability of the priority ratio being between the best guess ± 1 unit of the pairwise comparison scale? An application of the method is presented in the form of multiplecriteria evaluation of alternative management plans for a forest area.


Introduction
The result of forest planning is usually a management plan for the forest area under review.The plan presents a recommendation as to the action plan for the forest area and predicts the consequences of implementing the plan.Among other steps necessary in a planning process (see e.g.Kangas 1992), each alternative plan to be compared should be evaluated with respect to each objective.In addition, the weights of objectives having importance to the decision-maker should be assessed.Furthermore, the evaluations with respect to single objectives must be made commensurable in order to enable numerical prioritization of decision alternatives with respect to overall utility.
Nowadays, objectives other than those based solely on wood production and commercial benefits are of increasing importance in forestry decision-making, not only with the public, but also with private forest landowners (e.g.Hyberg and Holthausen 1989, Kreutzwiser and Wright 1990, Kangas and Niemeläinen 1996).Evaluation of alternative forest plans with respect to biodiversity considerations, for example, are ideally based on empirical models about the ecological effects of implementing the plans.Unfortunately, producing models via empirical research is laborious and too slow from the viewpoint of the acute needs of the forestry practice.In addition, some of the objectives of present-day forestry are of a subjective nature; e.g. the scenic beauty of the forest landscape.One way to satisfy the most urgent needs is to use evaluation models produced on the basis of expert knowledge and/or subjective preferences.Expert knowledge and subjective preferences have been used in applications of the Analytic Hierarchy Process (AHP) (e.g.Mendoza andSprouse 1989, Kangas 1992).In the study by Kangas et al. (1993b) a group of experts evaluated a set of forest areas with respect to the habitat requirements of a chosen wildfowl species in a pairwise manner.Comparisons were analysed and the relative priorities of the forest areas were calculated using Saaty's (1977) eigenvalue technique as applied in AHP (Saaty 1980).The same principles were applied in the estimation of scenic preferences of non-industrial, private forest landowners for the purposes of decision support concerning their own forests (Kangas et al. 1993a).Crawford and Williams (1985) showed how pairwise comparisons data can be analysed by means of regression analysis.The greatest advantage of the regression approach when compared to Saaty's eigenvalue method (Saaty 1977(Saaty , 1980) ) is that there is a well-known statistical theory behind it.Sound statistical methods enable the analysis of uncertainties in judgments in a more versatile manner and in greater depth than is possible when using the eigenvalue technique.Alho et al. (1996) extended the work of Crawford and Williams to the case of multiple judges.They applied variance component modelling in producing quantitative estimates of the uncertainties.Alho and Kangas (1997) showed how the results of the analysis can be utilized in multi-level decision hierarchy when evaluating alternative forestry strategies in the single-judge case.They also provided a Bayesian approach to the regression technique, which makes it possible to summarize judgments in a manner more understandable to decision-makers.
All the elicitation techniques presented above make use of pairwise comparisons data.A major disadvantage of the way the comparisons are gathered lies in that exact comparisons as absolute bounds for the ratios should be given by the decision-maker or the experts.However, in most evaluation tasks, it is difficult to accurately indicate the weights, preferences and priorities.According to the results of previous studies conducted with the purpose of trying to take into account subjective preferences via pairwise comparisons made by the decision-makers, a problem of central importance is the difficulty of presenting exact single-comparison values (e.g.Kangas et al. 1993a).The same problem holds true for expert judgments (Kangas et al. 1993b).This could be addressed through the use of interval judgments instead of accurate pairwise comparisons and corresponding single numerical values (Salo and Hämäläinen 1992, 1995, Moreno-Jimenez and Vargas 1993).
Knowing the probability distributions of the comparison values as a measure of preference or priority structure would facilitate the analysis of uncertainties in the judgments.However, dealing with only minimum and maximum values for the comparisons -as has previously been the case -produces only limited information about the comparison uncertainties.In addition, a crucial problem is to determine which points of a probability distribution these minimum and maximum values refer to.
This study presents a technique for deriving the probability distributions of pairwise comparisons, and shows how these distributions can be utilized in the Bayesian analysis of uncertainties in judgments.The technique is based on the approach by Alho and Kangas (1997), and it can be utilized in multiple-criteria evaluation of decision alternatives in single-judge cases.The probability distributions of pairwise comparisons are based on judge's own statements.If there would be several judges, as was the case in the study by Alho et al. (1996), it would appear to be possible to extend the procedure to a multiple judge case as well.
In the technique, the judge is first queried as to the best guess of the priority ratio of the attributes being compared.Following this, the judge is asked an adjusting question on the basis of which the distribution can be derived.An application of the technique is presented in the form of multiple-criteria prioritization of alternative forest plans for a forest area.

Methods
The regression model for pairwise comparisons data in single-judge case is (Crawford andWilliams 1985, Alho andKangas 1997) log( ) where r it is the relative value of attribute i compared to attribute t as perceived by the judge, α i is the logarithm of the true value of attribute i, and the error terms ε it are uncorrelated with the expected value zero and variance σ 2 .Parameters α i and σ 2 can be estimated by using ordinary least squares (OLS).To ensure identifiability, it is required that α I = 0, where I is the number of attributes to be compared.In multi-level decision hierarchy, Equation (1) will be applied repeatedly.Further, Alho and Kangas (1997) used non-informative prior proportional to σ -1 for the pair (α, σ), where α = (α 1 , ... , α I-1 ) T and outlined a procedure in which α can be simulated from the joint posterior distribution of (α, σ) based on OLS.Thus, the posterior distribution for the priorities of the decision alternatives can be studied by means of simulation.
The variance of the posterior distribution of priorities (non-informative prior) is positive, if the pairwise comparisons are inconsistent, i.e. σ 2 > 0. (For inconsistent pairwise comparisons r it ≠ r ij r jt , and for consistent pairwise comparisons r it = r ij r jt , e.g.Saaty 1980).If, for some reason, pairwise comparisons are consistent, then σ 2 = 0.This would imply that estimates of α i are certain, because the variance of the posterior would be zero.However, the judge may still be uncertain of the priorities.For example, when comparing only two attributes, say attributes A and B, the judge might say that attribute A is "four times better" than attribute B, but considers that "three times better" and "five times better" are also possible with a certain subjective probability.In this example, the uncertainty of the judge is not reflected at all in the posterior, because comparing only two attributes automatically leads to consistency.Next, a simple method is proposed that will lead to positive posterior variance also when pairwise comparisons are consistent.
The variable r it will be recorded as numbers ..., 1/9, 1/8,..., 1/3, 1/2, 1, 2, 3,..., 8, 9,...An open and discrete pairwise comparison scale is used together with corresponding verbal comparisons as recommended by Saaty (1980), although the use of a continuous scale would also be possible.It is also possible to replace Saaty's numerical scale for r it by a sequence with geometric progression (Lootsma 1993), for example.The entry 1/9 means that the priority of attribute A is equal to 1/9 times the priority of attribute B. Correspondingly, the entry 9 means that the priority of attribute A is equal to 9 times the priority of attribute B, and similarly the entry 1 that the priorities of the attributes A and B are equal.For example, the quantity 9 can be presented verbally as "absolute importance or preference".
Suppose that the opinion of the judge could be described in each pairwise comparisons by a normal distribution in Saaty's scale, although the modifications to other distributions are also possible.Now the opinion can be characterized by asking the judge: Question 1: "What is your best guess of the priority ratio of the attributes A and B?" Question 2: "What is the probability for the priority ratio to lie between the best guess ± 1 unit of the pairwise comparison scale?" An alternative to asking for the probability, when the upper and the lower limits are fixed, would be to ask for the limits, when the probability is fixed.In either case, the variance can be computed by using normal distribution, when the upper limit, the lower limit, and the probability that the relative goodness is between these limits, are known.Because the difference from one value to another is not equal on the Saaty's scale ..., 1/9,..., 9,..., the parameters of the normal distribution must be estimated using some other scale.The scale ..., -8, -7,..., -2, -1, 0, 1, 2,..., 7, 8,... is used, where, for example, -8 represents the original value of 1/9 and 8 represents the original value of 9.
When the distribution of each variable r it is known on the scale ..., -8,..., 8,..., one can generate a data set from these distributions, round it to integers, and code it back to the Saaty's scale.Suppose, for example, that the judge's answer to Question 1 is "1/2" and to Question 2 "0.95".Then the normal distribution N(-1, 0.765 2 ) is used (on the scale ..., -8,..., 8,...), where the variance, say V 2 , is obtained from the equation 0.5 -(-2.5)= 2 • 1.96 • V.It should be noted here that because of the rounding carried out, the lower limit of -2.5 and upper limit of 0.5 must be used, and not the lower limit of -2 and upper limit of 0. As a result, on the Saaty's scale, one have the expected value of 1/2, and the probability of getting values 1/3, 1/2, or 1, is 0.95, as was intended.Further, the probability of getting the value 1/4 or smaller is 0.025, and the probability of getting the value 2 or greater is 0.025.Now the response variable log(r it ) in Equation (1) can be generated.It reflects the views of the judge, including the probability asked for.After generating a single response variable, the parameters associated with the response are estimated by using OLS.If the generated response variable leads to inconsistency, this is taken into account by generating one sample (not several) of α from the joint posterior of (α, σ) associated with the response according to Alho and Kangas (1997).After doing this for all regressions needed in multi-level decision hierarchy, a single sample from the posterior distribution for the priorities of the decision alternatives can be computed.It reflects both the inconsistency and the variance of the pairwise comparisons given by the judge.The nature of the posterior can be studied by repeating the whole simulation procedure as many times as necessary.
The posterior of α was based on OLS.If the intervals given by the judge had been taken only as a measure of relative uncertainty between different pairwise comparisons, then weighted least squares (WLS) should have been used instead of OLS.This is because the error terms of the data have different variances, and therefore some of the pairwise comparisons are more reliable than the others.WLS would lead to a smaller variance in the estimator than OLS would, and the result would be smaller uncertainty of the priorities.However, it is now considered that the intervals as such reflect the uncertainty in the comparisons made by the judge, together with the inconsistency of the best guesses.In this case, OLS is a reasonable choice, because it uses equal weights for all pairwise comparisons.

Case Study
The case study area located in Kuusamo, northeastern Finland, covering 321 hectares of forestry land.The area is owned by the State and administrated by the Finnish Forest and Park Service (FPS).At time of this study being conducted, most of the forest stands within the area were more than 100 years old, mostly mature stands of Norway spruce (Picea abies), with admixtures of Scots pine (Pinus sylvestris) and birch (Betula pendula and B. pubescens).For the purposes of simulation of the development of forest trees and planning calculations, the area was divided into 71 compartments, each compartment having relatively homogenous tree and soil characteristics.
The forest management staff of the FPS determined the planning objectives: timber production, effects on the scenic beauty of the forest landscape, and game management considerations.Six alternative forest management plans were generated using a linear-optimization-oriented forest simulation software.In order to assess the management plan alternatives, the primary objectives were decomposed into second-level attributes.Priority with respect to timber production was determined as a function of the net income in the first ten-year-period, net income in the second ten-year-period, and the stumpage value of the forest area at the end of the second period.Scenic beauty was divided into far-view scenery and within-stand scenery.Priority with respect to game management was determined on the basis of the habitat requirements of moose (Alces alces), capercaillie (Tetrao urogallus), and black grouse (Tetrao tetrix, Lyrurus tetrix).Pairwise comparisons were applied in the evaluation of alternative management plans with respect to each second-level attribute, as well as in determining the relative importance of the objectives and attributes.For more details on the case study material and the original planning situation, readers are referred to Kangas et al. (1992).
Original comparisons, i.e. the best guesses, were made by members of the staff of the FPS.The same data were also applied by Alho and Kangas (1997).Due to organizational changes in the FPS, the same management staff could not be used as judges in this case study.Consequently, the interval comparisons and the results of the comparison process cannot be considered as being those of real decision support.The aim of the case study is to test the method and illustrate it, rather than produce immediate support for a real life choice problem.
The data on best guesses (Question 1) was used, and in addition to that, a judge was asked for the probability of the interval (Question 2) (see Appendix).In Appendix, there are also probabilities equal to 1, which means that the judge considered the event described in Question 2 to be certain.In this case, the variance cannot be computed.Instead of 1, the probability of 0.99999 (ad hoc) was used.Another solution would have been to ask the judge the probability of the best guess.If, for example, the judge had answered to Question 1 with "1/2", to Question 2 with "1", and to the question of the probability of the event 1/2 with "0.95", it could be concluded that the probabilities of the events 1/3, 1/2, and 1, are 0.025, 0.95, and 0.025.
The data presented in Appendix was analyzed by computing the summaries of the priorities based on 3000 simulations from the posterior (Tables 1 and 2).The simulation procedure was programmed by using Minitab.In Table 1 the row "Mean ±" tells how uncertain the computed posterior mean is due to the number of simulations.For example, a 95 % confidence interval for the posterior mean of Plan 1 is (0.180, 0.184).When comparing posterior means and posterior medians, it is found out that all posterior distributions are skewed to the right.The probability of a plan uniformly beating all the other plans gives an alternative way to examine the goodness of a forest plan.The 95 % confidence limits of the probabilities vary from ± 0.00 to ± 0.02.The 95 % Bayesian credible intervals are such that the posterior probability is 2.5 % that the relative utility of the decision alternative in question is smaller than the lower limit, and similarly for the upper limit.Table 2 gives the pairwise posterior probabilities that a row plan beats the column plan.The probabilities that a column plan beats the row plan are probabilities of the complement events of Table 2.The 95 % confidence limits of pairwise probabilities vary from ± 0.01 to ± 0.02.
In order to demonstrate the effect of Question 2, some additional simulations were performed by using the probabilities of 0.95, 0.725, and 0.50 as answers to Question 2 for all pairwise comparisons (Tables 3-7, rows c), d), and e)).The number of simulations used were 2400, 3500, and 4900.Because of this, the uncertainty of the  3, rows c)-e), is approximately the same as in Table 1, that is, the row "Mean ±" in Table 1 can also be used here.Furthermore, the length of the confidence intervals of the pairwise probabilities, and the probabilities of a plan uniformly beating all the other plans, are approximately the same as in the case of Tables 1 and 2. Tables 3-7 include also results of Alho and Kangas (1997), rows a) and b).In Table 3, row a), are the regression estimates for the priorities of the forest plans obtained from the data of best guesses.The probabilities in Tables 5 and 6 in row a), are obtained by comparing the size of these regression estimates.For example, the probability of Plan 1 beating Plan 2 is one, because the regression estimate of Plan 1 is bigger than that of Plan 2. Similarly, the probability of Plan 1 beating all the other plans is zero, because Plan 4 has a bigger regression estimate than Plan 1.The row b) in Tables 3-7 is based on 1500 simulations from the posterior on the data on best guesses.
The row a) may be considered to represent a situation with no uncertainty at all.The uncertainty caused by the inconsistency of the best guesses is being noticed in row b).In rows c)-e), also the different probabilities of the interval are allowed to have an effect on the results.In other words, the level of uncertainty of the pairwise comparison data increases from a) to e).
From the posterior means it is found out that the level of uncertainty has an effect on the goodness of the forest plans (Table 3).Plans 1, 2, and 6 seem to be worse off, Plans 3 and 5 better off, and Plan 4 the same, when the level of uncertainty increases.The reason for this phenomenon lies in that the symmetry assumption made in Saaty's pairwise comparisons scale by assuming normal distribution does not lead to symmetry in priorities.This can be seen by comparing the means to  1997), and row b) the posterior means, when the only source of uncertainty is the inconsistency (Alho and Kangas 1997).In addition to the inconsistency, the probability of all the intervals is supposed to be 0.95 in row c), 0.725 in row d), and 0.50 in row e). the medians in Table 4.One alternative to assuming normal distribution on the Saaty's pairwise comparisons scale would be to assume log-normality, i.e. normality on the scale of logarithms of the pairwise comparisons.This would lead to a closer symmetry of the priorities, but still some asymmetry would be observed.This is why also the row b) seems to fit in the trend of the posterior means caused by the uncertainty.The posterior medians behave much like the posterior means, but with the exceptions that Plan 4 seems to be slightly worse off, and Plan 5 the same, when the level of uncertainty increases.It should be noted, that the posterior medians of different forest plans does not sum up to one like the means do.
The effect of uncertainty can be seen clearly in pairwise posterior probabilities (Table 5).In general, if the pairwise probability in row a) equals one, the increase in uncertainty will make the pairwise probability smaller, and correspondingly, if the probability in row a) is equal to zero, it will get bigger.The probabilities of Table 6 have the same quality, with the exceptions of Plans 1, 5, and 6.These plans seems to be independent of the level of uncertainty.Finally, approximate 95 % Bayesian credible interval limits (Table 7) increase in width with increasing level of uncertainty.
Now the results of the data in Appendix can be interpreted.In general, if Question 2 is taken into account, the results will be different compared to the results of analysis that uses only the information from Question 1.In the Appendix, the probabilities of the interval given by the judge vary from 0.6 to 1 (1 was interpreted as 0.99999).Thus, the results are somewhere around the results obtained by using the probabilities of 0.95, 0.725, and 0.50 for all the intervals.

Discussion
The present study involved developing a technique for analysing uncertainties in judgment data gathered via interval pairwise comparisons and for taking the uncertainties into account in multiple-criteria assessment of decision alternatives.Judgments based either on expert knowledge or on subjective preferences (of the decision-maker) can be dealt with when using the technique.Both expertise and preferences can be utilized in the one and the same multiple-criteria evaluation process.The ability to deal with uncertainties involved in the assessments of decision alternatives is an important quality in any approach of decision support.
Although the computation technique, using which the interval comparisons are analyzed, may seem complicated, the use of the technique is relatively easy.However, for at least some forestry decision-makers not familiar with the fundamentals of statistics, answering the questionnaire needed in order to derive the distribution describing the preference structure can be difficult.In practical decision support, a difficult task is that of choosing the most accurate planning approach and preference modelling technique for each planning process which, at the same time, is convenient and easy enough to apply.Another tough problem is to implement the results of the analyses to decision-makers.Empirical research is needed in order to study forestry decision-makers' ability to answer different kinds of questionnaires and to understand  The normal distribution was used in the Saaty's pairwise comparison scale to describe the uncertainty of the judge.This leads to asymmetry in the priority scale and causes trend in the posterior means and medians, which depends on the probability of the Question 2. One alternative would have been to assume log-normality, although even this would cause some asymmetry in the priorities.The justification in making distribution assumptions depends on how well the assumption describes the opinions of the judge.However, it is perhaps more natural to assume normality in the very same scale that is used in comparisons.The method makes it also possible to analyse data, where the judge has determined the distribution entirely, i.e. without any distribution assumptions.In the present study, the Saaty's scale was used, and normality was assumed in the same scale.Due to normality assumption, the demonstration of the effect of uncertainty was done simply by changing the probability of Question 2.
The simulation of the posterior distribution enables computing several kinds of summaries that can be used to evaluate the goodness of the decision alternatives.Some summaries were computed in this paper, but there are also other possibilities.For example, posterior probabilities for events "alternative A is better than all the other alternatives in a given subset" can be readily estimated.Also, the attitude toward risk could be taken into account when choosing the final forest plan (Pukkala and Kangas 1996).So, flexibility is one of the strengths of the Bayesian approach.
If the analysis of pairwise comparisons data is based only on the best guesses, it is assumed that the judge can give the pairwise comparisons exactly, although some inconsistency may exist.However, this may not be the case in practice: the judge may be able to give the best guess of the relative goodness of the alternatives, but, at the same time, he/she may consider that some other values are also possible.If these other values are not taken into account in the analysis, the uncertainty of the priorities will be underestimated.

Total of 18 references
Appendix.Pairwise comparisons, and corresponding probabilities of the interval.For example, the entry 6 means that the criterion timber production is six times more important than criterion scenic beauty (Question 1), and entry (0.85) is the corresponding probability of the event "5, 6, or 7" (Question 2).
net income during first ten years (n1), net income during second ten years (n2), and stumpage value after twenty years (st) with respect to timber production.within-stand scenery (wi), and far-fiew scenery (fa) with respect to scenic beauty.
..., 6 with respect to net income during first ten years.

Table 2 .
Pairwise posterior probabilities that a row plan beats the column plan.

Table 1 .
Posterior means, 95 % confidence limits for the posterior mean, posterior medians, probabilities of a plan k uniformly beating all the other plans, and appoximate 95 % Bayesian credible interval limits for the priorities of the forest plans.

Table 5 .
Pairwise posterior probabilities that a row plan beats the column plan (see explanations for a) -e) in Table3).

Table 3 .
Regression estimates and posterior means.Row a) shows the regression estimates for the priorities of the forest plans(Alho and Kangas

Table 7 .
Approximate 95 % Bayesian credible interval limits for the priorities of the forest plans (see explanations for b) -e) in Table3).

Table 6 .
Probabilities of a plan k uniformly beating all the other plans (see explanations for a) -e) in Table3).