Issues in the specification of regret-only choice models: a rejoinder to Chorus and Van Cranenburgh

Before starting the debate, we would like to state explicitly that we share Chorus and Cranenburgh’s (C&C) concerns about prevailing confusion about regret-based models and their fascination of exploring the value of context-dependent choice models in transportation research and beyond. Although the number of applications of random regret models has gradually increased over the last decade, still there are many non-believers as evidenced by critical, skeptical questions at conferences, questionable paper reviews and the contents of informal discussions. First, some peers have questioned their theoretical foundation, as they are not based on classic axiomatic theorems of rational decision-making. Second, the relevance of random regret models has been downplayed as empirical studies have shown that even if these models perform better than random utility models, their performance is only marginally better. Therefore, it has been argued their further development is not very worthwhile. Thirdly, compromise effects may have been observed in some marketing studies, doubts have been raised about their importance in transportation. Finally, we have often heard that because choice probabilities in both the MNL and random regret models are based on utility/regret differences, the models are fundamentally the same; although most MNL models assume a linear-additive utility function, there is no reason not to use non-linear (reference-based) functions, fading out their difference. These skeptics argue we do not need the concept of regret; the concept of utility is sufficient to formulate more flexible choice models. These reactions suggest a lot of confusion and lack of understanding of regret models. We have no desire to further contribute to this situation and therefore like to take this opportunity to further explain the considerations that led to our 2016 paper (Rasouli and Timmermans 2016) and articulate our opinions in response to C&C’s comments.

To start, it may be fair to acknowledge that part of the confusion may have been caused by the gradual evolution of regret-based models. Over the years, small incremental changes have been suggested and new insights and interpretations have been added. It is therefore critically important that readers are aware of the state of the art at the time of the design of the study reported in our paper and the writing of the results. This is particularly the case because C&C argue that we “overlooked past work on the topic”. In response, we did not overlook past work; the paper is based on the state of the art when the paper was written. When we designed the study, the only available publications on regret-based models in travel behavior research were the original Chorus et al. (2008) paper on the random regret model (RRM2008) and the paper on Chorus’ (2010) specification (RRM2010) plus a series of article, listed in our paper, comparing the empirical performance of the latter model against the MNL model.¹

C&C use the opportunity to promote and summarize their latest (2014–2015) publications. For those readers, not familiar with their recent publications, it gives an excellent overview of developments in regret-rejoice models after we conducted our study. It is tempting to express our (full) assessment of these new models, but we decided not to as it would not be fair to the authors, who would not have the opportunity to respond to the discussion. Rather, we prefer to rely on submitted papers and work in progress for a detailed discussion of some specific issues. Jang et al. (2016) is the first of a larger series of papers in which we aim at improving and extending regret and regret-rejoice models, both under certainty and uncertainty, often based on a critical assessment of a particular assumption or property of existing models. At the same time, we realize this may be disappointing for readers as most of C&C arguments are based on their latest work and latest insights.

Therefore, we limit this rejoinder to the comment on “erroneous theoretical claims” and include their more recent work to the extent we think it helps in avoiding confusion. Critical for the positioning of our paper is to realize that, in contrast to seminal work on regret models, the 2008 specification is a regret-only model. Regret for each attribute was assumed to be proportional to the difference between the attribute levels of the considered/chosen alternative and the attribute level of the BEST foregone choice alternative. In addition, overall regret, defined as the sum of attribute level regret, was also defined against the best non-chosen choice alternative. In equation:

It should be noted that the formulation of this regret-only model does not imply we believe regret-only models are necessarily superior. Rather, the regret-only models serve as an interesting benchmark to differentiate the specification from seminal regret theory introduced in decision sciences² and regret-based decision heuristics examined as rules of bounded rational behavior (see e.g. Rasouli and Timmermans 2015). We believe it is valuable to explore the theoretical properties of regret-only models and assess their empirical performance against utility-maximizing and regret-rejoice models. The main contribution of the 2008 study is that it showed the potential of random regret models. Still, several technical, econometric and behavioral aspects of the model could and should be systematically investigated and possibly improved.

In his 2010 paper (RRM2010), Chorus addressed one of these issues: the discontinuous regret function may cause problems in the estimation of the regret function. As an alternative, he suggested the following specification.

Note that if we leave out the constant, the two regret functions are identical. If attribute differences are large, the effect of the constant is negligible, but differences between the two specifications increase if attribute differences are small. When the attribute values of the considered and the best alternative are the same, regret is positive while we argue logically it should be equal to zero because the considered/chosen alternative is the best and therefore there cannot be any regret. This is also clearly shown in Chorus’ RRM2010 Fig. 1.³ We argued this is theoretically weak. In turn, C&C claimed that our argument is incorrect and concluded that subtracting ln(2) from the value of the attribute regret function will solve the problem that RRM2010 gives non-zero regret even if the chosen alternative outperforms the other(s).⁴

Their wording is as if this “solution” is something we did not realize. This is surprising and misleading because in our paper, we presented an equation that captures the difference between the original regret function and the RRM2010 approximation. It shows that under the same attribute values the difference between the two specifications is ln(2): exactly the “solution” they suggest. It is even more interesting to read C&C’s diagnostic conclusion that we do not understand the basics of choice modeling and regret theory. Upon self-reflection, we like to believe that we know the basics of discrete choice and regret theory at least well enough to understand their argument and to argue it is not a satisfying “solution” as it would enhance the confusion in the field that we would like to avoid or temper: we learned that many peers do think that because choice probabilities under utility-maximization only depend on utility differences, the properties and assumptions of the MNL model can be fully generalized to regret models. Mathematically, this is correct; parameter estimates only depend on the differences; goodness of fit and predicted choice probabilities are not affected; no discussion and no confusion.⁵ Our point is that as behavioral modelers we need to superimpose additional requirements. The concept of utility is a continuous concept, usually measured at an interval scale. The concept of regret is not. Logically and operationally it is only defined in the positive domain. Allowing for violations would fuel the notion that the concepts of utility and regret are behaviorally very similar. We do not question that choice probabilities are a function of regret differences. However, we do prefer model specifications that have a direct behavioral interpretation and that do not depend on some ad hoc correction. It is a non-issue in utility models as the concept of utility goes from minus to plus infinity. In contrast, however, regret is only defined for the positive domain and we prefer arguments in the regret function that have this property. We realize that one can take a different stance on this issue. We simply prefer to avoid any motivation based on classic discrete choice models, as C&C do, to correct matters in an ad hoc fashion. We particularly prefer this because we can easily assume strictly positive regret functions for regret-only models.

More importantly, subtracting ln(2) is NOT a solution. Indeed, as implied in the equation we presented, regret in RT2016 becomes zero for zero attribute differences. However, it will shift the complete regret function. Consequently, regret may become negative when the attribute of the considered alternative outperforms the attribute of the best/another choice alternative. This violates the very notion of regret. It also is problematic because our discussion concerns regret-only models. Recall the reason for the new regret specification was to obtain a smooth approximation of the original function. Chorus (2010) states: “Second, the (2008) model specification’s likelihood function is non-smooth, which creates difficulties with respect to the derivation of marginal effects and elasticities, and triggers a need for customized optimization routines to successfully estimate the model. This means that model estimation relies on handwritten code, which in turn hampers the model’s general applicability, especially among practitioners”. This is a practical argument. The underlying problem could be easily solved in other ways (e.g., dedicated software (stand alone or included in general software such the 2010 model now is), smaller constant, other function), avoiding a regret function that in C&C own words seems (at first glance) counterintuitive.⁶

By focusing on this minor issue, C&C seem to take away the focus from the more important point that the two regret functions differ when attribute differences are small. Subtraction of ln(2) makes the value of regret based on the max operator and logarithmic function equal only when the two alternatives have exactly the same values for the corresponding attribute. Equalizing the two regret specifications is not as straightforward as they claim.⁷ There remains a difference between the two specifications for small attribute differences, which led to the dedicated design of our stated choice experiment.

Another, and maybe more important difference between the two models, is whether people experience or consider regret against the best choice alternative only or against all better choice alternatives. The RRM2008 paper provides arguments in favor of the best only model. Stimulated by one of our reviewers, our 2016 paper also emphasizes arguments in favor of best only models. In contrast, C&C discussed limitations of best only models, and we have also expressed theoretical reasons why regret-rejoice models may be more appealing. Anyhow, the RRM2010 paper suggests regret is based on all pairwise comparisons.⁸ As said, stimulated by one of the reviewers, we emphasized in RT2016 why regret may be viewed against the best foregone alternative only. Ultimately, however, we said whether regret is defined against the best only alternative or against all other alternatives is an empirical issue. We can imagine that if a person uses the first petrol station he encounters and then finds out that all next stations along his route have cheaper gas, his regret will be higher than in the case he finds out all other stations but one (the absolutely cheapest) were more expensive. On the other hand, we doubt regret will significantly increase if a traveller knows that sometimes his habitual route from work to home becomes extremely congested and he would have been better off by choosing one of approximately equivalent, strongly overlapping alternative routes. More importantly, because we view behavioral models as mathematical representations of assumed behavioral processes, we have difficulty assuming that individuals cognitively assess the amount of regret in large choice sets, such as housing, route choice and the like by systematically comparing all choice alternatives. Thus, we feel it is important to formulate a model of the mental representation of the choice problem that will predict the number of choice alternatives that will be included in the comparison.⁹ Our experiment therefore allowed the comparison of the best only versus all comparisons specifications, admittedly in a still limited choice problem context.

The argument of the number of case studies is too far-fledged in our opinion. We would never claim the general superiority of one model against another on the basis of a single data set. Also, we do not fundamentally prefer the 2008 model to the 2010 model: we just compared their performance. Our criticism only concerns the specification of the regret function that has counterintuitive properties for regret-only models that can be easily avoided. We again want to emphasize that the focus of our comparison concerned the relative performance of these two model specifications for small attribute differences, because the two model specifications are approximately identical for larger differences.

If C&C’s argument is meant to suggest that no one should submit papers for publication until a model has been successfully applied to many data sets, we fundamentally disagree. We would not have seen the original regret model until much later. Waiting until a particular model has shown to outperform another for a large number of data sets does not seem very productive. We should stimulate that if a new model seems to have potential (there is evidence it outperforms a well-established model), it will be accepted in journals. This increases the chances that peers will estimate the model on new data and suggest improvements. That is called progress and avoids inertia. We do expect that the performance of a particular choice model will depend on the choice problem and choice context. It is more interesting to examine whether it is possible to find conditions or type of problems under which a particular model tends to outperform others and to identify the reasons why it does not. This research may enrich the specification of the context in which a particular choice mechanism seems dominant. We need a spectrum of principles and corresponding models and can only hope that we find some degree of regularity between choice mechanisms and choice context. Rather than taking a defensive stance, we would like to stimulate our research community to present their findings (positive and negative) of applications of existing model specifications at conferences and workshops and that reviewers be more liberal with respect to acceptance of any new specifications in the journals. Having said that, in the meantime, we did compare the two model specifications using new, both revealed and stated preference, data sets. The conclusions in our 2016 paper were confirmed for these other data sets (see e.g. Jang et al. 2016). It shifts the support ratio slightly, but again this is less important than trying to understand why a particular model outperformed another.

In summary, filtering out and ignoring the unjustified, unnecessarily suggestive and unwarranted comments, we arrive at the following contentions.

We wish to emphasize, although this should be self-evident, that our reflections are not meant in any way to question the general quality and important contributions of Chorus and his co-authors. In fact, we think their work has given new impetus to behavioral choice modeling in transportation that started to show signs of satiation, consolidation of old positions and marginalization of progress.

In that context, we would like to argue that the issues debated here are minor compared to the issues mentioned in our introduction. A larger research agenda continues to further critically question and examine the behavioral and econometric assumptions of current regret-only and regret/rejoice choice models, including C&C 2014 and 2015 generalized versions and our own work, explores the performance of alternative formulations, and expands these models into more complicated classes of choice problems. We expect to systematically investigate these issues with our PhD students and in joint publications with Chorus and van Cranenburgh in future work.