Human Decision Making

The third in a series.

The third in a series with Mauricio on critical rationalism and bayesian epistemology. See one, two, three, four, five, six, and seven.


One part of your response that I especially appreciated was your point about how Bayesians / EAs often rely heavily on probabilities with little backing (or none?) as if they were well supported. I’ll argue that what EAs are doing that looks like this is often fine, but after reading your last response I expect to be paying more attention to how probability estimates were formed, and to what this implies for their value as information, and for the value of further information.

There’s a few points I want to skip over for now:

my main concerns don’t really hinge this… I’d rather not argue about the meaning of ‘existence’ if we can avoid it 🙂

Fair enough 🙂

As well as:

Without [probabilities based on measurements], we are claiming knowledge where we don’t have it and this is dangerous.

…we don’t have a total order across explanations, so can’t coherently talk about “second best”, “third best”, etc

If the Future of Humanity Institute dedicated itself to coming up with societal mechanisms to prevent conflicting ideas from becoming violent, I think it would go much further to achieve its goals.

But there are no objective probabilities - no measurements from which to derive probabilities - when dealing with the unconditional future.

Our disagreements on these points seem downstream of another disagreement: do humans in fact make decisions using subjective credences, and should we? I agree that this is “an important disagreement worth spending time on.”

To address that question, you write:

…What ideal rational agent would use a decision making process which starts with an infinite regress?

Infinite regress isn’t a problem when decision theory is used in machine learning, reinforcement learning, econometrics, etc, because the actual decisions are being made by human beings. The word “decision” in this case is just referring to a statistical procedure of learning from data, and is not the same as the (unknown) psychological/biological phenomenon called “decisions” in steps 1-5.

“Subjective credences” are often talked about as if they are just formed in your brain automatically and allow us to avoid having to make decisions (and thus avoiding the infinite regress problem). But this isn’t true. Try it - try filling in a few entries of the conditional probability table in step 3 for example. What number are you going to write down for p(burns mouth | drinks tea) = c ? Or p(has a productive day at work | drinks coffee) = d? What are c and d? One could tally all the times one has burned their mouth drinking tea, but that is collecting measurements and switching back to objective probabilities, which we agree are legitimate.

I understood your argument here as being the following. Please let me know if I misunderstood or am interpreting uncharitably.

Humans do not (and should not) use expected-utility-maximization (of the sort in which consequences are weighted by subjective credences), because attempting to do so requires first making decisions about one’s options, probabilistic predictions, and utility functions. This amounts to beginning with infinite regress, which is not a great way to make decisions. Computers that maximize expected value avoid this problem by having humans make these initial decisions, and then updating their probabilities by using data according to programmed rules. If humans worked like this, then we could follow decision theory without falling into infinite regress. However, humans don’t work like this–our inability to introspect on precise, numerical subjective credences is strong evidence for the claim that we do not automatically calculate subjective credences. Even if we formed subjective credences by (consciously?) examining data, we would just be following frequentist statistics, and this would not support the claim that Bayesian statistics offer further value.

​My current impression is that humans do work roughly like the computers you mention, in that we are born with (or otherwise develop, without choosing) at least a few assignments of value, credences, and ways of updating our credences based on data.

Could you say more about why you think this is not the case? I know very little about the relevant psychology.

Some reasons why I think that it is the case:

Neuroscience and introspection, then, suggest that humans do automatically track probabilities with subjective credences. These processes mean that humans following decision theory would not begin with infinite regress–we already have some beliefs about actions we can take and their likely consequences (as well as some preferences over outcomes), and these offer a starting point for decision.

Given that decision theory does not lead to infinite regress, claiming that it only constrains preferences, given some other preferences, seems to sell it short. Given beliefs and preferences, decision theory constrains actions–given a mapping from actions to probabilistic outcomes, and a preference order over probabilistic outcomes, there is a rational preference order over actions. If I believe that it will probably rain, and I do not want to get drenched, then it is a good idea (by my own lights) to pack an umbrella.

I find “maximize expected value” appealing as a way of making decisions under uncertainty for at least three reasons:

  1. It offers a way to account for things I care about: the desirability of consequences, and their likelihood if I take some action
  2. It is demonstrably best in the situations where it is most feasible to judge which strategy is best: If our credences match objective probabilities exactly, then expected value equals average value over many trials. So anyone seeking to maximize total value over many trials would do best by maximizing expected value at each trial. This would still pretty much be the case if our credences closely approximated objective probabilities. As a result, any strategy that deviates from “maximize expected value” will do predictably worse than “maximize expected value,” at least in these circumstances. -. This is, admittedly, limited to situations that involve many trials. But that seems fine–it seems unwise to accept a general strategy that does not work well if used many times.
    • I don’t know how to judge which strategy is best in other circumstance: when my subjective credences deviate significantly from objective probabilities. That seems like asking “what would be the case if I’m just terrible at figuring out what is the case?” If I’m terrible at figuring out what is the case, then how would I know?
  3. I know of no other general approach to making decisions with these or similarly appealing qualities

I’ve argued that humans have credences, that rationality constrains actions, and that maximizing expected value seems best. Next, I’ll argue that the weights we should use when making decisions should draw on more than just frequentist analysis, because there are other useful measures of reality, and there is empirical support for the relative accuracy of Bayesian forecasting. After that, I’ll address a few loose ends.

One could tally all the times one has burned their mouth drinking tea, but that is collecting measurements and switching back to objective probabilities, which we agree are legitimate. Calling them “subjective credences” doesn’t actually help us to decide anything.

Drawing on credences can help us make better decisions, because credences can be usefully informed by more than just datasets:

Carefully formed subjective credences have several advantages over probabilities informed only by frequentist statistics–they account for a wide range of predictively useful considerations: causal dynamics, arguments, and datasets that are not explicitly accessible.

Do our subjective credences track these measures of reality well enough to be useful? It seems that way (especially when people are careful to mitigate the inappropriate application of heuristics):

As Scott Alexander’s review of Superforecasters discusses, Philip Tetlock’s team of “superforecasters” probabilistically predicted future events (the details of which were unprecedented) with greater accuracy (closeness to objective probabilities) than chance, and with greater accuracy than CIA forecasters working with classified information. (And they didn’t just get lucky–the same people tended to be superforecasters over multiple rounds of testing.) Tetlock writes:

The superforecasters are a numerate bunch: many know about Bayes’ theorem and could deploy it if they felt it was worth the trouble. But they rarely crunch the numbers so explicitly. What matters far more to the superforecasters than Bayes’ theorem is Bayes’ core insight of gradually getting closer to the truth by constantly updating in proportion to the weight of the evidence. That’s true of Tim Minto [the top superforecaster]. He knows Bayes’ theorem, but he didn’t use it even once to make his hundreds of updated forecasts. And yet Minto appreciates the Bayesian spirit. “I think it is likely that I have a better intuitive grasp of Bayes’ theorem than most people,” he said, “even though if you asked me to write it down from memory I’d probably fail.” Minto is a Bayesian who does not use Bayes’ theorem. That paradoxical description applies to most superforecasters.

Assuming that Tetlock’s claim is supported by the thousands of data points he collected, frequentist analysis seems to provide support for the usefulness of Bayesian analysis. More generally, this is frequentist evidence for the capacity of people to make better-than-random forecasts of events without information about previous trials of the events.

Tetlock found that several other factors were correlated with accuracy, including well-informedness, intelligence, deliberation time, belief updating, training on forecasting skills, team-based forecasting, and questioning one’s initial intuitions. This looks pretty good for Toby Ord, and for thinking that such forecasts are far from probabilities “made up out of thin air.”

Note that the predictions that Tetlock’s forecasters made were unconditional probabilistic predictions. And the vast majority of forecasters did better than chance (not just the superforecasters, although they did especially better than chance). This is (frequentist) evidence against the claim that

there are no objective probabilities - no measurements from which to derive probabilities - when dealing with the unconditional future.

Other implications for Bayesian analysis:

You write that:

The probability calculus is so powerful because it can be used to learn about the world by abstracting and aggregating information from complex phenomena which are often too difficult to analyze by other means. But this always requires there to be a dataset - measurements of reality - behind these calculations. Without this, we are claiming knowledge where we don’t have it and this is dangerous.

I agree, and I think that this is an important point. I would add that, as I have argued, extensive datasets of highly relevant reference classes are not the only useful measures of reality–logical arguments, information about causal dynamics, somewhat-relevant reference classes, and datasets that can only be accessed by considering our subjective credences also measure reality, and our subjective credences (more or less) successfully account for these. Our subjective credences are erroneous enough that I’m happy to act in accordance with solid frequentist data when we have it. When we don’t, our subjective credences still offer useful measures of reality. Without these, we are acting as if we have ignorance where we don’t have it, and this, too, is dangerous.

…we don’t have a total order across explanations, so can’t coherently talk about “second best”, “third best”, etc

If explanations are particular claims about what the world is like, and if, as I have argued, we have partial credences in such claims, then we can coherently talk about second best explanations, and thus make use of them.

To address a few other points:

_ There is a result in epistemology which proves that certain events are not knowable in principle - namely those events which are causally dependent on future knowledge, which we by definition don’t have_ […] It’s important to recognize that this result only applies to unconditional historical predictions (“X will happen”.) but not conditional ones (i.e. “If X happens, then Y will happen”). Relevant to our discussion, it also applies to probabilistic unconditional predictions (“X has a Y% chance of happening”), for to assign probabilities would require us to put a distribution over a set of events we do not know.

I like this argument–very neat result. Your last step here, going from ruling out definite predictions to ruling out probabilistic forecasts, doesn’t make sense to me. We cannot predict all (or much) of how things will go in the future, but we are not completely clueless either–we can still make useful probabilistic forecasts, on the basis of aspects of the future about which we are not clueless.

One could try to get out of this by switching to a subjectivistist view probability, and rely instead on the “credences of experts”. But this is arbitrary. How do we select which experts to listen to? How can we ignore experts in theology…?

I agree that using expert credences requires us to make choices. Why would it follow that these choices must be arbitrary? A few ways in which the choice of which experts to listen to may be made non-arbitrarily:

All we can ever do is try to criticize the best explanation through arguments and experiments, and try to come up with better ones.

And then how do you make decisions? I’m still confused about what the decision-making alternative to [something that’s roughly expected-value-maximization] is.

Update

(Added a few days later)

I came across some material that surprised me, so I want to add a correction to my response:

Reading a little more Tetlock, it looks like he makes a major caveat that I hadn’t been aware of: “there is no evidence that geopolitical or economic forecasters can predict anything ten years out beyond the excruciatingly obvious… These limits on predictability are the predictable results of the butterfly dynamics of nonlinear systems. In my EPJ research, the accuracy of expert predictions declined toward chance five years out.”

While this finding is not directly about technological forecasts, it’s suggestive that my previous optimism about them doesn’t hold up at all. I’m not sure what to make of this. Maybe:

Expected-value maximization still makes sense, although now the best way of going about it for long-term predictions involves overriding one’s automatic/intuitive formation of credences, and giving all non-negligible possibilities equal weight, except when one has very strong reasons to move away from equal weights.

Starting with a prior probability of 50% makes sense if one is completely clueless. Using the broadest reference class–all possible predictions–half of all possible predictions are correct, as for every possible prediction that is correct (“A will happen”), there is a possible prediction that is incorrect (“A will not happen”). (This feels sketchy.)

Given quite strong evidence, one should adjust one’s estimate significantly (since Tetlock does grant that people can predict “the excruciatingly obvious”):