I'm just gonna quote what I said when this was on Metafilter a few months back:

"Anyway, yes, the fundamental problem in these questions is not how people think -- it's that the question they want the respondent to think they're asking, and the question they claim they're actually asking, are two different things.

The lawyer/engineer one is a classic example of this; what they hope is that you will read it as "how likely is it that these personality traits correlate to an engineer", so that they can then swoop in and say "what we were really asking is the mathematical definition of a percentage!"

Which ultimately tells us very little about the respondents and quite a bit about the people conducting the quiz..."

The author of this quiz seems to have completely misunderstood the relevant research. Either he has to assume that all individuals are identical (in which case, the little story is irrelevant) or he needs to apply Bayes rule according to the probabilities associated with the factors expressed in the personality exposition.

Either way, the article's explanation for that particular question is wrong.

Exactly. Here's the example used in Kahneman's book:

"Dick is a 30-year-old man. He is married with no children. A man of high ability and high motivation, he promises to be quite successful in his field. He is well liked by his colleagues.

This description was intended to convey no information relevant to the question of whether Dick is an engineer or a lawyer."

The description in the quiz is very different.

This still conveys a lot of information.

Engineering is more male-heavy than lawyering.

Your assuming a US bias, the sample could have come from India which has different ratios.

Yes, there's more to it. The experiment was done with 70/30 and 30/70 ratios for different subjects. The book doesn't say whether they specified they were all males, my guess would be that they did.

a frequentist would take issue with the two sons, one born on a tuesday problem. you can actually count up the permutations.

let's say we have 10 engineers, 9 of them are male. we also have 10 lawyers, 6 of them are male. Let's say one in 10 people likes doing math on the weekend.

There are 90 out of 100 ways to have a group of male engineers, one of which who likes math, but only 60 out of 100 ways to do the same with a male lawyer. furthermore, if we add in the four kids as another 1 in 10 thing, the situation gets even worse. This isn't bayesian, this is just counting boxes on a permutation table.

> This isn't bayesian, this is just counting boxes on a permutation table.

It's the same thing. Bayes' theorem allows you to shortcut straight to the answer without having to draw out a full probability tree / permutation table. But the underlying math is the same - in each case you have a different probability of B given A, versus B given (not A).

Personally I wouldn't call it "Bayesian" so much as just "a conditional probability." The question doesn't ask what the probability is that a randomly selected participant is an engineer, it asks what the probability is that a participant is an engineer given that he has "typical" engineer-like traits.

But then yes, ideally you could use Bayes' Rule to find that probability.

They didn't do a good job representing that particular heuristic. Here's a better example: http://en.wikipedia.org/wiki/Representativeness_heuristic#Th...

A better way to explain the common response to question #2 would be "quiz questions usually only give me information relavant to their answers" heuristic.

It's worse than that - the probability is biased as you say if the sample were chosen randomly, but they did not claim that.

Given that the only distinguishing characteristic mentioned is a boolean (engineer or lawyer), and that they only chose one sample, the probability might just as well be 50% - the probability of the person picking the sample deciding one way or the other.

30 engineers, 70 lawyers, the probability of being an engineer is 30%. What Jack likes to do is irrelevant. Why can't a lawyer like math and dislike politics (winning a case framed by certain rules is just a puzzle/game to hack)?

Here's an alternative story:

To the question "Are you an engineer?", Jack answered "Yes".

Would you still argue that the probability is 30% that he is an engineer? A lawyer can claim to be an engineer, after all. However I think it is clear that if we actually did the experiment, it would be much more likely than 30% that Jack is an engineer. A way of testing what you really believe the probability to be is this: I bet you a dollar that Jack is an engineer. If you wouldn't, that means that you really believe the probability to be larger than 50%.

This is because the probability that he answers yes to the question is much higher when he is in fact an engineer than when he is a lawyer. Bayes' law says:

    P(E|Y) = P(E) * P(Y|E)/P(Y)

You should read P(A|B) as "the probability that A is true given that B is true". In this case E = "a person is an engineer" and Y = "a person answers yes to the question 'are you an engineer?'". As you can see the original P(E) = 30% gets multiplied by P(Y|E)/P(Y) given the information that the person answered yes. The probability that a person answers yes given that he is an engineer is higher than the general probability that a person answers yes. So P(Y|E)/P(Y) > 1. So P(E|Y) > 30%.

This same law applies to other characteristics, for example Y = "person likes mathematics".

What Uhrrr is saying is that prob(Person is an engineer | Named Jack and has Jack's description and initial 30/70 ratio) > prob(Person is an engineer | 30/70 ratio)=30%. I think the article goofed on that question too. I'd rather have seen an illustration of the conjunction bias, which is similar. Given Jack's description, what's more probable: Jack is a lawyer, or Jack is a lawyer who likes classical music. (The first one, but people overwhelmingly pick the second one.)

It's not that a lawyer can't like maths or dislike politics. The issue is whether they are less likely to on average. From my personal experience of knowing several of both groups, I would say that on average the lawyers I know are less interested in maths than the engineers and more interested in politics than them. It doesn't apply universally (some of the engineers I know are obsessed by politics, just not all of them and not as many of them as lawyers). It's possible that there may be research that demonstrates this doesn't hold when you look at the sum total of lawyers and engineers, but that's not what the author is trying to rely on.

Assuming that there is a skew of preferences, then this info isn't irrelevant you can perfectly reasonably use this to help identify the likelihood of this person being in one group or another. It doesn't guarantee that you're right, but it will improve your chances.

I agree. In many areas of pop-culture we seem to have a lot of people trying to convince us that "We don't know what we know", often with hilarious results. Like the EU officials who ruled that "drinking water has not been shown to reduce dehydration."

The more I see this trend, the more stubbornly I find myself clinging to "What I know"

I think the point that many are missing is that is not known for certain that an engineer is more likely to enjoy certain hobbies over others. People use their personal experience to develop a heuristic which this test is designed to reveal.

Getting hung up over the specificity of the hobbies and interests and the likelihood of those hobbies and interests representing either or a lawyer or an engineer is irrelevant, because the only factual data that was provided by the questioner is that 30% of the participants were engineers, and 70% were lawyers.

It's not "getting hung up" about the specifics. They are relevant, and clearly deliberately chosen. You're right that we don't know this likelihood with absolute 100% certainty, but that doesn't mean we should dismiss our personal experience, and a bit of logic (maths is typically more useful for, and a more practical path into, engineering than law, and there's far more politicians in my country with a background in law than there are with a background in engineering) out of hand.

What this article is trying to present is heuristic errors - like question 1, where ignoring of the fact that sample size is relevant gets you to the wrong answer. Ignoring the likelihood that there is a correlation between personal interests and career choice seems to me to be the equivalent heuristic error for this question.

Let me give you an alternate example. There are roughly 700 million Europeans and roughly 300 million Americans. If I randomly picked one person out from this, gave you no other information and asked you where they came from, you'd have a 70% chance of guessing correctly by saying "Europe". If I told you that their first language was English, that they loved American football and baseball and hated soccer, and that their favourite TV show was Conan, and then asked you to guess where they came from, it would be hugely naive to ignore that information and still assume that they were probably European. Yes, it's entirely possible that there are Europeans who fall into all of those things, and I've not done a survey to find the exact percentage of each group that answer this description, but I'd be prepared to put a fair amount of money on the fact that there's a larger overall number of Americans who answer it than Europeans, so the smart guess would now be that they are American.

I think it still depends which school do you belong, Bayesian or frequentists. A real frequntist may not assign a probability to a single instance of society! he is either an engineer or not!

And the killer mistake in # 2 is that he "He shows no interest in political and social issues" Now Law is inherently political (both big and small P's) even more so in the USA where Judges etc are elected.

I would bet good money that hes more likely an engineer than a lawyer.