Monday 6 August 2012

How should we measure general intelligence using IQ tests?


General intelligence (g) is a construct used to explain that (in group studies) each and all cognitive abilities are co-correlated - being good at one implies being good at all the others. The hypothesis is that this co-correlation of abilities is due to a single underlying ability of general intelligence or g, with specific abilities (having various levels) on top of g.

Since g cannot be measured directly, IQ is derived from measuring cognitive abilities and putting people into rank order for ability - for instance, measuring one, several or a lot of cognitive abilities in 100 people, marking the test, then putting the 100 people into rank order (best to worst marks) - highest to lowest IQ.

(The validity of IQ testing comes from the fact (and it is a fact) that the rank order on the IQ test is statistically highly significantly correlated with a wide range of outcomes including exam performance, job performance, health and life expectancy.)


So IQ is ultimately a matter of rank order in tests.

The actual IQ score a person gets comes from a statistical manipulation of the rank order data, to make the distribution into a 'normal' or Gaussian curve, and the average score of a 'representative' population into 100 with (usually) a standard deviation of 15.

This is the 'standard curve' of IQ, since it is the standard against which individuals are measured.

The standard curve is constructed such that it describes the proportion of people that would get a particular IQ score - for example, an IQ of 115 is one standard deviation above the average and therefore about 16 percent of the population would have an IQ of 115 or above.


But there are difficulties in generating an IQ score for individual people, and in moving between the rank order data generated in a group study (used to generate the 'standard curve') and the score of an individual person doing an IQ test.


The individual score in an IQ test ought to be measuring a fundamental property of human ability (a property of the brain, roughly speaking).

Yet many or most IQ tests in practice require non-g abilities such as good eyesight, the ability to read, ability to move hands and fingers quickly and accurately; they require concentration (that a person not be distracted by pain or other interferences), many tests require stamina, a degree of motivation and conscientiousness in completing it... and so on.

In other words there are a range of non-g related factors which might reduce the test score for non-g reasons.

This means that the most valid measurement of intelligence is the highest measure of intelligence in a person.

So the best way to measure intelligence is for a person to do a series of IQ tests on different occasions and to take the highest score as the true-est score.


BUT this must also apply to the standard curve used to generate the IQ score.

The standard curve must be constructed from the highest IQ score of (say) 100 randomly chosen people - and these highest scores put into rank order and made into a normally distributed curve with the correct properties.


Yet this is not what happens.

The standard curve is typically generated using a one-off test on the representative sample, but the individual IQ is derived from the best performance in an IQ test - this systematically biases individual IQ scores towards being higher than they really are.


Of course, there are great logistical difficulties in using multiple tests (on several occasions) and best performances to generate a standard curve - much easier to get a representative group together just once for testing.

But this emphasizes the imprecision of individual measures of IQ.

If an individual gets their IQ score from a single test, it is likely to underestimate their real g, if the test is done in a way or at a time when their performance is impaired.

Yet if the individual has several tries at IQ test on different occasions, in order that their best possible level of performance be used to generate their real, underlying g, then this will overestimate their IQ.

(Doing several tests and taking an average does not work, because the bad performances drag-down the average.)


So, in practice and as things are - I do not feel that individual, one-off personal IQ measurements can be regarded as precise.

Probably individual IQ should be banded into roughly half-standard deviations.

Something like average as 96-104, above average as 105-114, high as 115-124, (above this 'g' begins to break-down as the component tests lose co-correlation) very high as 125-140, and above that we have the super-high and strange world of potential geniuses.

(Below average would probably be a mirror of this - but the meaning of low IQ is a bit more variable, and the levels may be very low.)

But IQ differences between individuals of less than half an SD (less than about 7 or 8) are uninterpretable - even around the average. 



dearieme said...

I once commented at Steve Sailer's blog that I had twice taken IQ tests but did not know my scores. I was flooded with suggestions about how to take new tests. But why should someone of about sixty years (as I then was) bother with an IQ test?

Bruce Charlton said...

@dearieme - no reason, except curiosity.

I made the mistake of doing a Mensa IQ test with my wife a few years ago, in which she *slightly* out-scored me on one of the two main tests - a result which I have had cause bitterly to regret on numerous occasions since... ;-).

But as Steve Sailer, I think, has clearly pointed-out, the main value of IQ tests are to predict group differences, at which they are highly reliable - and therefore they can (and should) be used as one of the most valid selection methods.

Thus IQ tests were honed by educationalists involved in selecting kids for grammar school (eleven-plus) education in the UK, and college (18 plus) education in the USA.

But IQ tests also have a vital 'clinical' role in evaluating individuals whose performance (e.g. at school) has big dissociations - for example where their maths is much better (or worse) than their reading ability - or where their verbal ability is not reflected in written exam results.

If we had not lost understanding of intelligence in the past 50 years (or rather, if this understanding had not been supressed and subverted by the Left) then we would find some huge discrepancies between performance in modern systems of educational evaluation (so-called 'exams' such as GCSE and A-levels, or most undergraduate degrees) and the innate general intelligence as measured by a valid IQ test.

Interestingly, so profoundly has the understanding of IQ been lost, that highly selective universities such as Cambridge use unvalidated (and invalid) subject-specific tests to select e.g. medical and law students, when they ought to be using proper IQ tests - which were specifically designed for the purpose of picking out the best academic prospects with the minimum of interference from differences in schooling, social class, sex etc.

(Ah yes - I hear you say - but they don't *really* want the best academic prospects, do they? And I would agree, no they don't. And certainly, they are not getting them!)

dearieme said...

they don't *really* want the best academic prospects, do they?

When I did admissions "they" bloody well did. We had to teach the buggers, so bright enthusiasts were what we wanted, and other criteria trailed in far behind.

Many subjects are structured, so that it also mattered that those admitted knew enough to be able to engage with their course. But our questioning in interviews was designed to see what they could do with knowledge, rather than assess how much knowledge they had.

Bruce Charlton said...

What *you* did (and in your specific subject) is one thing - but there is no doubt about the general pattern.

As of 2005 there were equal proportions of men and women at Cambridge (well, 51 percent men...)

which means that 'they' (the university as a whole) certainly were NOT selecting the brightest; because there is a much higher percentage of men than women of very high intelligence for reasons I explain here:

Insofar as you were selecting the brightest, others must have been even more biased-against intelligence in order to make-up-for it...

dearieme said...

Admissions weren't done by the university - they were done by each college, subject by subject. As someone interviewing for one college in a subset of Natural Sciences - where a finite number of places were available to be filled - I had no influence at on who was admitted into that college in other subjects, or into another college in any subject.

If the best of the failing candidates in my set of interviewees was much cleverer than the worst of admitted candidates in some other subject, that had simply to be tholed. You could make a case for being assigned an extra place to fill, but your colleagues in other subjects would be reluctant to surrender one.

I dare say that things have changed a bit since I was last involved, but I'm confident that it remains impossible usefully to generalise about admissions unless you understand how it is organised.

You can't of course do admissions purely by IQ, because many subjects have a structure such that you need to bring adequate preparation from school. Others need less - I doubt whether Law needs any more than a command of English and a decent IQ. That combo alone, without your having learnt some science, would see you sink in most of the scientific subjects.
If you did it purely by IQ (plus suitable preparation) you might end up with armies of students in Maths, Engineering, Medicine and the more physical parts of Natural Sciences, and a dearth elsewhere. It's not obvious that that would be a good thing. (Certainly not to teachers of "elsewhere".) It might answer your complaint about the lassies though.

Bruce Charlton said...

@dearieme - "your complaint about the lassies though"

No complaint, just a fact! Or rather, I do complain that elite colleges try to have it both ways - they like it to be thought that they are a bunch of super-intelligent types, yet they select super-industrious types.

Now, super-industrious types (of decent intelligence) may be just what is needed - for example in my subject of medicine and yours of engineering - but colleges can't have it both ways - that's just plain dishonest!

As for knowledge versus intelligence - the olden English system of specialized degrees built upon O- and A-level knowledge has almost disappeared outside of a tiny minority of mathematical subjects - in most subjects there is no assumption of prior knowledge - degrees teach from 'ground zero' (or, at least, it is very easy for students to 'catch up' on what is missing).

This means that prior study is nowadays all-but irrelevant for most undergraduates at even the, most selective universities. (It should not be, but it is).

The other problem for top universities has been selecting among scores of applicants all of whom have identical qualifications of three grade-A A-levels.

For this purpose (when everybody has almost identical 'knowledge' and sufficient conscientiousness), IQ tests are clearly the best predictors of future performance - and other types of selection have never been shown to do much more than weed-out the frankly psychotic.

dearieme said...

As it happens, my admissions experience at Cambridge was in NatSci, not Engineering, but I doubt if that made much difference. (Though I did have a pal who said that there was quite a difference between the youngster who said "I'm very good at physics therefore I must do a physics degree, and the youngster who said "I'm very good at physics therefore I'll choose to do an engineering degree.)

I don't remember that anyone I interviewed had qualifications of three grade-A A-levels, although I'll bet they were all predicted to get those when they took their exams. (Therein lies one of the lunacies of the English system of university admissions.)

Personally, I'd have been delighted had we had the balls to set IQ tests but that was not within my powers. Similarly, It's my belief that comparing people on the results of A-level exams that are far too easy for them is a nonsense. It would have been like selecting people in my day based on their O-levels. Bloody silly: O-level examiners weren't on the outlook for unusually clever youngsters; that wasn't the point of O-levels. Still, far more people are predicted to get 3As than we can ever admit, so we ended up interviewing in hopes that we could distinguish between those who had merely mugged up lots of science and those who showed indications that they could think about, and with, science.

"outside of a tiny minority of mathematical subjects": but those are what interest me. Of admission in, say, the Arts, I cannot speak. Except that when a young chum of mine was interviewed and tested for admission to Philosophy (plus something or other) at Oxford, he was convinced that what he took - a test overwhelmingly about logical reasoning - must have been somewhat similar to those ancient IQ tests of which he had vaguely heard. And his interview was on similar lines: could he reason? Bully for Oxford, say I. I understand that Cambridge increasingly relies on tests: too recently to have been any help to me. (Presumably we have to test for IQ while having plausible deniability to the politicians?)

At least our interviewers' incentives were clear: we would have to teach the people we admitted, so we really would like to find people with a bit of spark to them. Of course, when you come to mark their Tripos exam papers (and there I have experience both in NatSci and Engineering), you learn that quite a few are much better at mugging up than they are at using material.

I think the greatest flaw in the classical English undergraduate admission system is the ambition to fail nobody - the mad belief that at application you can accurately choose people who are guaranteed good enough to pass all their exams and graduate on schedule with the degree they sought. I prefer the old Scottish system - be much more comprehensive at entry and chuck 'em out if they can't cope. Then you can take gambles. When I've tried to explain the merits of this idea to the Cambridge-bred, they've been appalled. In fact, one Senior Tutor made the remarkable claim to me that she had never made a mistake at admissions. About lunatic colleagues one can do little.

I pointed out that our tenure system was essentially "Scottish" - we could appoint to Assistant Lectureships and then dump the appointee if he didn't come up to scratch. And if he passed that scrutiny, he would have a second exposure to dumping part way through his Lectureship. No dice: 'that's quite different' I'd be told. Mad, mad, mad.

I'm guessing, but my instinct is that it's more likely to be the postgrad and postdoc experience in science that really would grind down people of spirit. It is not unknown in Cambridge for academics to comment that it's more fun dealing with the bright sparks in final year undergraduate classes than with postgraduates.

dearieme said...

Here's a suggestion: there must be some online material at the Oxford and Cambridge websites to help applicants to prepare for application "tests". Why don't you take a shufti and use your expertise to assess whether these tests are IQ-test-like? I'd love to know your opinion.

Bruce Charlton said...

I looked at these -

Thinking Skills Assessment -

Verdict - a badly-designed IQ test


Bio-Medical Admissions Test

Verdict - a *very* badly-designed IQ test.

This is absolute rubbish. Quite honestly, aptitude testing had got much further than this 100 years ago.

It would probably be a more accurate measure of intelligence simply to test reaction times, or do an MRI scan and measure brain volume, or measure hat size (not really, that is hyperbole...).

The interesting thing is that there are a number of politically correct themes interspersed - that is very revealing of the real motivation of the test setters (as well as their incompetence).

FHL said...

Some random thoughts (broken into two different messages):

1. With regards to finding an accurate individual score, would it be best to take several tests and drop the lowest scores? Say, take 5 tests and drop the lowest two? Or perhaps: take several tests, say 10, average the score, then drop any that are significantly above or below the average, then average the remaining tests again to get the score?

I've never taken a statistics course, so perhaps I am mistaken, but I think this would lead to a better assessment of individual IQ scores.

I've always been afraid to take an IQ test for this reason, thinking that I may have an off-day, score too low and have people think me less than my actual abilities, or have a lucky day, score too high and disappoint.

2. I think there needs to be more discrimination when people try to arrange groups for IQ purposes (or any other purpose, for that matter).

I've always thought that the average intelligence of Copts in Egypt was different from that of Muslims (although I do NOT think that this is why we are Christian and they are not), since we are two different racial groups. If IQ is an indicator future exam performance and the generation of wealth, then the reverse is true. Copts have the highest grades nationwide (in the U.S.) out of all the immigrant groups, even compared to stereotypically intelligent groups like the Jews. (I've never actually seen this report, and it was told to me by a Coptic priest, but straight A students and wealthy families are a regular occurrence in my local Coptic church here in Texas, so it is believable to me).

But everyone is lumped together as "Egyptian" in the IQ results that I've seen.

3. Now here's a thought: it may be possible that by failing to discriminate between two dissimilar groups that you would end up with a completely fictional score.

(I am not referring to Egypt or any specific place in the following hypothetical example)

Assume two separate groups of people exist that have roughly equal numbers of people. Assume one group has an average of 60 and another has an average of 100. If you mistook these two groups as one group you would get an average of 80.

So the results would say that "such as such" people (who are not a cohesive people, but you mistook them to be) have an average IQ of 80, so when you meet someone from that group, you can expect them to have an IQ of 80. But in reality, very few of them, if any, have an IQ of 80.

4. I've never believed that being good at math meant you were bad at the humanities, or vice-versa. When I first heard this as a child, I said “Screw that, I plan to be good at everything!”

My aunt once complementing me telling me I was very smart, I replied that some people are smart in some things, while others are smart in other things. She also had a reply: “No, the smart person is smart in all things, and you know that's true.”

But there are some things, the most important things in life, that do not depend on intelligence, which brings me to my next point...

FHL said...

post 2 out of 3

5. "(The validity of IQ testing comes from the fact (and it is a fact) that the rank order on the IQ test is statistically highly significantly correlated with a wide range of outcomes including exam performance, job performance, health and life expectancy.)"

I always laugh a very bitter laugh when I hear this. I've never taken an IQ test, but I'm quite certain I am more intelligent than most. I had the best grades, I beat everyone in my class on both the ACT and SAT (I went to a Catholic high school known for its high standards, had about 100 people in my class, and I was the only non-white student), represented my school at math and debate tournaments, completed science and literature projects that got me placed in the local newspaper, I received glowing recommendations from my teachers and employers, and won many awards.

I used all that to enter a private Christian university with a full scholarship (they gave it to me almost automatically for my ACT score) and took the hardest classes and was regarded as one of the brightest

And then I broke. In my sophomore year I had a complete psychological breakdown in which I stopped attending and failed all of my classes, experimented with drugs, went completely broke, as well a whole host of nonsense and sinful behaviors that I have yet to fully recover from.

I was taken to a psychologist who never got anywhere with me, besides diagnosing me the vague term “major depression,” so I stopped going, and most people thought I was just going to "curse God and die." So people gave up trying to help me.

But I did make a long and shaky recovery and am much better now (although not completely, I'm not sure if I ever will be...).

But now no-one looks at me as the smart guy anymore. I've been branded as the screw-up, my professors look at me with a morbid curiosity, wondering why I'm even still taking classes, and people are afraid to even include me in their study groups, thinking my very presence will somehow disrupt their efforts. My job prospects have been ruined, my GPA is tragic, and I've burnt all of my bridges.

FHL said...

post 3 out of 3

But I will say that it was during this time I started taking Christ seriously. It was during this time I read Augustine, Athanasius, C.S. Lewis, the books of the Old Testament, and many, many essays on various topics from Christians of all sorts, from the Church fathers to Alvin Platinga. I was also shaken free from my liberal views, as well as any views I had belonging to the "alt-right" or whatever-you-call-it (stuff like Game, which I thought was awesome during my high-school and early college years).

I became obsessed with it. Sometimes I would start telling a friend about something St. Augustine wrote and then suddenly break out into tears. I'm a man, so of course this was embarrassing and awkward. They would ask "why do you care so much about this stuff?" to which I could only reply "It's the only thing that's real, everything else in this life is fake, God's the only thing that's real, like really real!" They never understood, and I was never able to explain further

I was also during this time that I actually prayed, like really honestly prayed, not just the quick “give me an A on the test, please” that resembles a superstition more than a prayer.

And so I count my breakdown as a blessing from God, a warning of sorts. I also needed it to humble me. I used to be exceedingly clever (indeed, I think most of my successes could be attributed to simply being clever), much like the people whom you call "clever sillies." And although I still am more than likely too clever for my own good, I began to realize the vanity behind it all.

I'm not really sure why I felt like sharing that story, I've never shared it before (at least not like that, preferring only to say "I had some problems"). Perhaps it's because you're one of the very few people who might understand what I meant when I said that God was the only thing was real.

dearieme said...

Oh dear, oh dear.

At the risk of repeating myself, I'll tell you about a conversation with a mathematical friend who'd been pressed into service to do admissions in Computer Science.

HIM: How should I choose whom to admit?

ME: Reverse the classical test: chuck a ball and admit everyone who drops it.

HIM: That's no good, they'd all drop it.

Bruce Charlton said...

@dearieme - aside from being of poor validity, the Cantab tests looked much too easy. By contrast the Mensa test I took (by Cattell) began fairly straightforwardly but got more and more difficult until it was impossible (at least to me - I couldn't even imagine how to answer some of the questions! Of course some people can get perfect scores on the Mensa tests which is why there are even more selective IQ societies) - all done under a tyrannical pressure of time.

Exhausting (at least, for a fifty year old).

FHL said...

"none of these address the problem - it is an intrinsic limitation that the standard curve for IQ must be generated in the same way as the individuals who are measured against it."

Shoot, I forgot about that!

"of course IQ is only one measure of ability - nobody claims otherwise. But because it is more precisely measured and well validated, its importance may be exaggerated (where it is not completely denied!)"

This is true.

However, nobody claims otherwise, but the modern world believes otherwise. Hence the opposition to IQ testing- if a higher IQ makes you smarter, wealthier, healthier, and taller and more attractive, well damn, what else is left for moderns?

Also, you yourself thought to say the same thing-

"I suppose it depends what is meant by 'true' in 'only true men' - but for a Christian, genius is of secondary importance.

Genius is orthogonal to, independent of, salvation which certainly is not restricted to men of genius!

(Of course we cannot know for sure, but...) If anything, the statistical relationship between IQ and salvation would be inverse and negative rather than positive."

I merely meant to confirm the same thought.

I had read that article "the Outsiders" when you first posted it. I certainly did not know Latin at age two (and I still don't know it), and I don't think I'm a genius.

But I can see the similarities. When I went to the psychologist several years ago, she asked me what I wanted to do for a job. I told her "anything that does not require thinking, I think too much already and wish I could stop. I can't go to sleep because I can't stop thinking about stuff. I wish there was an off button on my head I could push. I want a simple job- like placing books in alphabetical order. Or stacking boxes. I could stack boxes all day. I would love that job."

My parents would get infuriated when I told them things like that, telling me I wasting my natural talents.

FHL said...

And this, from The Outsiders article, times a million: "The very gifted child or adolescent, perceiving the illogical conduct of those in charge of his affairs, may turn rebellious against all authority and fall into a condition of negative suggestibility--a most unfortunate trend of personality, since the person is then unable to take a cooperative attitude toward authority. A person who is highly suggestible in a negative direction is as much in bondage to others around him as is the person who is positively suggestible. The social value of the person is seriously impaired in either case. The gifted are not likely to fall victims to positive suggestion but many of them develop negativism to a conspicuous degree."

I told one of my friends a few years back that I realized the only reason I do anything at all was because I liked to prove people wrong. I eventually was able to humble myself, but it seems like I have two modes: either I submit to an authority completely, and attempt to shoo all thoughts from my mind. Or I decide to "figure things out for myself," which almost always results in me rebelling against everyone and all viewpoints. So far, the Orthodox church is what keeps me check. When I hear a bishop make a statement, I get into the mode of "ok, this is a Holy message, don't try to dispute it, the bishops of the church wouldn't try to trick you, just accept it and figure it into your worldview somehow."

Lemniscate said...

Here is the data for Oxford mathematics and statistics admissions for 2011 entrance:

Male: 123
Female: 50
% Male: 71

This does not seem terribly inconsistent with selecting on ability. Looking at the admissions test, it looks like a harder version of A level mathematics with more capacity for independent mathematical reasoning required. This is surely about as good a proxy for at least the mathematically correlated aspects of g as one could ask for. For computer science and physics, which share similar admissions procedures, the male %s are 80 and 84.

In English literature, the acceptances were only 41% for males, but that does not seem incompatible with selecting on ability. In fact, the male success rate was higher (23.9% for males versus 17% for females). The male success rate for mathematics was also higher: 17.4% versus 11.4%. Oxford as of yet does not appear to have too much difficulty giving a higher acceptance rate to males.

It's the human sciences and the arts and humanities that change the sex ratio to near balance. Is this indicative of not selecting on IQ? I doubt it, as the male advantage is more in the mathematical domain, and many more females are attracted to softer subjects, which are also inherently less g-loaded.

Bruce Charlton said...

@L - If Oxford was selecting from the top few percent of ability, there would be at least 2:1 M:F in the university as a whole, but even more if you are selecting on ability and motivation to go beyond the undergraduate level.

Maths is, until now, probably the least corrupt of academic subjects - even so I would have expected a higher ratio of men given that it is as near certain as anything is in such fields than men are much better than women at maths at the highest levels.

But although selection procedures are biased against native white men - and this is no secret but indeed a high profile top level and explicit aspect of public policy, monitored and subject to sanctions - much of the damage to selectivity is done at the school 'exam' level; and again, not accidentally.

Elijah Armstrong said...

Was this Mensa test the cattell verbal or the cattell culture fair? The CCF has an EXTREMELY high ceiling - IQ 183.

Bruce Charlton said...

@EA - The Mensa test I did had two components - a short Cattell culture fair test (about 20 minutes?) - non-verbal, using symbols; and a much longer multi-subset Cattell test done in segments with rests between. As I recall this second one had a lot more verbal IQ components, and I did approx 1 SD better on it!

The CCF may have a high ceiling qua test, but several SDs from the mean what is being measured is not g (it is something important, but not g) - I suspect that g in the sense of general, underlying intelligence) with all its predictive qualities) is something that has a plateau, probably at about 130-145 in modern measurement.

This plateau in g would be related to maximum processing speed in a biological system having certain properties.

Once up against this plateau in processing speed, then gains in one aspect of intelligence will be only at the expense of other abilities - either cognitive or perhaps physical: a trade-off, in other words.

I think this is confirmed by the attributes of people with ultra high IQ - it is not just a question of them having quantitatively more of the same kind of thing as people of IQ 130, but of different cognitive ability/ attribute profiles - a qualitative difference, in other words.

Theramster said...
This comment has been removed by the author.
Bruce Charlton said...

TNP - IQ tests don't measure creativity - general intelligence is essentially independent of creativity in the sense that most highly intelligence tpeople are not creative while most creative people aren't highly intelligent.

But a genius is the combination of high intelligence and high creativity (plus other things - mostly internal high motivation).

I don't think that real creativity can be measured in a 'test'; although variables that are somewhat correlated with it can be measured, to some extent.

Note: Personality questionnaires are not 'tests', they are rating scales.

Theramster said...

Maybe we need to look at it taking the 'Genius' as our standard. In the Genius two elements that seem to be at odds in the general population are biologically and consciously a living synthesis or even an intertwined and inseparable wiring. This is why it's difficult to predict it or recognize it from the general perspective. And if recognized because of its oddity, its difficult to deal with because it's rather a different cognitive mode.

Here two elements are distilled out and spoken of as separate while in the types that we truly desire to have more of in order to have a better life are integrated.

My idea is that this type of integral intelligence or creative intelligence is ruled out by the restriction imposed by complex IQ questions that will require a single answer. That conformity to a certain restricted cognition is a prerequisite for such integral intelligence to score higher and be recognized. It will take a certain level of learning on their part, which they are perfectly capable of doing.

Forgive my careless musings about the topic and thank you for allowing us to learn and discuss such fascinating topics here.