Does it matter what year assessment test was normed?

[journey00]

I gave my ds the Cat test which was normed in 1986.  Should I be giving my kids a more updated assessment test? How can I be sure if what they're learning will be accurately assessed by any of these? Newer is better?

 

  IOWA
Form A
CAT
Survey
TerraNova
2nd Edition
Stanford 10
Online
Year Test Was Normed 2005 1986 2005 2007

 

[EKS]

The older the test, the higher (on average) a kid will score compared to the norm group.  This means that the percentiles will be higher than they would be if the student had taken a more recently normed test.

 

So, newer isn't necessarily better, but it is more accurate in terms of comparison to a same age/grade peer group.

[luuknam]

From what I gather, people seem to think the IOWA is a better test than the CAT/TerraNova. You need a bachelor's degree to administer the IOWA though, which is why we do the TerraNova. More recently normed tests are going to be more accurate in telling you how your kid compares to his peers, as opposed to his (grand)parents when they were his age (obviously, the 1986 version is not at the grandparent level, but there is at least one place that still offers the 1970 CAT, which is from way before I was even born). Also, the older a test is, the more likely it is to have unfamiliar things in it - I don't want a test where my 6yo is going to miss a question because it has for example a picture of a rotary phone instead of a cell phone and the kid doesn't know that it's a phone, know what I mean? So, I don't think it's super important to get the most recent test in existence, but I do think that at some point tests do get too old to be much more than curiosities.

[aaplank]

The older the test, the higher (on average) a kid will score compared to the norm group. This means that the percentiles will be higher than they would be if the student had taken a more recently normed test.

 

So, newer isn't necessarily better, but it is more accurate in terms of comparison to a same age/grade peer group.

Could you please explain why this is the case? I don't understand why. Thanks for your help.

[luuknam]

Could you please explain why this is the case? I don't understand why. Thanks for your help.

 

 

I know it's the case for IQ tests (Flynn effect). I haven't seen evidence that it's also true for standardized tests like the CAT, but I haven't seen evidence to the contrary either, so I'm curious too. I do know that now everyone is supposed to be on the college prep track, but I'm not sure that the average student actually performs better than a few decades ago. I know that the site that offers the 1970 CAT claims the opposite to be true - that the test from then is harder - but I don't trust that at all. It might also depend on which part of the bell curve you look at. Another difference between a few decades ago and now is that there are better supports for kids with disabilities, so the bottom might score higher now. And the top is more obsessed with the rat race into the most elite colleges, so they might be working harder and scoring higher too. Who knows (I'm sure someone does).

[nixpix5]

It is important to only used tests that have recent norms. It is actually considered not best practices to use outdated testing material. In fact a therapist can be breaking the code of ethics 9.08 by using outdated normed testing. This is due to a number of factors. One having already been mentioned which is questions and samples used will be dated and unfamiliar to current children. Another reason is known as the ceiling effect and another called the flynn effect. This is more complicated to explain but if you google these effects and testing you can read about it. Over time due to changing the way children are educated, exposure to educational components that were not as common long ago etc kids will do better on older tests over time. It is key to keep the testing within the zeitgeist of the Era for the child. Today's children have more practice with certain types of questions and tests therefore they will score better. Over time questions must be continually updated to account for this. Kids in the 70s would have had less overall testing exposure and therefore they would have scored lower as a cohort. This means today's child would receive a much higher score on that test in general. It does not accurately reflect a comparison for that child among his current peer group. I am probably failing at explaining this as it is early and I need my coffee :)

Edited May 13, 2017 by nixpix5

[EKS]

Could you please explain why this is the case? I don't understand why. Thanks for your help.

 

There are lots of theories, but no one knows why it happens.  In the IQ testing world, it's called the Flynn effect.

[EKS]

I know it's the case for IQ tests (Flynn effect). I haven't seen evidence that it's also true for standardized tests like the CAT, but I haven't seen evidence to the contrary either, so I'm curious too. 

 

You can see it if you look at school test performance over time.  The average scores will start out relatively low, and then get higher and higher each year until the school changes tests and then the scores go back down to where they were at first.  And the cycle repeats itself.

 

One interpretation of this phenomenon is that teachers/test prep centers are teaching to the test, either directly or indirectly.

[luuknam]

You can see it if you look at school test performance over time.  The average scores will start out relatively low, and then get higher and higher each year until the school changes tests and then the scores go back down to where they were at first.  And the cycle repeats itself.

 

One interpretation of this phenomenon is that teachers/test prep centers are teaching to the test, either directly or indirectly.

 

 

But the tests aren't normed on schools who have given that test for years and are teaching to the test, afaik. It's a 'new' test to them. Now, most standardized tests do have a fair amount of similarity, so I agree that kids (and teachers) likely have more experience with tests like that in general and so may very well do better than previous generations. But improved scores on one specific test over years of a school using it doesn't prove that kids now would score higher on the 1970 or 1986 CAT than kids in 1970 or 1986. If I'd given my kids the 1970 CAT instead of the 2005 CAT/TerraNova, it would've been just as new a test to me and the kids (I'm not from the US, so I've definitely never taken  any of the CATs), so there wouldn't have been the same effect as in a school where teachers give the same test year after year after year.

 

I still agree that you'd want to use a somewhat recently normed test if you want a meaningful comparison to the kid's peers.

 

ETA: and when I was reading up on the Flynn effect (for IQ tests) recently, I came across some stuff that in some first world countries scores have been leveling off since the 1990s, and in one or two subtests even have been dropping slightly (something with mental math, iirc). So, it's probably fine to use a test from anytime this century (though at some point in the future you'll run into the 21st century version of the rotary phone problem, of course).

Edited May 13, 2017 by luuknam