Americans May not Be Bad at Math but Some Journalists Sure Are

It’s that time of year again when we hear complaints about how terrible the U.S. is doing in math. This article by The Atlantic with the title American Schools vs. the World: Expensive, Unequal, Bad at Math is just one of many, many reports that showed up in my twitter stream.

The first question anyone who has had more than a week of statistics should ask is about the sampling. Didn’t anyone notice that the top “country” is not a country at all but a city. The article goes on to say “parts of China, Japan, Korea and Liechtenstein topped the ratings… ”

Wait, what? PARTS of China?

As a statistician, I am intensely interested in just how representative these parts of China might be. So, I track this down in another article, since many of those in the U.S. didn’t bother to investigate, I tracked this information on the demographics of the sample down from an article written for The Guardian by a correspondent in Beijing,

Its population is less than 2% of the country’s total, and its per capita GDP is more than twice the national average. According to Tom Loveless, an expert on education policy at Harvard University, 84% of its high school graduates enroll in college, compared with 24% nationwide.

I’m not saying that U.S. education could not use improvement. (Although the more I work with the Common Core Standards for mathematics, the more I’m convinced they have been over-hyped. ) What I am saying is that calls for improvement should be based on well-reasoned arguments that have some understanding of science – random sampling being a good start. It is also worth noting that the same article in The Guardian mentioned that 12 provinces in China had students take the test but only the results for one CITY were released.

There were also several articles that discussed Finland “slipping”, math scores 7 points, from 548 to 541.  A lot of potential explanations and remedies were given in this Washington Post blog, sadly, none of which mentioned regression towards the mean, sampling error or the nature of the test itself. Also, there is the question of how much in absolute terms that difference really means. Children don’t answer 600 questions. They’ll answer a test with 50 or 60 questions which will then be normed to, say, a mean of 500 with a standard deviation of 100. That drop may well represent that the average child answered 43.7 questions correctly last year and 43.2 questions correctly this year.

Then, there are the host of articles that went on about how the sky is falling because countries that have low PISA scores are also those who do poorly economically and therefore China is going to eat our lunch. Don’t even get me started on correlation and causation.

All of this has led me to conclude that the PISA data are unclear on whether or not American (and Finnish) children are really doing that terribly in math, but have led to the firm conclusion that most journalists’ articles that I read could certainly use a refresher in statistics. (See how I did not generalize to all journalists in the entire world? Take note.)

In fairness, the very same Washington Post had another blog on how public opinion is being manipulated  using the PISA findings. Is it or is it not an outlier? Discuss.