Know what the tests test

If I were in charge of the world, which, sadly, I am not, there would be a requirement that every statistical programmer be issued a pearl like Glinda the Good had, (the good witch in the Oz books). Glinda’s pearl was white if you were telling the truth and black if you were lying. I’d add another color, chartreuse (because it’s the ugliest color I can imagine) for if you had no idea what the hell you were talking about.

We’d all start college with our pearls chartreuse most of the time, but the pearls would eventually be white more often, as we figured things out.

I hear from so many young women, from the world’s most spoiled twelve-year-old to brilliant women doing their post-doctoral research,

“Oh, sure, statistics is easy for you because you’re so smart.”

Anything is easy after you’ve figured it out and done it a thousand times. I wrote all of this blog without looking at the keyboard and I’ll bet most people reading it could do the same, but that’s a skill that would be out of the reach of my granddaughter until she puts a little more practice in.

Just to give you one of my “chartreuse” moments, and hopefully spare you the same, let me tell you about my very first computer lab in graduate school at the University of California. I felt pretty confident about my programming skills, as I had worked as an engineer between my MBA and coming back to school for a Ph.D.

I had just picked up the results of my program at the computing center and was meeting with my advisor. I already knew how to write JCL and run SAS on a mainframe from my previous job and I had just done a multiple regression so I was feeling VERY pleased with myself and brilliant. I had used an LSMEANS statement to test for the difference between means.

Everything was significant. I was still chartreuse enough to believe that significant meant very important but I did comment to my adviser that I was a bit surprised all of the mean differences were significant, but hey, the experimental and control group were, which was the main point.

Very slowly and patiently, as if he were dealing with a particularly dull child, Dr. Eyman explained to me,

“Young lady, this first table doesn’t test if the means are significantly different from one another. It tests if the population mean is significantly different from zero, which we already knew it was not.”

One advantage I appreciate of being brown is that it is impossible for me to blush. I was especially grateful of that inability at that moment.

So, know what the tests test. If you don’t, and make a mistake, don’t worry about it, it’s not the end of the world. It doesn’t even mean people will think you are stupid. When I asked the professor how DID I get a post hoc test he shrugged and said,

I don’t know anything about computers. You’re the programmer. I’m sure you’ll figure it out.

Of course it took me about 30 seconds after that to realize that you need an option added to your LSMEANS statement to request differences, like this, where “effect” is replaced by the name of the variable for which you want to test the effect.

LSMEANS effect / DIFF ;

If you want the tests of mean differences to be adjusted by some method, say Tukey, you would add that

LSMEANS effect effect*effect2 / ADJUST = TUKEY ;

This will give you TWO tables. One, like I just showed with the means for each group and a SECOND table that tests for the mean differences.

There are so many important points here, it feels like one of those games they used to give us at St. Mary’s School to keep us busy, when we were supposed to make as many words as we could out of some word like “toothpaste”.

Here are just a few:

1. If you’re smart enough to have found this blog on the Internet and be reading it you’re perfectly smart enough to figure out all the statistical knowledge and programming you’ll ever need.
2. Just because you make a mistake, it doesn’t mean you’re dumb. You’re not perfect. Welcome to humanity.
3. DON’T accept “kind of” knowing how to do something, like kind of knowing how to code an LSMEANS statement or knowing how to do a GLM but not how to do a post hoc test. All those pompous asses who brag about “not being a detail person” are full of bull shit. Knowing whether you are testing the hypothesis that the population mean is zero or that the mean of the control group equals the experimental group mean, that’s a detail and a damn important one.
4. If you don’t know, ASK! I really miss Dr. Eyman. He died several years ago and I kept in touch with him up until a few weeks before his death from cancer. Not only was he a really nice person, but it was such a comfort to have someone to go to who had “the answers”. The last few years before he passed away, every time I would be in town, drop by his office and ask him a question, he would look  over whatever it was, give me his opinion and then add, “I don’t know why you ask me. You probably know more about this stuff than I do now.” So, no, I don’t miss those times in graduate school when I would go to his office hours, ask a question about Markov chains and he would look at me like I was the world’s most complete idiot and say something like, “I wrote an article that used Markov chains that was just published in the New England Journal of Medicine. I can’t believe you haven’t read it!” I do miss that time when I had someone to not only give me the answers but sometimes to give me the questions, too, like “What  did you think these tests were testing?” It’s like Watson said, never be the smartest person in the room. I’m the smartest person in the room right now, but that may be just because the only two left in the room are me, and Beijing, the cat.