One of the more interesting parts of any science endeavor is determining what you need to measure to answer a question and how you’ll know whether or not your measurements support your hypothesis. Consider this bit of discussion around cancer I recently read about in The Emperor of All Maladies by Siddhartha Mukherjee. On page 230, he discusses the problems with assessing cancer treatments based on survival-rate. Survival-rate measurements are based on the fraction of surviving patients after diagnosis at a set time interval. So the five year survival-rate is a measure of how many patients are still alive 5 years after diagnosis. Mukherjee goes on to point out that if you develop a new diagnostic test that catches the cancer earlier but do not develop any effective way to treat it, you may increase the survival rate but the same number of people are still dying around the same time. They just know what will kill them for a longer time. He discusses using overall mortality as an alternative but this runs into it’s own problems. In most first world countries, the overall population is aging – that is to say, that people are living longer than the generation before them. Cancer is a disease of time – my genetics professor in college was fond of saying that as long as we have DNA in our cells, we’ll have to deal with cancer. The older a population gets, the more likely it is cancer will be the thing that kills them. So when comparing mortality over time, you can produce a skewed picture looking at overall mortality. Other methods he mentions include a sort of time-gained-after-diagnosis-and-treatment – does a treatment add useful years to a person’s lifespan? Are they able to live more productively for a longer period of time? Also, prevalence of cancer. How common are various types of cancer and how does that compare to time-gained/mortality for these cancers.
The book is superb (I think I might write a review of it later) but these bits got me thinking about education. There is such a push for standardized assessments and measuring student achievement. But I get the impression that the people pushing for these assessments don’t actually know what they’re interested in measuring, how they’re going to measure it accurately and precisely, and when they finally do get data, what they should be doing with it. And I think a lot of this stems from the fact that we don’t have a cohesive view of what purpose education serves for our country and our society.
I joined a research project in college when it was in it’s second year; it was my intention to work on it as a summer job and be done with it. Then it morphed into my senior project and after I’d graduated and then decided to go back to school part time, I went back and worked on the project more.I even ended up with a publication out of the whole thing (first author, even!). But hell was it frustrating to work with our supervising professor. The project itself was a fairly well designed monitoring project but they had put 0 thought into how they would actually use the data once they collected it. It was easy enough to point out a few problems that the data highlighted to the farmer group we worked with. But when it came time to consider publication, the reviewers wanted hard statistics. And it fell to me to come up with those. I did (and still do) not have a strong statistical background, having taken only a biostatistics course (although I admittedly did quite well in it). I did what I could with the data but it was not an experimental design that lent itself to the simple analysis I was familiar with. It did not help that my PI had no experience with statistics so I had to teach her what every test I ran was and had to explain to her why we could or couldn’t do different tests. Fortunately, unlike the education system, we had a cohesive vision of what our project’s purpose was. But we still didn’t know what to do with our data once we had it and I have to say, I am neither super proud nor super confident in the statistical analysis I worked up with what we had because of it.
I think we need to have a strong dialogue about what the purpose of education is in this country and if we can agree on a set of goals, move from there to deciding how and what to measure that we are making progress towards these goals. Rather than just collecting data and trying to make it fit whatever framework we desire, post-hoc.