Re Post: Is David Brooks That Gullible? by Aaron Pallas

FYI - If you want to see some great back and forth debate on the issue, go to the original website to see all the comments afterward.

Re-posted from www.thisweekineducation.com

May 8, 2009

Just How Gullible Is David Brooks?

by Aaron Pallas

Now that I have your attention … Today’s New York Times column by David Brooks touts a new study by Roland Fryer and Will Dobbie of the Harlem Children’s Zone (HCZ) Promise Academy charter schools, two celebrated schools in Harlem. Fryer and Dobbie’s finding that the typical eighth-grader was in the 74^th percentile among New York City students in mathematics leads Brooks to state that HCZ Promise Academy eliminated the black-white achievement gap. He’s so dumbstruck by this that he says it twice. Brooks takes this evidence as support for the “no excuses” model of charter schools, and, claiming that “the approach works,” challenges all cities to adopt this “remedy for the achievement gap.”

Coming on the heels of yesterday’s release of the 2009 New York State English Language Arts (ELA) results, in which the HCZ schools outperformed the citywide white average in grade 3, but were well behind the white average in grades 4, 5 and 8, skoolboy decided to drink a bit more deeply from the datastream. The figure below shows the gap between the average performance in HCZ Promise Academy and white students in New York City in ELA and math, expressed as a fraction of the standard deviation of overall performance in a given grade and year. The left side of the figure shows math performance, and the right side shows ELA performance.

It’s true that eighth-graders in 2008 scored .20 standard deviations above the citywide average for white students. But it may also be apparent that this is a very unusual pattern relative to the other data represented in this figure, all of which show continuing and sizeable advantages for white students in New York City over HCZ students. The fact that HCZ seventh-graders in 2008 were only .3 standard deviations behind white students citywide in math is a real accomplishment, and represents a shrinkage of the gap of .42 standard deviations for these students in the preceding year. However, Fryer and Dobbie, and Brooks in turn, are putting an awful lot of faith in a single data point — the remarkable increase in math scores between seventh and eighth grade for the students at HCZ who entered sixth grade in 2006. If what HCZ is doing can routinely produce a .67 standard deviation shift in math test scores in the eighth grade, that would be great. But we’re certainly not seeing an effect of that magnitude in the seventh grade. And, of course, none of this speaks to the continuing large gaps in English performance.

But here’s the kicker. In the HCZ Annual Report for the 2007-08 school year submitted to the State Education Department, data are presented on not just the state ELA and math assessments, but also the Iowa Test of Basic Skills. Those eighth-graders who kicked ass on the state math test? They didn’t do so well on the low-stakes Iowa Tests. Curiously, only 2 of the 77 eighth-graders were absent on the ITBS reading test day in June, 2008, but 20 of these 77 were absent for the ITBS math test. For the 57 students who did take the ITBS math test, HCZ reported an average Normal Curve Equivalent (NCE) score of 41, which failed to meet the school’s objective of an average NCE of 50 for a cohort of students who have completed at least two consecutive years at HCZ Promise Academy. In fact, this same cohort had a slightly higher average NCE of 42 in June, 2007.

Normal Curve Equivalents (NCE’s) range from 1 to 99, and are scaled to have a mean of 50 and a standard deviation of 21.06. An NCE of 41 corresponds to roughly the 33^rd percentile of the reference distribution, which for the ITBS would likely be a national sample of on-grade test-takers. Scoring at the 33^rd percentile is no great success story.

How are we to make sense of this? One possibility is that the HCZ students didn’t take the Iowa tests seriously, and that their performance on that test doesn’t reflect their true mastery of eighth-grade mathematics. The HCZ Annual Report doesn’t offer this as a possibility, perhaps because it would be embarrassing to admit that students didn’t take some aspect of their schoolwork and school accountability plan seriously. But the three explanations that are offered are not compelling: the Iowa test skills were not consistently aligned with the New York State Standards and the Harcourt Curriculum used in the school; the linkage of classroom instruction to the skills tested on the Iowa test wasn’t consistent across the school year, and Iowa test prep began in February, 2008; and school staff didn’t use 2007 Iowa test results to identify areas of weaknesses for individual students and design appropriate intervention.

If proficiency in English and math are to mean anything, these skills have to be able to generalize to contexts other than a particular high-stakes state test. No college or employer is ever going to look at the New York State ELA and math exams in making judgments about who has the skills to be successful in their school or workplace. I’m going to hold off labeling the HCZ schools as the “Harlem Miracle” until there’s some additional evidence supporting the claim that these schools have placed their students on a level academic playing field with white students in New York City.

An Older Post, from my birthday about life in general

I'm eating oranges on my birthday, on my last day of vacation before the semester begins (kind of). The sun is shining a little brighter every day and I'm about ready to hang my grandfather's clock above the piano.

AK is a wonderful place full of all kinds of interesting adventures. I'm attending school at UAA, and have recently (finally) decided to major in math. Carey has a great job teaching in the anchorage school district and after years in the cooking profession, I'm ready for the steadiness and great hours (vacation too) of teaching - probably high school.

We bought a house last Spring/Summer, I spent the whole summer on a new ASC yacht (last year for that probably), and jumped right back into school. Before summer was over, Carey and I spent my two week vacation together in Juneau, ferry hopping to Petersburg for a week, then Sitka for a night (or two?). In Petersburg we were greeted by friends who took us fishing (in the rain no less). After a whole day, each of us had caught a beautiful fresh Halibut. Carey took the prize with her approx. 108 lb Halibut, which Nathan and Trinity helped us haul home and clean, eventually sending us the meat via Alaska Airlines (video of the catch is currently available on Carey's Myspace page). Petersburg is nothing if not quiet (other than fish noises). Sitka has a to-die-for restaurant - nuff said.

I like long walks on the beach and cozy campfires.

I'm taking piano lessons so I can get decent enough to play in front of people, was on the UAA debate team but have since quit that (maybe resume to a limited degree soon); I'm assistant coach to the East High DDF (debate, drama, and forensics) team which is way more hoot-hooty than I thought it would be.

We miss family and friends in the lower 48, but Anchorage has been very good for me. It's easier to find yourself when you isolate yourself a bit, but you also feel guilty for leaving other people out of your life, as much as you'd like to be around them all the time. We've made more than a couple new friends up here who we will be in contact with even if we move back to Oregon/Washington (which we plan to do in about, oh...2 more years I'd say roughly).

I've been into cooking sushi a bit, and I just got a slow cooker/crock-pot that I'll be delving into (thanks Dad). The slow cooker is some Euro- somting or another and has a two chambered insert that allows you to use it to keep two seperate buffet items hot, like a chaffing dish. Very exciting for the rag-tag get togethers we've been hosting for random (mostly teacher) friends who we invite over for holidays and poker nights and what-not. Really just an excuse to cook for people - it's hard to keep "the skills" in working polish when just cooking for Carey and me.

Alaska is fascinating, and home prices are steady (though gas prices are crazy - diesel esp. like 3.59 or some s*($, but we use regular which is about 2.3something I think, maybe 2.5). On a walk with the dog the other night, just around the block, his feet got so cold that he just wanted to stop. It must have been at least ten below zero, and I had to cajole/drag him along behind me - a very unusual position for our big black lab. The liberal talk radio station KUDO is great too. There's tons of crusty strange people who have radical takes on life - similar to Oregon, but not nearly so mellow and touchy feely. To be fair, it isn't too much different, except that Alaska is bush-country (not Bush country). Despite this, there is very little beating around the bush, and more just going straight for it. Anchorage doesn't have a very broad progressive/green community, or doesn't seem to - lots of Independent parties and an old guard of Alaskan separatists (though I'm still checking on this). Then there's the whole Palin thing...

I got a job as a Supplemental Instruction Leader (study group coordinator/leader) for a Calculus class at UAA. I'm excited to be paid to go to class for once - not that I'm economically motivated, but even at 9.50 an hour, the perceived value of any commodity goes up when someone starts paying you for acquiring it. I'm trying to get a job at the campus paper too, but we'll see.

Carey likes teaching and has been warming up to the debate thing. She has many loyal fans among the students owing to her high expectations, candid communication, and interpersonal enthusiasm. I think she mostly likes being the boss, but who knows. Either way, she has had many marvelous success stories credited to her name since joining with the Anchorage School District.

Buddy on the other hand is just idling until the good days coming in the latter part of this Spring, when spittle and breath vapors don't freeze to your fur moments after they hit the air. Also, his view out of our big living room windows is obstructed by a mostly permanent layer of ice coating the interior, and only melting a little bit on top when the sun blares into it for a few hours. We're in process of making giant curtains to hang in those big windows; the fabric is awesome and we can't wait to see it done. Between that and replacing the windows someday, we hope to keep our house a little warmer next winter than it has been during this cold spell.

Our favorite things lately, other than laying in the afternoon sun on the couch and dreaming about seventy degrees, have been: our organic produce box every other Wednesday from Full Circle Farms, the hoarfrost on the trees, Payday + Costco, video games, regular games, the internet, and Babylon 5 DVD's. And Buddy of course, and friends near and far. Music, art, writing, and self-guided study distract me in phases and ebbs - I am ever the undergraduate 'absent-minded professor'.

I met a man this summer fondly referred to as "Old Goat." His spanish is way better than mine and we enjoyed many conversations together; he told tales of pyrotechnics at Burning Man that he worked on, Alice in Chains Allison Krauss and a slew of 70's psychadelic rock band members he's known or ran security for, and long yarns about the Bohemian struggles of the formerly unpopular borough of Fremont in Seattle where he's been actively involved since the Seventies. -except when he was living in Mexico which is a whole other story.

All things being equal, Carey is well and I am well. The glasses are half full and progress is being made. We're dreaming of some kids and a hobby farm in the Coastal or Cascade regions of Oregon, Washington, (Northern California...Italy...Vancouver...)

Well, even though there's many other wizard things to discuss, I'm going to get moving here - it's getting cold even in my pajamas.