Sunday, September 30, 2018

TEST EXPERT FRED SMITH ANALYZES FLAW IN THE CURRENT STATE EXAMS

Fred Smith, testing expert and fellow Leonie Haimson Skinny Award winner, describes another problem in NY State achievement test results:
Folks,
I believe the opt-out movement is viable and capable of growth in NYC--even though we have a Mayor and chancellor who are advocates of mass testing in grades 3-8.
The Grade 6 ELA results for New York City are screwy.  They strike me as a weak link in Questar’s testing chain.  The percentage of students deemed proficient this year is 48.9%.  It was 32.3% last year.  That’s a 16.6 difference– or a shift of from nearly one-third to one-half of (65,000) sixth graders who are now “proficient.” In no other grade is it more than 8.0.
Surprisingly, differences of the same magnitude hold for all ethnic groups.
[I know we were warned not to compare the 2018 results directly with the 2017 results. Still that’s a singular difference since the same publisher, Questar, produced both tests under a $44 million, five-year contract with SED.]
NYC ELA Percent Proficient by Grade
Grade
2018
2017
Diff.
3
50.6%
42.6%
8.0%
4
49.3%
42.0%
7.3%
5
38.0%
36.1%
1.9%
6
48.9%
32.3%
16.6%
7
42.6%
43.3%
-0.7%
8
50.7%
47.5%
3.2%
3 - 8
46.6%
40.6%
6.0%




NYC Grade 6 ELA Percent Proficient x Race/Ethnicity
Group
2018
2017
Diff.
Asian
69.3%
55.0%
14.3%
White
70.3%
53.2%
17.1%
Black
35.0%
19.1%
15.9%
Hispanic
38.4%
21.4%
17.0%
And how does this useless testing program serve educators who are judged by such inexplicable data and who must design programs to meet the academic needs of students–based on such shaky (as in meaningless) information???
An outcome like this is an example of why we need to have timely information about how the items on the examination functioned.  Yet, SED and DOE have not provided data at their disposal that would shed light on the matter.  Instead, NYC parents are expected to march their children off to the deadening testing drumbeat for the next three years uninformed about the workings of the exams.
We must figure out a way to demand and obtain the information hidden behind the curtain of the test questions.*  If SED and the DOE are unwilling to disclose the facts, this would give impetus to a citywide campaign that builds on the reported four percent (4%) opt out rate and escalates it in 2019.
~~~~~~~~~~~~~~~~~~~~~~~~~~~
*The information consists of item-level statistics that SED and DOE routinely keeps.  It would allow multiple-choice items and constructed response questions to be studied to see how students answered them.  For M-C items, we should have classical item analysis data on the percentage of students selecting each option.  For CRQs, we should have the percentage of students receiving each score from trained raters.  Having both sets of information would give us a picture of the response and scoring distributions generated by students and lead us to evidence-based insights into the quality of the exams. Not only must SED and the City already have such overall data, they also have—or should be able to produce it by subgroup—i.e., for ELLs, students with disabilities and for students by race/ethnicity—that would give us further understanding.
(If you agree, please post and share the above with allies and potential allies in places I am incapable of reaching.)
Fred
I guess the sixth graders are all geniuses or maybe they just had better teachers. No, the tests are invalid and unreliable.

11 comments:

James Eterno said...

Please no comments on how this proves the failures of liberalism. I get tired having to delete stuff but still trying to be fair. Stupid overreliance on standardized tests is bipartisan stupidity.

ed notes online said...

Important point James. The liberals are the ones who pushed this as much as conservatives. I had a reso at the DA on this issue 20 years ago on testing and Randi led the debate about how important testing is. Same with the teacher tests which told us nothing about how well someone can teach.
Our generation rose in a testocracy - where your intelligence was measured by where you stood on a test -- this a person with a 93 was smarter than one with a 92. I fell for it until life experiences taught me differently. My best friend now never went to college and did poorly in high school for whatever reasons but is as smart as anyone I know.
I had all kinds of students in terms of how they did on tests - and all were from poor families in varying degrees but the more stable a home the better results generally.
I think I did reasonably well as a teacher because I tried not to blame the kids or their parents for what life has brought them.

Anonymous said...

Whom did you blame ENO: "the system"?
Did you explain to kids how that system has lifted more from poverty and misery and ignorance than any other?
Did you show them what happened to Venezuela et al and why?
I'm pretty sure you didn't. But you did "reasonably well" indoctrinating impressionable young people. That's sinething to reflect upon I guess.

Anonymous said...

On uft.org....Testing overkill and its consequences: New York's mindless embrace of high-stakes exams has produced widespread confusion 1 hour ago Well then why hasnt uft stopped it? For $60+ per check? And we should keep paying? Oh yeah, it will get worse if we don't pay. I guess we are stuck. And suckers.

TJL said...

The author would be surprised how little data is actually kept/maintained. I bet you couldn't even do basic Item Response Theory analyses because unless someone wants to pore over ATS sheets that look like they are out of the days of DOS and dot-matrix printers, the data isn't there. This is not to mention that many, many districts across the State don't even pay attention to this information.

Anonymous said...

the tears and anxiety are in full force... I'm so unhappy with my assignment this year. I had the final reorganization grievance hearing on Friday downtown... I doubt it will go my way but I just couldn't be the doormat any longer. I'm trying to convince myself to just not kill myself and take whatever I get this year. Short version... I am underneath a supervisor who hates me and is very unprofessional. She goes through my para to tell me things... for one example. Siigh. Ive requested to attend trainings because i have not taught K before, nor am i licensed for it, and was denied. (Im a k/1 bridge split evenly...5 autistic children and multiple behaviors) I've decided to accept my fate and if she gives me developing, so be it. There is only so much i can do. But the anxiety and depression that hit me is insane... i'm trying to hold off a panic attack. If you got this far... thanks for reading.

James Eterno said...

You sound like a very brave person 7:48. Please email us for support.

Prehistoric pedagogue said...

7:48, I feel for you. Try to get your rest and keep your sense of humor, if possible. The year will pass, like they all do. Next year will be better.
Hell,it’s almost Columbus Day which is practically thanksgiving which is just about Christmas. Happy New Year and relax

TJL said...

7:48, this too shall pass.

Let's remember, you only need a 1.75 obs average to get developing and a 2.51 to get effective. Regardless, even if you get a developing obs rating for the year, in most cases the tests will bump you up to effective.

In the unlikely (3%) event you end up developing or ineffective for the year, nothing happens until NEXT September, when you get a "Teacher Improvement Plan".

Even if things still don't go well, you can't be removed from your job until you get two consecutive ineffectives for the year.

Even then, you get a hearing, you can drag out the process and continue to get paid and benefits.

And even if 3020 doesn't go your way, this is still just a job. There are many, many jobs out there that will put food in you and yours' mouths and a roof over your heads.

This sounds like a bad situation for sure, but you're still getting paid and still getting benefits. Hopefully the spring will bring you a better opportunity. In the meantime do your best and look forward to Friday afternoons, long weekends and vacations.

Anonymous said...

Because you have a union that fights for you, you are entitled to be compensated for the two 4 percent raises that Michael Bloomberg gave to members of some municipal unions in 2009 and 2010 but refused to give to public school educators and other city employees at the time.

When Bill de Blasio became mayor, he agreed to pay the money owed but said the city could not afford to pay it all at one time so we negotiated a contract in 2014 that ensured that UFT-represented DOE employees were made whole by 2020.

I am pleased to remind you that all eligible UFT-represented DOE employees now on payroll who worked for the Department of Education between 2009 and May 1, 2018 (plus those who retired after June 30, 2014) will receive a lump-sum payment of 25 percent of the money they are owed in their October paychecks.

This payment comes on the heels of a 5 percent rate increase that all DOE-employed UFT members received this past May and June.

The 2018 lump-sum payment, which will be added to a regularly scheduled paycheck, is the third of five lump-sum payments between 2015 and 2020 (see the graphic below).



For in-service paraprofessionals, teachers and other pedagogues, the money will be part of your Oct. 15 check. For nurses, therapists and other UFT-represented DOE employees who are paid on the H-Bank payroll, the money will be in your Oct. 19 check. The DOE has not yet determined when per-diem and per-session payments will be issued.

If you are on paid parental leave this October and you return by the last day of February in the same school year, you will receive your lump-sum payment on March 15, 2019. If you return to work after the last day of February, you will receive both your missed payment and the next lump-sum payout, if any, in October 2019, provided you are back on payroll. If you are on another form of leave this October, you will receive your money on the date of the next scheduled lump-sum payment that you are back on payroll.

Your TDA will be updated, along with all other payroll contributions and deductions.

While the calculations can be complicated, the truth is simple: You deserve this money.

For every check you have received since late 2009 until May 1, 2018, when the last of the two 4 percent increases went into effect, lump-sum money has accrued representing the difference between what you would have been paid if your paycheck had reflected those two 4 percent increases in 2009–10 and what you were actually paid.

Think of it as a bank account. If you have been continually employed, you have been depositing money in this account since Nov. 1, 2009. This October, you’ll make your third withdrawal.

See this handy chart to learn more about the salary increases and lump-sum payments you will receive as a result of the 2014 UFT-DOE contract.

Thank you for everything that you do.

Sincerely,

Michael Mulgrew
Michael Mulgrew

waitingforsupport said...

@7:48 you probably won't believe me but everything will be okay. My sister was you...3 beast admins were tormenting her so badly. Unfortunately the union wasn't much help. Her fear was being 3420'd. She went for PDs, workshops,etc to no avail. When the gotcha squad is after you, they're after you. Well it happened (3420) and it saved her life. Their bid to take her license failed but into the ATR pool she went. It saved her life because the harassment was killing her. We told her to just quit but she has a family.My advice: if you can't quit, just do your best. It sounds ridiculously simple but it will relieve the pressure of trying to be one step ahead of the beasts. Just do your best each day. Stressing isn't going to change a thing. They will be who they will be. Having a tape recorder on in their presence doesn't hurt. It's legal. I will pray for you.