Monday, October 29, 2012

The Inner Workings of the SFUSD Lottery

Confused about how the SFUSD school lottery works?  It's a knotty problem.  The 2012 Nobel Prize in Economics just went to Roth and Shapely, for developing the algorithm behind many school lotteries today.
    Luckily, we don't have to be Nobel Prize winners to understand how the SFUSD lottery is supposed to work. The group who proposed the redesign of the school lottery algorithm has several helpful and detailed powerpoints.
    The first powerpoint on SFUSD's school choice algorithm gives an excellent background of SFUSD's history with desegregation and school choice. It also has a diplomatic assessment of SFUSD's political climate and previous school lottery. The presentation looks like it's intended for economists, but it's worth a read even if, like me, you have zero economics background.
    The 2nd is a presentation to SFUSD's school board about the new algorithm, which the school board adopted. It gives details of the algorithm in lay terms, including an illuminating diagram on swaps on Slides 17-18.
    The 3rd is a presentation to PPSSF, similar to the presentation to the SFUSD school board. It also describes the results of simulations when the algorithm's priorities were changed.  Of interest, prioritizing CTIP over AA, or vice-versa didn't significantly affect either diversity or the percentage of students who get their first choice.

Advantages of the new algorithm*
  1. No way to game the system: If you really want Clarendon, you shouldn't rank Sunset first, thinking that you're more likely to get your second choice. Ranking Clarendon as your top choice gives you your best chance of getting in. Addendum 6/2013: In practice, families may be able to game the system by strategically positioning popular schools on their list for "swap value." For example, if you were in Alvarado's AA but didn't want it because your child was Asian and their Asian enrollment is very low. You might put it on your list anyway, just below the schools you really want, in hope of swapping up.
  2. Tries to match every student with a school that's as high on their list as possible:  Before, Student A might be assigned to Jose Ortega and Student B to Miraloma, even though  Student A preferred Miraloma, and student B preferred Ortega. Swaps deal with that situation.  (See Slide 18 of the SFUSD school board presentation).
*paraphrased from Times-Picayune on a school choice algorithm in New Orleans designed by the same group, as quoted on the this blog):

Disadvantages of the new algorithm
  1. "Justified Envy" This is a real economics term! Swaps let you trade on your priority at one school (sibling, AA) for a priority spot at another.  Even though swaps maximize the number of students who get their top choices, they feel unjust, because a student can get assigned to a school over a student who was supposed to have priority over them.
  2. Increases Inequity?
        Swaps also feel unjust because they improve lottery outcomes for the haves (with desirable spots) compared to the have-nots (no spots/undesirable spots).  Students in attendance areas of desirable schools are more likely to find other students to swap with, and thus get an even-higher choice.  And wasn't school choice supposed to be about decreasing inequity?
A disclaimer now about theory vs. practice. The consultants from Harvard, Stanford, Duke, and MIT who designed the algorithm noted the tendency of SFUSD to make ad hoc changes to the school lottery criteria nearly every year.  They discouraged this, but anticipated that SFUSD would continue to do so. They offered to implement the algorithm for SFUSD in a way that would allow SFUSD to adjust its priorities without undermining the integrity of the algorithm. They also wanted to monitor the effects of the new choice system, for free, so that future changes could be more informed.
     SFUSD declined and decided to have someone in-house program the algorithm (yipes!).  Al Roth wrote on his blog:
    "Unlike the case of the systems in New York and Boston... my colleagues and I don't know what algorithm SFUSD is using, even though we know what we proposed and the Board adopted. So...this post is a bit like the ads that sometimes appeared in the financial sections of newspapers when I was young, which, following a divorce, would announce that Mr John Doe was henceforth no longer responsible for any debts incurred by the former Mrs John Doe..."


8 comments:

  1. if you put a large number of schools on your list then EPC is more able to assign you to one (even one low on your list) and proclaim that they were able to offer you one of your choices. you could be penalized in this way-if you list some that are okay, but not great, you could very well be assigned that one while the district pats themselves on the back. so i think you can still "game" the system-list only a few schools...they will try to give you one and still be able to claim that they offered you one of your choices.

    ReplyDelete
  2. The algorithm doesn't take into account how many schools you list. If you list only a few schools, and they are all full, you don't get assigned anywhere.

    ReplyDelete
  3. yes, but because the swap is so mysterious they could always say you swapped out of your first choice and were placed in your
    15th choice-and they will still be able to say "we are pleased to offer you one of your choices" (which is what our letter said last year). this happened to us and several people we know. people who kept a small list or even only put one school on their list all got their first choice. we even had a preschool friend who put one school on his list-which was not his AA school and he got it. we live in that same AA and did not get that school, which was our #1 choice. we were told he swapped in. how can you swap with one school on your list?
    my point is, the EPC makes their own rules and they are not accountable to anyone!

    ReplyDelete
  4. I really don't recommend only putting a few schools on your list (unless you would only consider sending your child to that handful of schools). Everyone has anecdotes, but the best way to get a school you're happy with is to list every single you'd be happy with on your application form.

    I don't think it's statistically true that people who only put a few schools are more likely to get one of their choices. Anecdotally, I have several friends who tried that strategy, only to have it backfire completely.

    ReplyDelete
  5. As you noted, Roth and his colleagues offered to implement and monitor the system, and SFUSD turned this offer down for mysterious reasons of its own (probably so EPC could continue making ad hoc changes without oversight.) Parents would have a lot more trust in the system if at least they knew it was being implemented as designed by Roth and approved by the BOE.

    ReplyDelete
  6. i agree. this post thinks that it is all figured out, when in fact it is still a mysterious process that takes place behind closed doors. the epc will not offer any explanations for glitches or assignments that don't make sense, only to say it was the swap-which is an excuse to juke the stats as they see fit. read the 100s of replies from last year's assignment-which used the same assignment process. there is no mention of the swap in the enrollment guide and it affected so many people that they only acknowledged it after the assignments were completed.

    ReplyDelete
    Replies
    1. First of all, this is just plain silly, the planting of conspiracy theories. The fact of the matter is, the district has been up front all along about what you should do to apply. The instruction is simple: List your choices in order of preference. It's your fault if you don't understand how the process works.

      Second, the EPC does make mistakes sometimes. If you list a large number of schools and don't get anything, it's possible that something was miscoded. But no, you would not ever get 'swapped out of your first choice.' That's ridiculous.

      Third, you are aware that SFUSD has further assignment rounds, right? So you still have a chance of getting a slot at a school you're happy with, even if your initial lottery result doesn't turn out the way you want it to. But no, you can't expect miracles when some of the most desirable schools only have like 20 slots available to non-siblings. Deferred-acceptance algorithms can only work so well for a large number of people all vying for just a few spots in the district.

      It's really upsetting to see such false information sitting here unchallenged when we're down to the last few days and people are likely looking here for information.

      Delete
  7. The 3rd PowerPoint in the post (i.e., the presentation to PPSSF when the algorithm's priorities were changed) is an example of fun with numbers, and the conclusion is meaningless. While the simulation “didn't significantly affect either diversity or the PERCENTAGE of students who get their first choice,” I can guarantee that the simulation changed WHO got their first choice, and that is what concerns parents.

    ReplyDelete