I played around in a spreadsheet to study the New York State Form IT-201 line 22 worksheet, and I think I get it now.
To clarify, I'll restate the problem and explain how to work around it. New York gives up to $10K in tax deductions per year to residents for state 529 contributions. Therefore, it's possible to get into a situation where one has contributed $X over the years, but has received less than $X in tax deductions for it because contributions went over the annual deduction limit for some years.
To get the deductions in this case, one would have to rollover the excess amount into an out-of-state account, then roll it back in to get deductions for those contributions. Since the two rollovers need to be at least one year apart due to restrictions, the sooner you do the first rollover might be better, but you have some flexibility here.
Also, it benefits to rollover the exact excess amount from the New York account. When rolling back in, it shouldn't exceed $10K, again, because of the limit. For example, if you've contributed $55K over multiple years, but only received $40K in deductions, roll over precisely $15K to a non-NY account. Any less, and the calculations get needlessly complicated for the line 22 worksheet. Any more, and you get an addition to your state adjusted gross income, effectively paying the tax that you were able to avoid previously.
You can avoid going over the $10K limit in the first place by having an out-of-state account for any year that you want to contribute over $10K. Otherwise you'll need to take the excess amount out of New York with a rollover as described here.
I haven't done any rollovers myself before, so I have yet to see how that goes. Of course this maneuver would be unnecessary if line 30 worksheet would give credit for previous years' excess contributions automatically.