Don't you just love those times when you forget to negate a negation? When subtraction like x-y becomes addition if y is negative?
So yeah, I solved this on my own and feel stupid now. I'll leave this up for any GameMaker users wanting to learn ASM.
Below is the ASM for the vertical movement of Medusa Heads in Castlevania 3. This isn't the ASM straight from FCEUX, I took some liberties cleaning it up to make it a little easier to read. Note that when the result of "SBC x" is negative, the carry bit is cleared if it was previously set, and if the result of "ADC x" is greater than $FF, the carry bit is set.
LDA ordY //Load the value of ordinate Y into the Accumulator
SEC //Set the carry status bit
SBC ystart //Subtract the value of ystart from the Accumulator minus negated carry bit
LDY $00 //Load value 0 into Y
BCS 880E //If the carry bit is still set, skip to address 880E
DEY //Subtract 1 from Y (so, set Y to $FF)
880E STY 0x01 //Store the value of Y to variable 0x01
STA 0x00 //Store the value of the Accumulator to variable 0x00
LDA vert_2 //Load the value of vert_2 into the Accumulator
SEC //Make sure the carry bit is set
SBC 0x00 //Subtract the value of 0x00 from the Accumulator minus negated carry bit
STA vert_2 //Save the result back in vert_2
LDA vert_3 //Load the value of vert_3 into the Accumulator
SBC 0x01 //Subtract the value of 0x01 from the Accumulator minus negated carry bit
STA vert_3 //Save the result back in vert_3
CLC //Make sure the carry bit is cleared
LDA vert_1 //Load the value of vert_1 into the Accumulator
ADC vert_2 //Add the value of vert_2 to the Accumulator, set carry bit if over 255
STA vert_1 //Save the result back in vert_1
LDA ordX //Load the value of ordinate X into the Accumulator
ADC vert_3 //Add the value of vert_3 to the Accumulator
STA ordX //Set ordinate X based on the result
It's pretty straight-forward, I think. The biggest issue is Game Maker uses QWORD values but the NES used BYTE values, so I have to constantly truncate everything into bytes. That's easy enough: n + $100 & $FF. Last night I figured out how to convert an unsigned QWORD into a signed BYTE, so I could change $FE to -2 and $FF to -1 but keep $02 as 2: (n ^ $80) - $80. But I'm doing something wrong with my translation of the code because the value of vert_3 and vspd (which is basically vert_3 plus the carry bit) either constantly increases, constantly decreases, or alternates while constantly increasing. The way the code should work is vert_3 should always be either $FE, $FF, $00, or $01, and vspd should always be either -2, -1, 0, 1, or 2. Here's what I have so far:
vert_2 -= y - ystart + $100 & $FF
c_bit = vert_2 >= 0
vert_2 = vert_2 + $100 & $FF
vert_3 -= $FF * (y < ystart) + !c_bit + $100 & $FF
vert_3 = vert_3 + $100 & $FF
vert_1 += vert_2
c_bit = vert_1 > $FF
vert_1 &= $FF;
vspd = (vert_3 ^ $80) - $80 + c_bit