Technically, that dialogue should be the Crystal Knight's, not Simon's.
After all, Simon does not have the power of Clairvoyance, so it's not like he knows exactly what crystal he should be exchanging for which.
"I'd like to exchange a white crystal for a blue one".