Funny enough I can deal with skeletons. they can be annoying in groups, but they are fairly predictable, and it's easy to manipulate their AI into doing what you want.
Swordmasters though are just too fluid and dont have enough telegraphing for their attacks, which themselves come out super quick.
And in a group? well shit.