That's always been a problem when playing with random players. Be it Happy wars or another game.
There are a couple of thing that Is really anoying in the hard special challenge.
People not noticing that a slayer spawns when you destroy a tower or someone attacking a slayer when your trying to lure it in a trap.
That are some off the most fustrating things for the special challenge in my opinion.
But there isn't much you can do about it. People just won't listen.