We have some issues with the kids in our neighborhood, too. For the first year all was well and we even moved our swingset to the front yard to let everyone play but recently the neighbors dd AND her parents have been nasty so we've moved it all back to the backyard.
It's really hard, but the silver lining is that your children are learning lessons. :)