I think it's a little odd to just sit there and watch. I would not want strange boys in my backyard, so I would go out there and either make friends of them or shoo them away. It is likely that their response to my greeting would determine which of those outcomes occurred.
If I really wanted them gone, or wanted them to realize that people don't just go sit in other people's yards, I would go out there and raise my eyebrows as high as I could and ask, "Hello?", maybe adding, "Can I help you?" if they didn't catch right on. That's not rude, just a subtle cue to young people who are still at an age where they have frequent need to be cued by grown-ups.