When we took our fostering classes, one of the meetings was specifically to meet current foster parents. A time to ask *anything*... It was helpful. We heard some crazy stories, and still wanted to do it.
Have you taken a class yet? I think a lot of questions and concerns are answered during that time, and you can solidify your feelings while you learn.
I would not hesitate to foster, if we had more room. We took the classes twice - once as part of our adoptive parent training (foster and adoption classes were combined at the time)... and when we did it that first time we knew we wanted to foster next. The next time, we took foster-only classes, planning to foster-adopt, knowing the risks. In our case, we brought our DD home from the hospital and she's been with us ever since - it went straight to adoption. One of the only adoptions like this our DSS has done, and we'd never have dreamed it possible. So we *still* haven't fostered in the way we planned... and as soon as we have more room, we'll do it. It's something both DH and I want to do, for many reasons.
FWIW, in our area, there are quite a few infants needing foster care, too, on a regular basis. In fact, while our placement was going through there were at least 2 more in a 2 month period - that I know of.
I think respite care is a great option, too. Maybe that would be a good option to try as well? Lots of people start that way. :)