I have this same question too!!! The logical part of me says, "Duh, taking a swab of my cervix isn't going to cause any harm!" but the wierdo TTC part of me says, "Don't let anything touch it or disturb things!" I'm with ya and it's not an idiotic question. I think it's just a normal, however, irrational fear. Either that or we're all cracked.

I set mine up for the 21st of April. Got into a argument with the registration nurse, because she wanted me to come in right on O day (28th). I was like, no not gonna happen!!! I won, so I'm in a week before O. Yay for me!
Any way, you could just vocalize this fear to your OB. If he/she laughs at you, then you know you've got nothing to fear. LOL
Good luck sweaty, I'm sure it'll be fine.
P