Stress, hormone imbalance after pregnancy, lack of lubrication, etc. can all play a role in getting back in the 'swing of things'.
Did either of you get an episiotomy when you had your babies? Sometimes the repair of a tear or episiotomy can cause a reduction in the 'stretchiness' of the opening.
If you think your hormones are ok, you are getting the lubrication you need, and 'size' is the only issue, you might consider perineal stretching like what is sometimes recommended prior to delivery of baby (to keep from tearing, and to avoid an episiotomy).
Also, breastfeeding can sometimes play a large roll in how 'frisky' we feel. For some women they have absolutely NO desire for sex, and so the thought of it is scary and effects how everything feels if they decide to 'give in'.
Seeing your OB or midwife is a great place to start if you do think there is a medical or psychological issue.
