Oh hun, i'm so sorry you are feeling this
People normally say insensitive things, when they don't understand a situation. I'm sure your brother didn't mean any harm, and maybe he didn't see it as a mean thing to say, but ... i know from your point of you, that must have seemed harsh.
My future-step-mother-in-law (

) is always saying i'm obsessed with babies, and even said to my OH "Oh is Ally still taking her pill? I have a friend who deliberately stopped taking it to get pregnant" Blah blah, and that hurt.
But, the thing i have learn't through this journey is, actually, *excuse my language*, fuck them, fuck what they think, they don't know anything about how i feel, and i very mcuh doubt your brother knows how you feel. He is a man at the end of the day!
With TTC i've learn't not to actually care about what people think anymore, it's you and your OH special, exciting time, and if no-one can appreciate that, then fuck them, does it really matter to you what they think??
It doesn't bother me anymore, i may be 'obsessed' .. but don't you think it's better to be obsessed with having a baby that you desperately want, and can give a secure, loving, stable enviornent, than be 'obsessed' with alcohol/drugs etc .. !!
Try not to worry hun, and sorry about my mini rant, i just hate it when people insult us girlies at this most delicate times in our lives xx