I can only echo what everyone else has said. I think he's so used to your smell and sounds when he is being held and fed that anything new is a change in his routine. Andrew does Matthew's 7pm feed and its always been like that because of working. But at the weekend Andrew will want to do more, but sometimes Matthew will only take half the bottle and will be a lot more fidgety than with me.
Do you spend any time with friends or other family members during the day? Maybe you could get them to help out with cuddles and feeds for a while, then your littleman will get used to it not just being you and then may warm to Daddy more when he comes home from work. My best friend and I spend a lot of time together during the week and if I need to do my hair or do some housework she'll feed him and cuddle him to sleep. I don't know whether it helps but Matthew will go to anyone with a warm arm to lay on
