I understand your situation. You answered yourself for your problem. As you herself committed that you can not go with your friend then why you are wasting your time with him. See your interaction with your friend will give you temporary peace and pleasure. But this pleasure and happiness will cost you more. Later you will feel restless and unhappy.So if you immediately stop this interaction then initially you have to feel pain but later all things will go fine.
You can not travel in two boats simultaneously. As you both (I am not sure about that boy why he is afraid of accepting you?) are not ready to face the consequences and afraid of taking bold steps. So don’t proceed further in this relation.
As you have only one option left then why not to make this option comfortable. Try to increase the frequency of interaction with your husband. Sometime he and sometime you visit each other place and meet each other. If possible for you to shift to his place then it will be great. If it is not possible then no problem. 3 years in not too much of time that you can not spent.
Stop watching other people life. Everybody has their own life. People who have smiley faces on Facebook does not mean that they are very much happy. So forget other people life. Try to focus on your life and your relationship. Try to comfortable with your husband physically. Initially it will not but as time passes you will realize that.
Remember that if your relationship with your boy friend come in light that you will be in big trouble so immediately stop that.