Can you please let us know the reason of leaving your husband? Situation is different now. Now you are a mother of a 5 years old girl. Your decision will impact on her future.
As of now you are guessing all the things. He is confirtable with you does not mean that he is loving you. Before reaching any conclusion think about future also. Suppose he has a same feeling like you.
Will your daughter will accept him and how she will react about your relation? Will his family also accept you?
If you will face problem in this then how these incidents will make an impression on you daughter?
You have to adopt his religion also. Is it ok for you and your daughter?
Your entire feelings give me an impression that you are missing something in your life that’s why this kind of feelings coming into your mind. So be practical rather that emotional in making any decision.