Husbands female best friend?

My husband left me with the kids 3 years ago and had a relationship with a colleague, that all ended and we remained friends for our children. Now he wants to come back but my problem is his female best friend, ( apparantly they have been best friends for years, she even lives around the corner from where we lived together but I did not know) she is also a colleague of his and they are close, they have been out go to the gym go for drinks alone etc, she has met all of his new friends since us breaking up so she is in his circle but I am not, he confides in her and they buy each other nice birthday gifts. He even wears a bead bracelet that she gave him, she is married but I am pretty sure her husband is not aware of how close they are. Is it unreasonable for me to think this is impossible for us to build our marriage back up in these circumstances? Should I walk away? My kids are desperate to have dad back.