The First Rule of Infidelity
Protect the kids.
These guys can suffer unimaginably badly if you get caught cheating on your husband/wife. If you can perceive any way that they could become collateral damage, end it right away.
I know that having an affair is the perogative of the selfish jerk, of which I number. There are some lines that just should not be crossed.