One-night stand, hopefully turning into "fun buddy"?

I would say add him. Then you could simply suggest becoming "fun buddies". Most likely he'll be up for it, and if not then who cares.