Well, later he's going to be concerned that people don't actually find his rare jokes funny.

"I don't typically engage in conversations like that, no. They aren't appropriate in a professional environment."

It's also entirely possible he's just walked away from people attempting to flirt with him, but now he's thinking over some of his interactions on the islands, wondering if he's missed something.