The Durhams were at a minimum socialists and the grandfather was a close friend of Frank Marshall Davis,
hence the patriarchal relationship with Barry.

Sounds like he was preparing the groundwork for a kind of "excuse", in a way, for having gravitated toward those people. He suspected (or...his handlers did quite early on) that he might later have to explain these associations. So he prepares the way by saying he deliberately chose them for some sort of cover, or social survival. Say that IS the case and these bonds did not come about naturally, out of common interests, he'd rather admit to having USED all these people. Either way it fits him to a tee, as far as I can see.
~ ~ hey, I'm a poet.~ ~