He means that he won't belittle himself by stooping to reply to your comment due to it's lack not only of factual reference but of factual basis entirely. In other words:
"Well that's just, like, your opinion, man." - the dude
buq feels it's unfair to classify the entire group as clowns when only one of them ever was trained as a clown. After deciding it wasn't the career for him he moved into music and from there, etc etc.