Re-read the sentence you quoted, friend sheriff. It says, "anybody who seeks to destroy pluralism deserves one label only and that label is extremist. We should just call them extremists and we should treat them as such."
"anybody who seeks to destroy," not anybody who is apathetic or anybody who is wishy washy or whatever.
In his discussion, Patel specifically states that, while there are extremists in all religions (I would add: and all walks of life), the majority of people of all and no religions rever peace and have respect for others.
But in the US, this view is under increasing endangerment.