In the wake of the white supremacist rally in Charlottesville, there has been a lot of discussion about free speech, and whether this right protects such groups. This debate is coming out of a larger one about harmful speech, particularly on college campuses. I struggle with this question (for reasons noted below), but find that the way this debate is popularly framed is misleading in two respects.
(1) It is not a debate between free speech advocates and opponents of free speech. Rather, it is a debate about what free speech means, and how this right is best implemented politically. Consider those who want to restrict the speech of hate groups. The argument for this position is that such hate speech limits the ability of other people in society to speak. Thus, limiting hate speech may actually *increase* the access to free speech in a society. The basic idea is that we should think about free speech as the ability to do things with words (like make assertions that others listen to), not to merely vocalize syllables. When others engage in hate speech, they help create an environment in which the oppressed are unable or unwilling to speak, because others do not listen, or out of fear for their safety. It is this latter idea which motivates the ACLU's recent decision not to defend armed protestors (their guns, it is argued, have a chilling effect on speech).
On the other side, defenders of greater legal protection for speech tend to argue that the legal right has to be absolute, otherwise it will be selectively applied in ways that favor the powerful. While this may mean permitting harmful speech, it would be more harmful, it is argued, to give those in power (who can do more to reinforce systems of oppression) the legal authority to silence the voices of the powerless.
It is a difficult issue, and one I struggle with myself. The important point, however, is that it is not a matter of supporting or opposing free speech. It is about what kind of society best promotes free speech.
(2) Following this point, it is not a matter of "offensive" speech. Taking offense is a fairly weak property. People can take offensive to all sorts of things, including the innocuous and the true. There probably are white supremacists who marched at Charlottesville who would take offense at being called a "white supremacist." I hardly think that would be adequate grounds for denying my right to say so.
If, however, we understand the debate in terms of speech that silences or removes the voices of others from public discourse, we have firmer grounds for distinguishing truly harmful speech from that which merely offends someone. I'll be offended if you call me an idiot, but your doing so in no way reduces my ability to speak in our society (it is not systemic, and it is not part of any broader social structure that oppresses me; in fact, I am a benficiary of social power structures).
As I said, I still struggle myself with these questions. What I think is crucial, however, is that we engage the actual points of disagreement, rather than a caricature of the positions.