Freedom in a free society is supposed to be for all. Therefore, freedom rules out imposing on the freedom of others. You are free to walk down the street, but not to keep others from doing so.
The imposition on the freedom of others can come in overt, immediate physical form — thugs coming to attack with weapons. Violence may be a kind of expression, but it certainly is not "free speech."
Like violence, hate speech can also be a physical imposition on the freedom of others. That is because language has a psychological effect imposed physically — on the neural system, with long-term crippling effects.
Here is the reason:
All thought is carried out by neural circuitry — it does not float in air. Language neurally activates thought. Language can thus change brains, both for the better and the worse. Hate speech changes the brains of those hated for the worse, creating toxic stress, fear and distrust — all physical, all in one's neural circuitry active every day. This internal harm can be even more severe than an attack with a fist. It imposes on the freedom to think and therefore act free of fear, threats, and distrust. It imposes on one's ability to think and act like a fully free citizen for a long time.
That's why hate speech imposes on the freedom of those targeted by the hate. Since being free in a free society requires not imposing on the freedom of others, hate speech does not fall under the category of free speech.
Hate speech can also change the brains of those with mild prejudice, moving it towards hate and threatening action. When hate is physically in your brain, then you think hate and feel hate, you are moved to act to carry out what you physically, in your neural system, think and feel.
That is why hate speech in not "mere" speech. And since it imposes on the freedom of others, it is not an instance of freedom.
The long–term, often crippling physical effects of hate speech on the neural systems of those hated does not have status in law, since our neural systems do not have status in our legal system — at least not yet. This is a gap between the law and the truth.