When I was much, much younger I would have agreed with everything you said here. But since then I've realized that:
(1) there is *no* racial insult anyone can use against whites that carries anything approaching same level of emotional offense that the n-word does against Blacks. Why? Because we did not suffer centuries of race-based cradle-to-grave slavery (followed by Jim Crow) as Blacks most certainly did, and
(2) the use of phrases such as your "issues on both sides" is erroneous to the point of being an insult in and of itself, for it is not only impossible to compare the history of whites over the past five centuries to that of Blacks, but the great disparity in experiences is still extant to this day, in pretty much every aspect of life.
After all, how can a culture take five centuries of basest cruelty and suddenly make such words as the "n-word" have "no actual power"? Nazism lasted but a fraction of that time, yet see how most international culture abhors it.
The story I like to relate is an experience that Oprah Winfrey - the billionaire - described in a visit to Switzerland. She walked into a store, saw a purse she liked, and asked the attendant to let her see it more closely. The attendant - who obviously didn't recognize her client - said, "No, you can't afford that purse." Yes, this was in Switzerland, but anti-Black racism isn't limited to America. In Alabama, a judge was recently fired for - among other things - assuming a Black defendant was a drug dealer because he drove a Mercedes. Both experiences above were based solely upon race.
These are experiences you (assuming you're white like me) simply would never have. Thus, it is a great mistake to assume that you can know how they feel. You cannot have experienced life as they have, and so you cannot possibly know how they feel. The best you can do is to *listen*, and realize that they're trying their best to help you to gain at least a worm's-eye view of the world they've always known.