And, by the way, the reason it matters is because the alt-right movement is about mainstreaming white supremacy through pretext and deception. Trump, Bannon, and the rest want to make it acceptable for regular folks to align with Nazis and white supremacists on policy while maintaining deniability about being racists themselves.
And that's precisely what Trump was doing with his "both sides" comments. He invented a group of good people who marched alongside Nazis for non-racist reasons, and Republicans and useful idiots like Onboard scurry around defending the proposition that such a thing could even exist, let alone that it actually happened at that event. "Hey he clearly said at the end that he didn't like Nazis, just the good people who were with them, there's nothing racist about that!" And the seed is planted and grows: it's no problem for people to join a coalition with Nazis and white supremacists, or attend Trump rallies cheering alongside Nazis and white supremacists, or fight for the policies that Nazis and white supremacists support for racist reasons. You can be arm in arm with Nazis without being one of them yourself, so why not join them if you think we should honor people who fought to preserve slavery (because history!) or if you think we should stop Latino immigrants from coming (not because they're brown, but because America is full! No room left even in Montana!) or if you think we should ban Muslims from coming (not Muslims, just people coming from a list of these countries that happen to be Muslim!).