I actually think this is the biggest issue currently with Gillis, from both sides. I think where he is most comfortable is to be the contrarian voice in a room full of left-leaning people. You see this dynamic, and him thriving in it, in his early specials.
The times are currently too polarized however for a nuanced viewpoint like this, so he just gets written off (or embraced) as MAGA... and people go into his sets already having decided whether or not they will like him based on this conclusion. I get the mantra that "a man is the company he keeps" and he hangs around some of the worst dudes on the planet, but if you listen to what he actually says in jokes/interviews and the endorsements, etc. that he HASN'T done, I think the label he has gained as a MAGA mascot is more due to our country's inability to handle nuance rather than a reflection of his actual beliefs.
I can tell you that I saw him last year with what was a VERY obviously right-leaning crowd, and he seemed just as uncomfortable as that SNL performance.