Indeed, as do I.
Right, and by no mean am I guaranteeing the slippery slope will occur. My impact calculus is that, the consequences if a President does take criminal advantage of the precedent, the US could be significantly worse off.
So, as a comparison, Turkey's President has, for the past fifty years, had the power to ease the rules determining what candidates could run in an election. This was initially a well-meaning rule that would make it easier for certain groups to run for office. Now, however, Erdogan has used this rule to expand Turkey's political participation in election candidacy to cement his dictatorship by effectively taking a focused opposition, allowing more parties to run as part of the opposition, and thus a lack of focus emerges.
Similarly, during the 2006 Palestinian elections, the US and Israel banned tons of political groups from running. They wanted a contest between the PLO and Hamas. The logic was that nobody would vote for Hamas, a group wanting more war. Well, Hamas provides welfare to those in Gaza, and the PLO wanted a dictatorship. Consequently, Hamas won.
Tl;dr; even the most well-meaning political decisions can have grave consequences. That's how I view suspension of free speech, even for a group as despicable as the neo-Nazis. I just see serious ways where a country who elected Trump could fuck itself by giving the President power to determine who does/not have access to free speech.