Here's something I didn't know until just now: The fleur de lis, the official symbol of New Orleans, was once branded on runaway slaves (as was done in other French-governed areas). This has some reconsidering it just like the Confederate flag.
The branding fact is troublesome and shameful. But this reminds me of the counterpoint the recent flag debate brought out: "If we're going to take down the Confederate flag for its racist connotations, shouldn't we also take down the American flag? After all, it once stood for slavery too!"
To that I say, no. Not because it isn't true (technically, it is), but because that isn't primarily what it's known for. You could say the American flag stands for any litany of dubious things, because our country's history is messy. But at its best, the banner stands for freedom and a nation that is constantly striving to live up to its ideals. The flag that once flew over a slaveholding nation now flies over one led by an African-American president, an imperfect nation that has progressed on civil rights in a way that was unfathomable 50 (and even 10) years ago. Whatever your stance on patriotism or America's track record, chances are you see the Stars and Stripes as, ultimately, an earnest symbol of liberty and good.
The Confederate flag, on the other hand, is primarily about celebrating a history of slavery and racism and rebellion related to both. That is its purpose and always has been. Others might have used that banner in more benign contexts, but those meanings are secondary to what most people associate with that particular image.
The distinction is important with regards to the fleur de lis. It was a French symbol first, and later became a positive representation for one of the world's most diverse cultural epicenters, New Orleans. In between, it was briefly co-opted by terrible people for a terrible purpose. That's a shameful (and previously obscure) chapter in its history, but that was never the fleur de lis' raison d'etre, even most likely at the time that such cruelty was being inflicted.
(That's not to say co-opting can't permanently change a symbol; the Nazis turned the swastika from a peaceful character into the literal caricature of evil. But, again, that became its primary meaning. Not the case with the fleur de lis.)
There's nothing wrong (and, in fact, a lot of things right) with re-examining our national symbolism through the lens of social progress. But that pursuit should be about confronting our aggressively divisive symbols, not about letting the bad guys win on our beloved logos.