As senior Conspirator Eugene Volokh has warned, one of the dangers of censoring "offensive" speech is "censorship envy." If one group is given the power to suppress speech offensive to it, others are likely to press harder to get the same privilege for themselves. As Eugene points out in the post linked above, many of the European Muslims who sought to suppress the Mohammed cartoons were partly motivated by the fact that many European countries ban Holocaust denial and other anti-Semitic speech.
This dynamic is clearly at work in the efforts of some Catholic leaders to ban the Da Vinci Code. As Cardinal Francis Arinze, one of the chief advocates of banning The Code puts it, "[t]here are some other religions which if you insult their founder they will not be just talking. They will make it painfully clear to you." The Reuters article where this quote appears notes that the Cardinal was referring to Muslim calls for censoring the Mohammed cartoons. He and at least one other cardinal "asserted that other religions would never stand for offences against their beliefs and that Christians should get tough [too]."
The cardinals are arguing that, if Muslims have the right to ban speech offensive to them, so too should Christians. Just as the Muslims previously made the same argument with respect to Jews! The rapid spread of "censorship envy" makes it all the more important to crush this vicious dynamic at its roots - by denying EVERY group the power to censor its critics. It is true that some of these critics are more offensive than others. Certainly, Holocaust denial is far worse than anything in the Da Vinci Code. But "censorship envy" ensures that such distinctions are unlikely to deter the spread of repression once it has begun.
Update: It is worth noting that Cardinal Arinze - the leading would-be censor of the Da Vinci Code - is not a minor fringe figure. As the Reuters piece I linked to notes, he was viewed as a serious contender for the papacy when John Paul II died last year. And he will likely be a contender again after the passing of the current pope (who is almost 80 years old).