Monoplies tho' in certain cases useful ought to be granted with caution, and guarded with strictness agst abuse. The Constitution of the U. S. has limited them to two cases, the authors of Books, and of useful inventions, in both which they are considered as a compensation for a benefit actually gained to the community as a purchase of property which the owner might otherwise withold from public use. There can be no just objection to a temporary monopoly in these cases: but it ought to be temporary, because under that limitation a sufficient recompence and encouragement may be given. The limitation is particularly proper in the case of inventions, because they grow so much out of preceding ones that there is the less merit in the authors: and because for the same reason, the discovery might be expected in a short time from other hands.
Monopolies have been granted in other Countries, and by some of the States in this, on another principle, that of supporting some useful undertaking until1 experience and success should render the monopoly unnecessary, & lead to a salutary competition. . . . But grants of this sort can be justified in very peculiar cases only, if at all; the danger being very great that the good resulting from the operation of the monopoly, will be overbalanced by the evil effect of the precedent; and it being not impossible that the monopoly itself, in its original operation, may produce more evil than good.
In all cases of monopoly, not excepting those specified in favor of authors & inventors, it would be well to reserve to the State, a right to terminate the monopoly by paying a specified and reasonable sum [to the holder.] This would guard against the public discontents resulting from the exorbitant gains of individuals, and from the inconvenient restrictions combined with them.
Update: Eugene usefully points out that Madison referred to "literary property" in his Vices of the Political System of the United States. I think this is a fair point and a useful correction. Indeed re-reading the quote, I notice that Madison refers to the monopoly grant for limited times as "compensation for a benefit actually gained to the community as a purchase of property which the owner might otherwise withhold from public use." So let me revise my point to the more limited observation that Madison saw these grants to be a narrow exception to the general objection to monopolies, that were dangerous and should be carefully limited. And however these limited grants are labeled, they do not have all the attributes we normally associate with tangible property. Given that they typically involved interference with the freedom of others to trade what was theirs, the sorts of monopolies to which the Founders strongly objected are not the same as the "monopoly" one has on one's own tangible property. And it is to this objection, Madison made a narrow exception to provide a public subsidy for writers and inventors.