Joseph was refuting the charge of adultery, not the fact that he had "seven wives"
History of The Church 6:410-411:
I had not been married scarcely five minutes, and made one proclamation of the Gospel, before it was reported that I had seven wives. I mean to live and proclaim the truth as long as I can.
This new holy prophet [William Law] has gone to Carthage and swore that I had told him that I was guilty of adultery. This spiritual wifeism! Why, a man dares not speak or wink, for fear of being accused of this.[11]....
A man asked me whether the commandment was given that a man may have seven wives; and now the new prophet has charged me with adultery. I never had any fuss with these men until that Female Relief Society brought out the paper against adulterers and adulteresses.
Dr. Goforth was invited into the Laws' clique, and Dr. Foster and the clique were dissatisfied with that document,[12] and they rush away and leave the Church, and conspire to take away my life; and because I will not countenance such wickedness,[13] they proclaim that I have been a true prophet, but that I am now a fallen prophet.
[Joseph H.] Jackson[14] has committed murder, robbery, and perjury; and I can prove it by half-a-dozen witnesses. Jackson got up and said—"By God, he is innocent," and now swears that I am guilty. He threatened my life.
There is another Law, not the prophet, who was cashiered for dishonesty and robbing the government. Wilson Law also swears that I told him I was guilty of adultery. Brother Jonathan Dunham can swear to the contrary. I have been chained. I have rattled chains before in a dungeon for the truth's sake. I am innocent of all these charges, and you can bear witness of my innocence, for you know me yourselves.
When I love the poor, I ask no favors of the rich. I can go to the cross—I can lay down my life; but don't forsake me. I want the friendship of my brethren.—Let us teach the things of Jesus Christ. Pride goes before destruction, and a haughty spirit before a downfall.
Be meek and lowly, upright and pure; render good for evil. If you bring on yourselves your own destruction, I will complain. It is not right for a man to bare down his neck to the oppressor always. Be humble and patient in all circumstances of life; we shall then triumph more gloriously. What a thing it is for a man to be accused of committing adultery, and having seven wives, when I can only find one.
I am the same man, and as innocent as I was fourteen years ago; and I can prove them all perjurers. I labored with these apostates myself until I was out of all manner of patience; and then I sent my brother Hyrum, whom they virtually kicked out of doors.[15]
Note the rejection of the term "spiritual wifeism". Note that "spiritual wifeism" likely refers to John C. Bennett's pattern of seduction and sexual license, which the Saints were always at pains to deny.
http://en.fairmormon.org/Criticism_of_M ... Chapter_13