Most superficial and ridiculous conversations about US politics in the middle of the 19th century sort of lose track of how, er, multi-faceted they were. During Reconstruction, the States which had Lost were not participating in national politics: they weren't represented in Congress. There was an ambiguous and contested relationship between the occupying Army and the state legislatures. There was debate about who should be allowed to vote. There were debates about who should be prosecuted and how. Etc.
You can think of roughly three interests participating in these debates: freedmen and that fraction of Republicans which thought they ought to be citizens and be allowed to vote, other Republicans who were incredibly pissed about the war and wanted to make sure Something Happened as a result, and a third group that hadn't been all that hot about abolitionism in the first place, Andrew Johnson appears to have fallen somewhere on the Democratic side of that last group. His horse in the race was the poor, white, scrappy farmer, and he was a lot more worried about freedmen voting for their former masters who they were still farming for just under a technically different arrangement, than about anything we might recognize as justice.
Andrew Johnson really got into it with his Congress. This seems sort of surprising, altho if you give any thought to how the Whigs went down, and the various assemblages that cratered until Lincoln finally got the nod (in exchange for Committing to Not Support Abolition), maybe not so surprising at all. But it's so odd: basically a completely one party setup, and Congress kept overriding his vetoes -- and then they impeached him.
Andrew Johnson switched parties.
If you are wondering why they didn't just find a way to get rid of him, apparently the Vice President favored Women's Suffrage, which is about the only thing all these white men could agree to be in opposition to.
The wikipedia article on Andrew Johnson has (at the time I read it today) a really great section on succeeding generations of historians and what they thought of him. In many ways, I feel bad for _anyone_ who was involved in politics through the middle decades of the nineteenth century (I mean, even more so than most other times). Basically, if you could convince other people to agree with you on anything at the time, there was guaranteed to be a whole wave of people then or later that would think you were Evil Incarnate for whatever it was. Given that Johnson was from Tennessee, and assuming he wanted to actually go home again, his politics don't seem that inept to me. At all. They really quite liked him when he went home. He got to be a Senator after a bit.
But whenever I think about what he did from the perspective of liberty, justice and Righting of Wrongs, well, he still doesn't seem inept. Evil, and damn good at it.
I'm not seeing the inept theory, I guess.
I also feel like I can see the underlying theme that was appealing to Fillmore and Johnson. They were both fundamentally status quo guys, who were freaked out about immigration, and freaked out about people who had historically been at the bottom of the pile somehow being elevated. Johnson had his own group that he wanted to succeed, but his was a weak populism, at best. They were absolutely centrist beings in a hyper-partisan time: for farm, family, and a white Protestant God.
This may be the Best Argument Ever against centrism, but it sure does a great job of explaining my mid- to late- 19th century ancestors living in what is now the Midwest. For them, it wasn't about being for or against slavery. It was about being _for_ the nuclear family climbing the economic ladder by settling new land.