To help simplify my life and make it less stressful, I have canceled all of my credit cards except for one, Chase Freedom. The two cards that I canceled (mainly because I wasn't using them) were Discover and Costco's American Express.
The customer service of these two companies were very pleasant and the Discover company even told me that this may affect my credit score by closing the card and suggested I maintain it even though it's inactive. They also informed me the obvious that there is no annual charge and that it wouldn't be closed due to inactivity unless I was at risk and since I wasn't, they would keep it open for me. Thanks Discover, but no thanks.
I have a lot less to worry about now that they are out of my wallet and mind.