It doesn't matter so much to me. But, my preference would be for them both to be invisible, and add a "Kudos" count for users to get recognition for contributions.
Also, I think the root of the problem is that the votes are not done for the right reasons.
Upvotes should be representative of things like "I learned something new", or "That saved me a lot of time", etc, etc, and downvotes should be for things like, "I asked you to address the error in your post and you refused", or "your answer does not answer the original poster's question", etc, etc.
Too many times, the votes are more representative of fleeting emotions or revenge. (In my opinion!)