I think, that instead of simply displaying the upvotes and downvotes seperately, we can have the upvotes in green, the downvote in red, and a "+" before the upvote number, and a "-" before the downvote number.
This can be done through a short script:
$( ".votes-up" ).before( "+ " );
$( ".votes-down" ).before( "- " );
and a short style:
span.votes-up, span.qa-upvote-count-data {color:green;} span.votes-down, span.qa-downvote-count-data {color:red;}
span.qa-upvote-count-pad, span.qa-downvote-count-pad {color:transparent; font-size:150%}
div.qa-voting {padding-top:10px;}
This will make the upvotes and downvotes look more clear, as to which is which. Here's a screenshot of this very question with this enabled: