Turns out I rely a lot on the presence of the arrows to remember what I've already upvoted. So you're likely to have people who visit more often contributing more upvotes.
Me too. And I found the lack of uparrows annoying at first.
On the other hand, this gives the community more power to control rankings in general. If you feel like a submission deserves more time on the front page, you can make it so.