The real problems with internet voting have almost nothing to do with personal security. I could care less if someone was able to see whom I voted for. Most people can already figure that out just be looking at the public donation records of political contributions. Legitimate hacker attempts at voter fraud would never happen on a per person basis, there would be too many connections to alter to achieve any significant fraud. Plus why attempt "voter fraud" when you can achieve "election fraud" so much more easily. Election fraud can already be effectively perpetrated downstream of the individual voting connections directly at the individual servers or tallying machines. In fact there is evidence that this already happened in Ohio during the 2004 election when election totals were diverted to servers in Tennessee owned by a GOP backed IT firm prior to their final tabulation at the Secretary of State's office in Ohio.
After what we have been through in 2000 and 2004 nationally and in the many regional elections that either had digital tallies allegedly altered or in some locations where a recount had to be administered, having an all online system of voting seems like a disaster to me. It opens up so many new avenues of fraud that would be completely un-transparent to the average citizen or election official.
First off every Secretary of State position is an elected political partisan and they control how the digital voting hardware is implemented and how it is maintained in each state. Ultimately only a handful of people truly understand how these machines work and how they will collect and tally the votes. Generally the technicians of these machines are employees of large multi-national corporations. It doesn't take a rocket scientist to see how ulterior motives and political alliances between politicians and corporations could quickly and easily subvert the collective will of the people of this country. Instead of having to subvert the tallies at hundreds of polling places you could with internet voting only have to target the single location where all the votes would be ultimately routed and recorded. There are so many things a good programmer could do to an electronic tallying machine that would be virtually untraceable and could alter votes to any result that they would see fit.
Having just gone through the recount in Minnesota I have to say our system functioned about as good as it can get precisely because we still have a paper trial that can be used should there be any dispute of fraud. The only improvement that I think is essential to the system is better campaign finance reform, better "get out the vote" efforts, better early registration efforts, and ultimately Instant Runoff Voting (IRV).
Eliminating the paper trail for something as critical as a democratic election should not ever be considered in my opinion. Instead of further streamlining and centralizing the vote counting and tallying we need to de-centralize the process and put the counting and tallying back into the hands of the many and not the hands of the few. There is not a single case of "voter fraud" that has been shown to alter the results of an election but there are plenty of allegations of "election fraud" in recent years that if ever proven would be shown to have altered the final outcome of the election.