I would imagine that most people would look at me (or my website full of random ideas and projects), and say that I have too much time on my hands, when in reality I am probably just more ambitious, have different priorites, and sleep less than most.
I do hear the phrase quite a bit, and hear it applied to people who are doing important, wonderful, and just interesting things. I recently sent a video to a friend (view it here) where their immediate comment was "man, these guys have waaaay too much time on their hands". These guys had spent countless hours filming and creating beautiful art, and that was the first comment that came to my friends mind—not how beautiful it was, how interesting it is, how dedicated the people making the video are—it was, the dreaded "too much time" statement! People who create are often times the targets of these lines. People choosing to spend their precious time doing things that interest them, instead of the normal, boring, societal norms.
Perhaps this phrase is more telling of the people who let it spill from their mouths, than the people they are targeting. Often times, people will spend countless hours, sitting in front of a television set, yet those who are creating, innovating, or just exploring have too much time on their hands! Maybe those who are judging feel bad that they don't actually do anything of significance.
Just because something takes time and success isn't guaranteed, or even sought, doesn't mean that someone has too much time.