what I found that works great, is nail polish remover pads from Walmart. They are little felt pads, that serve as a rough surface that will wipe that rubber junk right off. I painted the area around the door handle and window switches, and it came off so easy!
So that's another option.