I agree. In the books, Reacher's physical appearance is often pivotal in how other characters respond/react to him. Do they back down and run? Do they get all macho and escalate a confrontation? Do they dismiss him as a thug? These all have to do with his physical appearance. Not a small point.Reacher's size and appearence is very much the point in the novels.
To cast Cruise in the part was just plain silly.
I would have been fine with the Tom Cruise movie if he wasn't supposed to be playing Reacher.