SiFi about prisoners on a planet mining precious jewels underground

[Pashasan]

Ok so I read this a long time ago. Never got to finish it. I had thought it was Niven but I can’t find it in his catalog. It’s about this guy who is sent to a prisoner colony. It’s on a planet somewhere off earth. He’s a navy man and is very adept at hand to hand combat. Once at the prison planet they put then in a prison colony that is underground. There the prisoners mine a precious jewel. There is another layer to this prison where the prisoners keep the really crazy prisoners who do most of the actual mining of the jewels. Trading them food and provisions for there jewel harvest. When the navy man gets on the bad side of the head prisoner he is lowered into this second layer of the prison where he meets a girl and a "farmer" who is even better at hand to hand combat. They go on to have crazy adventures in the subterranean prison, ect, ect, ect. Ok so this is a bit fractious and poorly summarized. But is a crazy story and ive never been able to find it again. I thought it was one of the great classic SiFi writers, Polk, Niven, or someone of there stature but that might be wrong. Anyone got any ideas?