Mansion Misery! Holly Madison Exposes Shocking Reasons She Lived In 'Fear' Alongside Former Boyfriend Hugh Hefner

After spending years being part of the Playboy Mansion, Holly Madison, 35, is finally exposing her secret struggle back when she was Hugh Hefner's girlfriend. Especially the moment she knew she had to plan her escape.

"The Mansion was all about conformity and following the rules. I didn't like being a clone," Holly told PEOPLE magazine.

Holly admits that she was convinced to move into the Playboy Mansion when she was only 20 years ago, where she would soon turn into another girlfriend for Hugh, who was 74 at the time.

"There was a lot of fear because there was so much competition among the women. I learned not to confide in certain people — there were sharks in the water," Holly said.

So when Hugh's two other girlfriends, Kendra Wilkinson and Bridget Marquardt, decided to move out of the mansion, Holly admitted she was finally getting Hugh all to herself.

But the feeling quickly faded when Holly recounts that she "started seeing Hef's true colors" and "couldn't make excuses for him anymore."

"I was miserable. But I couldn't leave, because I felt like I'd be so judged for having lived there. I had a huge scarlet letter on my forehead," she said.

"He accused me of cheating on him — and I didn't. I realized how much he loved the drama and I was like, I can't do this. I woke up one day and it was as if the spell was broken. I had to run," Holly claimed.

When Hugh and Holly broke up in 2008, she told the publication that it was like starting her life and career from scratch. "I was never ready to answer questions about my past and that made relationships really tricky. I was embarrassed — it was like the elephant in the room."

But Holly wed her husband in 2013, which changed life for the former Girls Next Door reality star — especially later giving birth to her daughter, Rainbow. "I definitely had a period of time where I thought, I'm going to have to be happy without that picture of marriage and family. I felt completely alone."