Paul Newman Reveals How Wife Joanne Woodward Made Him a 'Sexual Creature' in New Posthumous Memoir

"I want to leave some kind of record that sets things straight. Pokes holes in the mythology that's sprung up around me, and keep the piranhas off."

With that, Paul Newman, one of Hollywood's most iconic and charismatic stars, set out in the mid '80s to compile an oral history project about his life.

The only rule for all interviewees, including himself, was to be "completely honest."

So when it came time to discuss how he became a sex symbol, the star of such classics as Hud, Cool Hand Luke and Butch Cassidy and the Sundance Kid, gave all the credit to his second wife, Joanne Woodward. "Joanne gave birth to a sexual creature," says Newman. "We left a trail of lust all over the place. Hotels and public parks and Hertz Rent-A-Cars."

It's one of the many astonishingly intimate stories told in his posthumous memoir, Paul Newman: The Extraordinary Life of An Ordinary Man, excerpted in this week's PEOPLE.

The new book, based on the long-lost interviews that were located a few years ago, describes his years as an insecure adolescent from Shaker Heights, Ohio, who was once so small he had to get permission to play on the high school football team. His confidence was nonexistent. Especially with women. "I felt like a goodman freak," he says. "Girls thought I was a joke. A happy buffoon."

But that would all change once he met Woodward in 1953. "I went from being not much of a sexual threat to something else entirely," recounts the actor.

Never miss a story — sign up for PEOPLE's free weekly newsletter to get the biggest news of the week delivered to your inbox every Friday.

At the time, they were both understudies in the Broadway play Picnic. He was also married to his first wife Jackie Witte. Torn between his wife and family (they had three young kids, Scott, Susan and Stephanie) and the attraction toward Joanne, they had a tumultuous affair. A time he describes as "brutal in my detachment from my family."

He eventually divorced Witte in 1958 and married Woodward.

He describes coming home one night to their new home in Beverly Hills where she had fixed up a room off the master bedroom with a "thriftshop double bed" and a coat of fresh paint.

"'I call it the F--- Hut,' she said proudly. It had been done with such affection and delight. Even if my kids came over, we'd go into the F--- Hut several nights a week and just be intimate and noisy and ribald."

They had three daughters, Nell, Melissa and Clea, and moved to Westport, Connecticut, where they lived in a rambling house full of writer friends and books. But as both he and Woodward make clear — as storied as their romance became — it was much more complicated. Often due to Newman's heavy drinking.

"Joanne and I still drive each other crazy in different ways," recounts Newman in the book. "But all the misdemeanors, the betrayals, the difficulties have kind of evened themselves out over the years."

Says their daughter, Clea, "They fought and it could be dramatic, but they also fought really hard to stay together."

"They didn't walk," she adds. "There were times it was pretty close but they worked hard at it. Ultimately they came together."

Now 92, Woodward, who has Alzheimer's, lives quietly at home on the property they long shared. But their love story comes alive once again in the new book. As well as the actor's humor, his intellect and his drive to do good in the world, especially with his philanthropic work, launching a network of camps for seriously ill children, which he considered his greatest legacy.

The new book is a chance to see him as he saw himself. Says Clea: "He was doing this for us so he could clear up the fairy tale and tell the real story."

Paul Newman: The Extraordinary Life of An Ordinary Man is out Oct. 18.