The title of Charles Grodin's first autobiography, "It Would Be so Nice If You Weren't Here..." was taken from an incident that happened on the set of this movie, as he explained in his introduction: "Candy Bergen and I were filming the movie 11 Harrowhouse in a castle outside London. We were sitting in a room off of the main hall where the cameras were being set up. After a few minutes an Englishwoman appeared. I don't know who she was, but she acted as though she had a Duchess-or-something title. She said: 'Did someone ask you to wait in here?' 'No', we answered, a bit taken aback. She responded: 'Well, it would be so nice if you weren't here.'"
In his book "It Would Be so Nice If You Weren't Here..." Charles Grodin explained that it took him a long time to find the right project after his first successful starring role in The Heartbreak Kid (1972). When he finally chose this movie, the studio offered him approval of the rest of the cast and screenplay input. But when the film flopped and Grodin's performance was heavily criticized, he found it difficult to get another acting job for the next two years.
While filming the scene in which Chesser (Charles Grodin) and Watts (James Mason) are meeting on a park bench, Grodin noticed that Mason was avoiding eye contact with him, and instead was looking straight ahead. He then decided to do the opposite and never take his eyes off of Mason during the scene.
Trevor Howard had a reputation of having a drinking problem, so Charles Grodin requested a meeting before they began working on this movie. Candice Bergen and director Aram Avakian, who had already began working on this movie, also joined them. Howard had been left to wait on his own in a cocktail bar, and when the others arrived they thought he was making a joke when he appeared to be intoxicated. Unfortunately, he wasn't. However, after this incident, Howard kept his word and never drank during filming days.
The first sneak preview in Los Angeles received a fifty percent unfavorable response. After this, Charles Grodin, Elaine May, and Herb Gardner came up with the idea to add narration by Chesser (Grodin) to the movie, which featured a lot of long silences in its original cut.