He lived and collaborated with Diana Molinari, who featured in many of his works of the 1970s. After that Rivers lived with Sheila Lanham, a Baltimore poet. In the early 1980s, Rivers and East Village figurative painter Daria Deshuk lived together and in 1985 they had a son, Sam Deshuk Rivers. At the time of his death in 2002, Jeni Olin, a poet, was his companion.