Robert Kavanagh was born around 1930 in Richmond (Staten Island), New York, the eldest child and only son of Patrick Kavanagh (a barge captain originally from Arklow, County Wicklow) and Sarah "Sadie" Rolf (b. Canada).
Growing up in Brooklyn, he was raised alongside his three younger sisters: Martha, Irene, and Barbara.
In the 1940 US Federal Census, nine-year-old Robert was attending school while living with his parents and sisters at 206 60th Street, Brooklyn. By April 1950, the family had moved to a second-floor apartment at 180 83rd Street, Brooklyn.
At age 20, Robert was unmarried and working 40 hours a week as a sales clerk in a retail candy shop, a role also shared by his younger sister Martha.