The 'location' shots of Belgravia are in fact CGI altered views of Edinburgh New Town; stucco has been added to the natural stone of the original.
Though it may be unfamiliar to American audiences, "Belgravia" is the name of a prosperous and exclusive Central London neighbourhood. It was a frequent setting for the novels of Anthony Trollope, who often wrote about nineteenth-century English society and class differences. Belgravia was also the setting for the 1970s British TV series Upstairs Downstairs, which was a major influence on Julian Fellowes's earlier projects Gosford Park and Downton Abbey.
The Duchess of Richmond's ball was a real event, hosted by Charlotte, Duchess of Richmond in Brussels on 15 June 1815, the night before the Battle of Quatre Bras. Charlotte's husband 4th Duke of Richmond, was in command of a reserve force in Brussels, which was protecting that city in case Napoleon Bonaparte invaded.
The Duchess of Richmond was based on a real person, her sister Lady Brockenhurst was fictional, The Duchess of Richmand was daughter of the 4th Duke of Gordon and the ambitious Jane Maxwell. Hence Lady Brockenhurst was said to be daughter of a Duke in the show.
The name of minor character, Jane Croft, is a compilation of two characters names in Gosford Park (2001), also written by Julian Fellowes. Below stairs, Eileen Atkins played Mrs. "Croft" & Helen Mirren played her sister, "Jane" Wilson, hence "Jane Croft" in Belgravia.