Felix was born at Český Krumlov Castle (German: Böhmisch Krumau) in Bohemia, the second

son of Prince Joseph of Schwarzenberg (1769–1833) and his wife Pauline of Arenberg.

The House of Schwarzenberg was one of the most influential Bohemian noble families; his elder
brother Prince Johann Adolf II of Schwarzenberg later initiated the building of the Emperor Franz
Joseph Railway line from Vienna to Plzeň (Pilsen), while Felix' younger
brother Frederick became Archbishop of Salzburg in 1835 and Archbishop of Prague in 1849.
The nephew of Prince Karl Philipp of Schwarzenberg (1771–1820), commander of the Austrian
armies in the last phases of the Napoleonic wars, Schwarzenberg after a short military interlude
entered the diplomatic service, where he became a protégé of State Chancellor Prince Klemens
von Metternich and served in several Austrian embassies [2] at Saint Petersburg, London,
Paris, Turin, and Naples. During his time as a London attaché in 1828 he had an affair with Jane
Digby, whom he deserted after causing her then-husband – Edward Law, 1st Earl of
Ellenborough – to divorce her, and making her pregnant. This episode led to the nickname of
"Prince of Cadland" being applied to him in London.