One sunny April day I set out to find Charles. Rue de Monceau is a long Parisian street bisected by the grand boulevard Malesherbes that charges off towards the boulevard Pereire. It is a hill of golden stone houses, a series of hotels playing discreetly on neoclassical themes, each a minor Florentine palace with heavily rusticated ground floors and an array of heads, caryatids and cartouches. Number 81 rue de Monceau, the Hôtel Ephrussi, where my netsuke start their journey, is near the top of the hill. I pass the headquarters of Christian Lacroix and then, next door, there it is. It is now, rather crushingly, an office for medical insurance.

It is utterly beautiful. As a boy I used to draw buildings like this, spending afternoons carefully inking in shadows so that you could see the rise and fall of the depth of the windows and pillars. There is something musical in this kind of elevation. You take classical elements and try to bring them into rhythmic life: four Corinthian pilasters rising up to pace the façade, four massive stone urns on the parapet, five storeys high, eight windows wide. The street level is made up of great blocks of stone worked to look as if they have been weathered. I walk past a couple of times and, on the third, notice that there is the double back-to-back E of the Ephrussi family incorporated into the metal grilles over the street windows, the tendrils of the letters reaching into the spaces of the oval. It is barely there. I try to work out this rectitude and what it says about their confidence. I duck through the passageway to a courtyard, then through another arch to a stable block of red brick with servants’ quarters above; a pleasing diminuendo of materials and textures.

A delivery man carries boxes of Speedy-Go Pizza into the medical insurers. The door into the entrance hall is open. I walk into the hall, its staircase curling up like a coil of smoke through the whole house, black cast iron and gold filigree stretching up to a lantern at the top. There is a marble urn in a deep niche, chequerboard marble tiles. Executives are coming down the stairs, heels hard on marble, and I retreat in embarrassment. How can I start to explain this idiotic quest? I stand in the street and watch the house and take some photographs, apologetic Parisians ducking past me. House-watching is an art. You have to develop a way of seeing how a building sits in its landscape or streetscape. You have to discover how much room it takes up in the world, how much of the world it displaces. Number 81, for instance, is a house that cannily disappears into its neighbours: there are other houses that are grander, some are plainer, but few are more discreet.

I look up at the second-floor windows where Charles had his suite of rooms, some of which looked across the street to the more robustly classical house opposite, some across the courtyard into a busy roofscape of urns and gables and chimneypots. He had an antechamber, two salons – one of which he turned into his study – a dining-room, two bedrooms and a ‘petite’. I try to work it out; he and his older brother Ignace must have had neighbouring apartments on this floor, their elder brother Jules and their widowed mother Mina below, with the higher ceilings and grander windows and the balconies on which, on this April morning, there are now some rather leggy red geraniums in plastic pots. The courtyard of the house was glazed, according to the city records, though all that glass is long gone. And there were five horses and three carriages in these stables which are now a perfect bijou house. I wonder if that number of horses was appropriate for a large and social family wanting to make the right kind of impression.

It is a huge house, but the three brothers must have met every day on those black-and-gold winding stairs, or heard each other as the noise of the carriage being readied in the courtyard echoed from the glazed canopy. Or encountered friends going past their door on the way up to an apartment above. They must have developed a way of not seeing each other, and not hearing each other, too: to live so close to your family takes some doing, I think, reflecting on my own brothers. They must have got on well. Perhaps they had no choice in the matter. Paris was work, after all.

The Hôtel Ephrussi was a family house, but it was also the Parisian headquarters of a family in its ascendancy. It had its counterpart in Vienna, the vast Palais Ephrussi on the Ringstrasse. Both the Parisian and Viennese buildings share a sense of drama, of a public face to the world. They were both built in 1871 in new and fashionable areas: the rue de Monceau and the Ringstrasse were so of-the-minute that they were unfinished, untidy, loud and dusty building sites. They were still spaces that were inventing themselves, competitive with the older parts of town with their narrower streets, and spikily arriviste.

If this particular house in this particular streetscape seems a little stagey, it is because it is a staging of intent. These houses in Paris and Vienna were part of a family plan: the Ephrussi family was ‘doing a Rothschild’. Just as the Rothschilds had sent their sons and daughters out from Frankfurt at the start of the nineteenth century to colonise European capital cities, so the Abraham of my family, Charles Joachim Ephrussi, had masterminded this expansion from Odessa in the 1850s. A true patriarch, he had two sons from his first marriage, Ignace and Léon. And then when he remarried at fifty he had continued producing children: two more sons, Michel and Maurice, and two daughters, Thérèse and Marie. All of these six children were to be deployed as financiers or married into suitable Jewish dynasties.

Odessa was a city within the Pale of Settlement, the area on the western borders of imperial Russia in which Jews were allowed to live. It was famous for its rabbinical schools and synagogues, rich in literature and music, a magnet for the impoverished Jewish shtetls of Galicia. It was also a city that doubled its population of Jews and Greeks and Russians every decade, a polyglot city full of speculation and traders, the docks full of intrigues and spies, a city on the make. Charles Joachim Ephrussi had transformed a small grain-trading business into a huge enterprise by cornering the market in buying wheat. He bought the grain from the middlemen who transported it on carts along the heavily rutted roads from the rich black soil of the Ukrainian wheat fields, the greatest wheat fields in the world, into the port of Odessa. Here the grain was stored in his warehouses before being exported across the Black Sea, up the Danube, across the Mediterranean.

By 1860 the family had become the greatest grain-exporters in the world. In Paris, James de Rothschild was known as the le Roi des Juifs, the King of the Jews. The Ephrussi were les Rois de Blé, the Kings of Grain. They were Jews with their own coat of arms: an ear of corn and a heraldic boat with three masts and full sails. Their motto, Quod honestum, unfurled below the ship: We are above reproach. You can trust us.

The masterplan was to build on this network of contacts and finance huge capital projects: bridges across the Danube, railways...