Makes sense.Brake dust is made up of both brake pad friction material particles AND brake disc particles - Brake discs for cars are all made of cast iron (except exotic carbon ceramic discs).
Most European cars (including All VAG group) use a low metallic friction material type which is black in colour - so I suggest the brown colour you are seeing isn't brake pad dust. I suggest it is brake disc dust which is iron particles, which will be very hot and embed them selves in the lacquer in your wheels.
this will be especially more prevalent with performance models which use a more aggressive brake pad friction material - which will wear the discs down quicker creating more iron dust.
@RUM4MO; might be worth investing in some fallout remover, such as Car Pro Iron-X or Valet Pro Dragon’s Breath.