Do the fumes smell oily? Maybe you have a small oil leak from the engine and the hot engine burns the leaked oil that dribbles down the hot engine and makes a bit of oil smoke. This could get in through the heating/ventilation.
I doubt that it's exhaust fumes because you would hear if your exhaust had a hole in it.