You can do it with the doorcard on.
Losen the door catch bolts this makes it easier because you can just let that drop down slightly so the rod comes away from the handle.
Then remove the Allen bolt you said and pull from the lock side on the handle, might be slightly tight.
It is easier with the doorcard off mind because you can see what your doing properly.