Handrail mount swivel assembly for 1" pipe