Duality of insanity

The world sees "Insanity" as a negative state of mind and those exhibiting it as "Mentally unsatble". However, to a "Mad man" the world is "mad" and he is the only "sane one". If we were all "mad" and there was just one "sane" person we would consider ourselves "sane" and the "sane" person "mad". Who then has the right to judge "sanity", is it the "mad" person or the "sane" person who, technically, have equal rights. Afterall we are in some context either one of the two or both at once.