A passport is a very odd document. It does not belong to the bearer. It grew from a “letter of introduction” to its present form. It is a request from the home state of the traveller to the state or states to which the traveller is going, to accord the traveller respect and not to infringe his/her rights.
As an instance of this Giovanni Belzoni, an Italian strongman (literally), in 1815 procured a letter from the British consul in Naples to allow him to travel to Egypt where he explored many of the sights of Egypt and used explosives to break into the great pyramid where he wrote a message on the roof of one if its large chambers, still to be seen today.
This instance was instructive; Belzoni was a native of Padua, part of the Venetian Republic. Napoleon had destroyed the Republic and by 1815 the British dominated the Mediterranean and Egypt in particular, filling a vacuum after Napoleon defeated the Mameluke rulers of that country. Belzoni had lived in Britain for a period. So, he seemed an appropriate recipient of a letter of introduction from the ascendant power.
In modern times, refugees can get the equivalent of such a document from the United Nations Commissioner for refugees; by definition, their own state will not give them a passport.
So, what started as a handy or valuable thing to have on your travels has become an essential thing to have. The “friendly” state to which you travel will be anything but friendly if you do not have your passport.
In 1981 Philip Agee, a US citizen ran into some passport trouble; it was revoked by Alexander Haig the Secretary of State . At the time Agee was in West Germany. Agee, a former CIA agent, had engaged in a programme to publicly disclose the identities of the agents of the CIA around the world. Agee issued proceedings in a US federal court to restrain Haig in his actions. Agee won and won again on appeal. The case came on in the US Supreme court on further appeal where Haig won. (The Chief Justice, Burger, was a Nixon appointee.) The judgment recited the history of US passports; the history had commenced with the Passport Act of 1856 (a fateful and dismal year for the US).
Haig v Agee has had a significant effect in that it expanded the power of the Executive at the expense of the US Congress. In effect, the Supreme Court assigned powers to the Executive because the Congress had not expressly denied the power to the Executive. (A case, surely, from Agee’s point of view, of great unintended effects).
As for Agee, he promoted tourism to Cuba for US citizens, finally dying in 2008, just two years before Haig.