Analyzing the Office of the Historian

Hello, welcome to The Aggregate, the newsletter on the in depth analysis on topical yet unusual datasets and technical topics. If you want to sign up, a button to do that is below, or just read on!

The Weekly Data:

Through the Office of the Historian, it is possible to look at a variety of declassified documents from various presidential administrations. For a class I took several years ago on the history of US diplomatic relations, the Office of the Historian provided very helpful in providing useful research documents, especially regarding Sino-American relations in the 1960’s.

Due to the number of documents provided in the dataset, I wondered if it was possible to analyze the various tables and lists contained on the website, to see if there are any interesting trends. Through some web scraping and functional programming, it is possible to grab a variety of data from The Office of the Historian to analyze various trends in the documents that the organization publishes.

To start, I wanted to see how many documents were fully available versus documents that were not available per year. Conveniently, The Office of the Historian provides various tags regarding the state of a given document. Available means that a document was Published and available in Full Text while everything else was classified as unavailable.

After the Nixon and Ford, the amount of fully published documents plummets, with the none being around after the late 80’s. You can see however one of the stranger effects of Watergate and Nixon is just how many documents under his presidency that are published.

In fairness, many of the Nixon documents, especially the ones regarding his trip to China, are highly entertaining. There are many conversations between Nixon, Kissinger, Zhou Enlai, and Mao that are actually amusing to read about, along with being rather informative of the state of Sino-American relations during that period in the Cold War.

Next, I wanted to see what the most common president/administration for a document was to be published in. Conveniently, that information can be grabbed for the URL of a given presidents page, and then joined as a column to the respective document.

Unsurprisingly, the specter of Nixon appears again as the Nixon-Ford administration has the most published documents overall.

Additionally, each document has a set of tags associated with them. These tags are either topics like trade, or the name of countries such as United Kingdom, or Germany. In this case, I filtered out all the non-country tags, to see what countries appeared most often as a tag within the series of documents.

Again, the specter of Richard Nixon rises again due how many documents Nixon has. While Western Europe appears plenty of times, one of the most common tags to appear on a document is China.

What is strange is that it seems that the tags have been modernized. There is no East/West Germany tag, along with a lack of a Soviet Union tag as well. This probably is one of the reasons why outside of Russia, the other former Soviet Bloc countries are not represented much in the Office of the Historian’s tag system.

Now, some links…

Other Internet: Squad Wealth

SQUADS HAVE EXISTED FOR THOUSANDS OF YEARS as vital forms of social and economic organization. Thanks to group chats and a wave of private online social platforms, squads are reemerging today as a potent cultural force that rejects a strictly individualist market philosophy. Squads play a key role not only in internet community dynamics but in emerging economic networks. Hawala, chit funds, chamas and other forms of P2P savings or credit associations are notable precursors to the kinds of financial relationships we anticipate decentralized cryptocurrency protocols will soon enable. These proto-squads are an example of Coaseian logic at play—that is, strong internal coordination decreases transaction costs, enabling greater productive capacities and financial opportunities as a group.

Carroll Quigley: Introduction to Weapons Systems and Political Stability: A History

Just as our ideas on the nature of security are falsified by our limited experience as Americans, so our ideas are falsified by the fact that we have experienced security in the form of public authority and the modern state. We do not easily see that the state, especially in its modern sovereign form, is a rather recent innovation in the experience of Western civilization, not over a few centuries old. But men have experienced security and insecurity throughout all human history. In all that long period, security has been associated with power relationships and is today associated with the state only because this is the dominant form which power relationships happen to take in recent times. But even today, power relationships exist quite outside of the sphere of the state, and, as we go farther into the past, such non-state (and ultimately, non-public) power relationships become more dominant in human life. For thousands of years, every person has been a nexus of emotional relationships, and, at the same time, he has been a nexus of economic relationships. In fact, these may be the same relationships which we look at from different points of view and regard them in the one case as emotional and in another case as economic. These same relationships, or other ones, form about each person a nexus of power relationships.

EconTalk: Martin Gurri on the Revolt of the Public

I'm sorry, 2011. Thank you. The year 2011. 2011 is a year I call the Phase Change Year, where it really showed the effect of this tide of information could affect power. And you had, of course, the Arab Spring in the Middle East, probably misnamed. You had the Indignados in Spain. You had a revolt called the 10 People Revolt or Social Justice Revolt in Israel. You had the Occupiers here in the United States. And, these all had similar origin. So the question, now, was what was going on? What caused these eruptions from below?

And, to my thinking, it has to do with the kinds of institutions that we have inherited from the 20th century, from the industrial age. They're all--how many people are aware of Frederick Taylor? He's sort of a forgotten figure in history. But he was sort of the prophet of industrialism and scientific management. And, if you read his writings, everything happens from the top down: the top manager figures everything is going to happen, all the tools that you need, and essentially what everybody, every layer below you--and there are many, many layers below you--is going to do. Everything is scripted.

Paul Berman (Tablet): Lynching and Liberalism

The larger question lurking behind the debate over “cancel culture” is the one about liberalism—to wit, what is liberalism, anyway? And why should we care about it? I signed the Harper’s “Letter on Justice and Open Debate” last month because it beams a clarifying spotlight on the cancel phenomenon, and on its progressive or left-wing version, in particular: “an intolerance of opposing views, a vogue for public shaming and ostracism, and the tendency to dissolve complex policy issues in a blinding moral certainty.” But I signed also because, in the course of making its point, the Harper’s letter scatters a few additional illuminations on the larger background question, as well. It is a nuanced letter. Its tone seems to me agreeably old school, recognizable from liberal debates and manifestos of long ago—which, to be sure, the letter’s detractors may regard as one more reason for dismissing the debate and the document and the signatories and their worries. But sometimes it is good to be reminded of times gone by.

David Kamp (Vanity Fair): The Birth of Bond

Enter Sean Connery, dark hair slicked with pomade, eyes locking hungrily upon a beautiful green-eyed girl. Her return glance leaves no doubt—the feeling is mutual. His slouch and casual banter exude languor and nonchalance, but there's an undercurrent of coiled menace to this man, as though he might, at any moment, spring into table-overturning, crockery-shattering action.

Except nothing of the sort happens. Instead, the other fellow in the scene cuts the tension by taking out his fiddle and favoring the room with a jaunty tune learned, he says in a stagy brogue, "in the old ruins on the top of Knocknasheega!"

This isn't a James Bond picture. It is 1959, and Connery is putting in time in a cornball live-action Disney feature called Darby O'Gill and the Little People. He's the second male lead, billed beneath not only Albert Sharpe, the elderly Irish character actor in the title role—a kindly farmhand who sees leprechauns—but also the green-eyed girl, the ingenue Janet Munro. Though verily pump-misting pheromonal musk into the air, to a degree unmatched before or since by any actor in a Disney family movie, Connery is still a jobbing scuffler, not a star. He has no idea of what lies in store for him.


My apologies for the newsletter being intermittent. With everything going on it’s been hard coming up with fresh data and material to write about. Also work has been pretty busy, so my mind has been focused elsewhere.

On the bright side, I been able to catch up on my reading. I am currently reading some Russian absurdist writing by Daniil Kharms.


Thanks for taking the time to read this, I will be back next soon! In the meantime, you can follow me on Twitter or reach out via email.