RTL-SDR & Dump 1090 for Data Analysis
|Aug 14, 2020|
Hello, welcome to The Aggregate, the newsletter on the in depth analysis on topical yet unusual datasets and technical topics. If you want to sign up, a button to do that is below, or just read on!
After attending a workshop at NICAR several months ago on flight tracking to track aircraft for journalistic reasons, from tracking dictators to seeing where Jeffery Epstein’s plan’s flew, I realized I wanted to do a project regarding flight data. I realized I could do this recently when after I bought a RTL-SDR + an antenna, I could use these tools combined with Dump 1090 to capture and analyze flight data in real time.
However, one flaw is eventually, one’s own work computer has to be turned … off. I had a Raspberry Pi on hand, so I got the final parts required (HDMI cable, SD chip with NOOBs installed, a charger, and keyboard to get everything set up. And wow you get quite a bit of information from running dump 1090
How to do it
For setting up the Raspberry Pi, I followed the instructions here. While you don’t need to do it, it will make your life a lot easier if you enable SSH (Secure Shell) on the Raspberry Pi. Considering the instructions to set up Dump 1090 on Linux (Raspberian) to be specific, involve quite a bit of copying and pasting, I advise that you do this. As for the antenna, as I mentioned above, a RTL-SDR is required, I suggest the model mentioned here.
I followed the instructions here to set up Dump 1090 on the Raspberry Pi. While it was written a few years ago, the instructions still work just fine and baring any surprises once everything is installed one should be able to run Dump 1090.
Finally to extract the data, one needs to ensure that the
net argument is flagged when working with Dump 1090, to ensure one can grab the data from the network specifically
localhost 30003 to be written to a CSV file or database of any kind.
So, What is All this Stuff Exactly?
A SDR is a software radio scanner that let’s one scan live radio signals without radio, and with other programs, do other tasks such as receive satellite imagery, decode maritime boat positions, and far much more.
Dump 1090 is a program by Antirez (of Redis fame) that let’s one use a RTL-SDR device to grab and detect mode S data from aircraft. Planes in the majority of the world are required to be equipped with mode S transponders, meaning anyone in practice can grab the signal to see where planes are going. This makes it possible for example, to track the private planes of dictators or those hiding their flight from public view for various reasons. The data provided ranges from the ICAO address, a unique address for a aircraft, along with statistics such as altitude, speed, latitude, longitude, and quite a few others.
A Raspberry Pi is a type of computer meant to be an introduction to computer science by providing an introductory experience in working with a computer from a more hands on perspective, from working more with the command line to setting up key hardware. A large amount of hobbyist and commercial activity has sprung around Raspberry Pi’s in the past few years, as they are highly versatile. Due to requiring low power and by design, it is very easy to run a Raspberry Pi “headlessly” (without a user interface), as one can SSH into the Raspberry Pi and work with it from another computer. By having a computer and as a result a server running 24/7, many programs that require to be run either constantly or run on a scheduled basis are possible. Additionally, by being a physical computer versus just being a computer in the cloud, one do additional hardware work to expand the Raspberry Pi’s functionality.
Having collected several hours’ worth of flight data, there are quite a few different forms of analysis one can perform.
For example, one can collect the altitude of a given flight while it is in range, and see if it is ascending, descending, or just staying at a consistent altitude.
Using Leaflet to grab Open Street Map tiles, it becomes pretty easy to visualize over a decently large radius the flight paths of the flights passing by. For example, one can clearly see a flight fly out of La Guardia airport. More impressively, for such as simple antenna, it was able to get the Mode S data from planes across the Long Island Sound.
Additionally, through the flight codes that each flight has, it is possible to check what were the most common airlines that flew in the area. Note that for this analysis, any flight that lacked the flight code was dropped.
Planes that start with
n as the first letter in their code are private planes. These flights while not available to see existing on the website of JFK or a regular airport, due to the work that other hobbyists do through sites like flightaware, are possible to see who nominally owns them. Given my location, the private flights unsurprisingly included several from various small airports to Montauk and Martha’s Vineyard. While some hobbyist planes are found, many of the models are private jets flying off somewhere for business related reasons.
What’s more interesting is the long tail, so to speak, on airlines. While the major airlines are indeed present, the sky is filled with many smaller airline companies and private planes flying off to places for various reasons.
Now, some links…
What We are Going to Talk About Today
Cell-site simulators AKA Stingrays or IMSI Catchers
How they work
Previous efforts to detect them
A new method to detect them
How to fix the problem
In machine learning, we typically work with input pairs (x, y), and we try to figure out how x and y depend on each other. To do so, we gather many such pairs and hope that the dependence will reveal itself if a) we have enough data, b) our model is expressive enough to approximate this dependency, and c) we get the hyperparameters right. In the simplest case, both x and y are just scalar values (or vectors x,y,); for example, given some measurements of a plant’s shape, we might want to predict its species. The measurements here are real vectors x∈X∈, where the input space X=Rd is usually Euclidean, and the species is a label y∈Y∈ (usually an integer or a one-hot vector), but it is common for x and y to have more structure.
We'll be guiding you through a crash course on satellites - their history, where in (well, around) the world they are, and how they send and receive data. Accompanying this guide (though not strictly required for it) is a set of equipment we've used ourselves to get everything going.
If you have the means, we recommend buying the equipment yourself. If you don't, we've put together a kit that we'll send to you for a very reasonable price, though supplies are limited. The list of parts is below, or you can click here to request a kit. We also have stickers and T-shirts here. If you don't want a kit right now, you can continue on to the next section.
I started my procurement career working in the automotive, aerospace, and biotech industries. These are highly regulated industries with tight controls around spend and vendor management. **For those unsure of what exactly procurement does—the function oversees purchasing software, goods, and services on behalf of an organization.**
When I decided to make the leap into tech, I was impressed by the freedom employees undertook in how they worked. But I was equally stunned by what I saw with procurement and spend management practices (or the lack thereof). There seemed to be an allergy to any sort of process in purchasing goods and services; people wanted what they wanted, and they wanted it now. Anything short of this was “too corporate” or “slowed them down”. While I intellectually understood the justification of prioritizing growth (ie, land-grab opportunity, or open/closing funding windows) it was very clear to me that these decisions were creating cultural norms that would be hard to overcome as a company matures.
Gregory Francois And Shruti Kedia (NiemanLab): South Africa’s Oxpeckers is powering up geo-journalism for investigative environmental reporting
In some places, it was the gray dust visible in the water buckets that showed something was wrong, polluted by run-off water from the mines. In others, it was the toxic waste piles, left by mining companies to contaminate the groundwater.
The investigation by Oxpeckers Investigative Environmental Journalism found that over 100 South African mines had been polluting local water systems as a result of water permit violations and inadequate environmental testing.
Raking through data on permit violations, Oxpeckers reached out to the companies flouting regulations to hold them accountable for their lack of transparency and disregard for community health and environmental welfare.
I apologize for the long delay between issues, life got busy recently, and only now have I gotten some time to spare to write this newsletter.
As I mentioned above, I am getting more into working with hardware lately, so if you have any suggestions & resources, please send them my way.