Analyzing Real-Time Satellite Locations with OmniSci
Download OmniSci Free, a full-featured version available for use at no cost.
OmniSci is a software platform purpose-built to help businesses understand their large datasets in an interactive way. At its core, OmniSci can be thought of as 3 separate but interrelated components: OmniSci Database, OmniSci Render, and OmniSci Immerse. In this blog, we will discuss the benefits of using OmniSci to ingest real-time satellite location data and how we can utilize the loading, visualization, and querying aspects of the product to find insights.
According to Nasa.gov, the International Space Station orbits the earth every 90 minutes or 16 times per day. This rapid movement results in some pretty spectacular orbits. In this blog post, we will explore one of OmniSci’s most impressive visuals, a rendered pointmap. The rendering engine is one of the most differentiating things about OmniSci. Not only does OmniSci use GPUs to query huge datasets in milliseconds, but it also uses them to produce beautiful visuals at scale. Thinking of this, I started digging into various satellite data sites. I ended up settling on n2yo because they had a free API you could sign up for. The API is pretty straightforward and is easily query-able, so I’ll be using Python to format the API results object into OmniSci-readable formats. In this example, we will take a look at how we can use the native integration of OmniSci with JupyterLab to stream satellite positions in real-time to OmniSci.
This is the python code we will use to query the API from n2yo:
The output of this code returns:
Let’s import a JSON library to help us format this into a data frame that we can pass to OmniSci’s python adapter (pymapd) to load from the API:
Now that we have the data formatted as a JSON object, let’s convert it to a schema we can use in OmniSci.
Now let’s check data types:
It looks like the posttimestamp column is coming through as an integer python datatype, lets change that:
Now that the data is in a dataframe format, let's explore creating a table with the correct data types in OmniSci. OmniSci uses a library called ibis to allow users to interact with the backend in a pandas-esque way. To do this, let's tell ibis to ask OmniSci to give us a schema based on our data frame:
Above is the schema that will get sent to OmniSci. The OmniSci ddl is:
As an extra check, we can also compare the schemas between an existing table and the data frame like so should the api / code change in the future:
Now that the basic schema is defined, let's start streaming data into OmniSci:
To visualize this dataset, we will be using OmniSci Immerse. In this example, I am loading the data above to an existing table that has historical data for numerous satellites.
Note the pointmap in the lower left corner. This shows the historical satellite movements that are in this table. All positions are overlaid regardless of time. It makes a pretty interesting visual, for sure! These types of visuals are where the render engine really shines. If we take a look at the network panel, you can see that OmniSci isn’t moving huge amounts of data to the client to display this visual. OmniSci instead uses GPUs to render a compressed png that gets overlaid on a mapbox map:
Another thing that ships with OmniSci Immerse out of the box is cross-filtering:
In the screenshot above, you can see that the “SPACE STATION” satellite is selected in the bar chart in the lower middle portion of the dashboard. When this is selected, OmniSci automatically cross-filters the other charts on the dashboard. This is useful because it allows users to quickly toggle between historical data and live data. Below is the same example that is filtering for “current day”:
This allows users to quickly switch between huge historical views and current day snapshots.
One thing also noticeable on the pointmap is the single, large, red dot. This is actually a separate layer in the pointmap that utilizes OmniSci’s SQL backend to calculate the current position of the International Space Station in Orbit:
OmniSci, at its core, is a SQL database that harnesses the power of GPUs. One of the main benefits of using a SQL database is that SQL is a language that is very familiar to data analysts and has a very low learning curve. To calculate the location of the point above, I created a simple view that joins the table on itself to get the satellite ids and their last known position:
This view will execute independently of the base table so we can always have the current location of the ISS.
Why is the SQL query engine so powerful for users? Because it allows them to query and visualize massive datasets in milliseconds. Take the below example. In Immerse, anytime a user interacts with a chart, OmniSci fires queries to the backend which in turn, brute force scans the columns being used by the visuals in real-time. The results of the GPU database performance are astounding. The queries on this dataset all come back in milliseconds. This allows users to interact with huge data, across space and time, very quickly. From Immerse, you can toggle a setting to output the queries, and their response times (screenshot below). The interactivity of the OmniSci platform coupled with the various other technologies aforementioned are what make OmniSci capable to query, explore, and gain insights from big datasets.
OmniSci allows users to view historical and real-time information via various connectors. This flexibility allows users to visualize and query large geospatial datasets through space and time. The combination of the database engine, render engine and Immerse frontend allow users to explore and unlock insights that otherwise would have been hidden. Try OmniSci Enterprise today!