UFO sightings datasets, the truth is out there... Part 3

In previous posts we manipulated UFO datasets, now we show you how to display your analysis.

UFO datasets: the truth is out there, part 3

After having manipulated UFO data in Part 1 and Part 2, it is time to build a dataviz dashboard like this one:

All the following WarpScript are explained in the previous posts about UFO datasets.

In order to build this dataviz, we will use:

NodeJS
Yarn
gridstack
HTML/CSS (just a little JavaScript, I swear)
WarpView

At least, we assume that NodeJS and Yarn are installed. Of course, you can just use NPM. This tutorial is made for Linux/macOS systems. Windows users can adapt easily.

Step 1: initiate the project

In a chosen directory, type:

mkdir ufo-dashboard
cd ufo-dashboard
yarn init // answer yes, yes and yes and again yes until the end

Now add dependencies:

yarn add -D http-server
yarn add @senx/warpview gridstack

Edit package.json with your favorite IDE (i.e. VSCode) and add the script block:

{
  "name": "ufo-dashboard",
  "version": "1.0.0",
  "main": "index.js",
  "license": "MIT",
  "devDependencies": {
    "http-server": "^13.0.2"
  },
  "dependencies": {
    "@senx/warpview": "^2.0.79",
    "gridstack": "^4.2.7"
  },
  "scripts": {
    "serve": "http-server . -p 3000"
  }
}

Now create new files: index.html and style.css

Step 2: bootstrap the web page

Edit index.html:

<!DOCTYPE html>
<html>

<head>
  <title>My UFO dashboard</title>
  <link rel="stylesheet" href="./node_modules/gridstack/dist/gridstack.min.css" />
  <link rel="stylesheet" href="./style.css">
</head>

<body>
  <div class="container">
    <div class="grid-stack">
      <!-- Dashboard tiles will be here -->
    </div>
    <script src="./node_modules/@senx/warpview/elements/warpview-elements.js"></script>
    <script src="./node_modules/gridstack/dist/gridstack.js"></script>
    <script type="text/javascript">
       var grid = GridStack.init({ cellHeight: "220px", staticGrid: true, column: 12, float: false, verticalMargin: 10 });
    </script>
</body>

</html>

In this file, we import the Gridstack javascript framework (not mandatory, of course) and WarpView.

Now we can run a small HTTP server to see the result:

yarn serve

Now open a browser (Firefox 64+ or Chrome) and go to http://127.0.0.1:3000

Step 3: add dataviz tiles

Now we will add tiles to our dashboard. We use WarpView for that, encapsulated into a Gridstack bloc and a div for look and feel purpose:

<div class="grid-stack-item" data-gs-width="12" data-gs-height="2">
  <div class="widjet">
    <warp-view-tile options="{'showStatus' : false}" url="https://sandbox.senx.io/api/v0/exec">
      // WarpScript Here
    </warp-view-tile>
  </div>
</div>

Gridstack parameters:

data-gs-width="12": A width of 12 units
- We set 12 as the maximum in the Gridstack init param column: 12
data-gs-height="2": 2 units height
- We set 1 unit height = 220px in the Gridstack init param cellHeight: "220px"

WarpView parameters:

url: Exec endpoint of your Warp 10 instance
chart-title: a title for the tile
type: kind of dataviz (line, plot, annotation, map, …)
options: a map of options like the color scheme for instance, could be passed by a WarpScript

The map tile

The map tile is quite complex, we will pass necessary options through the WarpScript:

'<Your Read token>' 'token' STORE
$token AUTHENTICATE
10000000 MAXOPS // yes there will be a lot of computation

// Get all the data
[
  $token
  '~(military.bases|sighting.ufo)' { 'country' '~(us|United States)' }
  [ 1900 01 01 ] TSELEMENTS-> ISO8601 
  [ 2021 01 01 ] TSELEMENTS-> ISO8601
] FETCH 'gts' STORE 

// extract sightings
[ $gts [] 'sighting.ufo' filter.byclass ] FILTER MERGE DEDUP 'sightings' STORE

// extract bases
[ $gts [] 'military.bases' filter.byclass ] FILTER MERGE DEDUP 'bases' STORE

// map sightings
[
  $sightings
  <%
    'point' STORE
    
    // get datapoint's location
    $point [ 4 5 ] SUBLIST FLATTEN 'loc' STORE

    // compute HHCode
    $loc 0 GET $loc 1 GET ->HHCODE 'hhcode' STORE

    // new datapoint return with the HHCode as value
    [ $point 0 GET $loc 0 GET $loc 1 GET NaN $hhcode ]
  %> MACROMAPPER
  0 0 0
] MAP
0 GET 'hhcodeGTS' STORE

// Sum same HHCodes
$hhcodeGTS VALUEHISTOGRAM 'repartition' STORE

// create a new GTS
NEWGTS 'count.by.location' RENAME 'locGTS' STORE

// For each HHCode
$repartition 
<% 
  'value' STORE 'key' STORE
  // convert HHCode to lat/long
  $key HHCODE-> [ 'lat' 'long' ] STORE 

  // Add the sighting count per location as value
  $locGTS NOW $lat $long NaN $value ADDVALUE DROP
%> FOREACH

// Prepare data for display
[ $locGTS mapper.tostring 0 0 0 ] MAP 'ufo' STORE

[
  {
    'key' 'sightings'
    'render' 'weightedDots'
    'color' '#31C0F6cc'
    'borderColor' '#31C0F6'
    'maxValue' $repartition VALUELIST MAX
    'minValue' 0 
  }
  {
    'key' 'bases'
    'render' 'dot'
    'color' '#f44336'
  }
] 'params' STORE

{
  'data' [ $ufo $bases ]
  'params' $params
  'globalParams' {
    // Map specific params like the initial position and zoom
    'map' {
      'mapType' 'CARTODB_DARK'
      'startLat' 39.8364989
      'startLong' -98.3276331
      'startZoom' 5
    }
  }
}

The final HTML bloc to insert into <div class="grid-stack"> looks like:

<div class="grid-stack-item" data-gs-width="12" data-gs-height="2">
<div class="widjet">
<warp-view-tile type="map"
                          url="https://sandbox.senx.io/api/v0/exec"
                          options="{"showStatus" : false}">
'<Your Read token>' 'token' STORE
$token AUTHENTICATE
10000000 MAXOPS // yes there will be a lot of computation

// Get all the data
[
  $token
  '~(military.bases|sighting.ufo)' { 'country' '~(us|United States)' }
  [ 1900 01 01 ] TSELEMENTS-> ISO8601 
  [ 2021 01 01 ] TSELEMENTS-> ISO8601
] FETCH 'gts' STORE 

// extract sightings
[ $gts [] 'sighting.ufo' filter.byclass ] FILTER MERGE DEDUP 'sightings' STORE

// extract bases
[ $gts [] 'military.bases' filter.byclass ] FILTER MERGE DEDUP 'bases' STORE

// map sightings
[
  $sightings
  <%
    'point' STORE
    
    // get datapoint's location
    $point [ 4 5 ] SUBLIST FLATTEN 'loc' STORE

    // compute HHCode
    $loc 0 GET $loc 1 GET ->HHCODE 'hhcode' STORE

    // new datapoint return with the HHCode as value
    [ $point 0 GET $loc 0 GET $loc 1 GET NaN $hhcode ]
  %> MACROMAPPER
  0 0 0
] MAP
0 GET 'hhcodeGTS' STORE

// Sum same HHCodes
$hhcodeGTS VALUEHISTOGRAM 'repartition' STORE

// create a new GTS
NEWGTS 'count.by.location' RENAME 'locGTS' STORE

// For each HHCode
$repartition 
<% 
  'value' STORE 'key' STORE
  // convert HHCode to lat/long
  $key HHCODE-> [ 'lat' 'long' ] STORE 

  // Add the sighting count per location as value
  $locGTS NOW $lat $long NaN $value ADDVALUE DROP
%> FOREACH

// Prepare data for display
[ $locGTS mapper.tostring 0 0 0 ] MAP 'ufo' STORE

[
  {
    'key' 'sightings'
    'render' 'weightedDots'
    'color' '#31C0F6cc'
    'borderColor' '#31C0F6'
    'maxValue' $repartition VALUELIST MAX
    'minValue' 0 
  }
  {
    'key' 'bases'
    'render' 'dot'
    'color' '#f44336'
  }
] 'params' STORE

{
  'data' [ $ufo $bases ]
  'params' $params
  'globalParams' {
    // Map specific params like the initial position and zoom
    'map' {
      'mapType' 'CARTODB_DARK'
      'startLat' 39.8364989
      'startLong' -98.3276331
      'startZoom' 5
    }
  }
}
</warp-view-tile>
</div>
</div>

Here we set mapType with CARTODB_DARK. There are other map layers: DEFAULT (OpenstreetMap), HOT, TOP, TOPO2, SURFER, HYDRA, HYDRA2, TONER, TONER_LITE, TERRAIN, ESRI, SATELLITE, OCEANS, GRAY, GRAYSCALE, WATERCOLOR, CARTODB and CARTODB_DARK.

Reload your browser to see the result.

Sightings through time

Now, and for the next tiles, we will use a custom color scheme: NINETEEN_EIGHTY_FOUR

Available color schemes are WARP10 (default), COHESIVE, COHESIVE_2, BELIZE, VIRIDIS, MAGMA, INFERNO, PLASMA, YL_OR_RD, YL_GN_BU, BU_GN, NINETEEN_EIGHTY_FOUR, ATLANTIS, DO_ANDROIDS_DREAM, DELOREAN, CTHULHU, ECTOPLASM, and T_MAX_400_FILM.

This is a chart of "area" type (it could be also "line", "spline", "step", "step-after", "step-before" and "scatter").

<div class="grid-stack-item" data-gs-width="6" data-gs-height="1">
  <div class="widjet">
    <warp-view-tile chart-title="Sightings through time" type="area" url="https://sandbox.senx.io/api/v0/exec"
      options="{ 'showStatus' : false, 'scheme' : 'NINETEEN_EIGHTY_FOUR' }">
'<Your Read token>' 'token' STORE
// get data
[
  $token
  'sighting.ufo' { 'country' 'us' }
  [ 1950 01 01 ] TSELEMENTS-> ISO8601
  [ 2021 01 01 ] TSELEMENTS-> ISO8601
] FETCH 'gts' STORE
// bucketize and fill gaps
[ $gts bucketizer.count ] @senx/cal/BUCKETIZE.bymonth
UNBUCKETIZE.CALENDAR
[ NaN NaN NaN 0 ] FILLVALUE 'bucketized' STORE
// reduce
[ $bucketized [] reducer.sum ]
REDUCE
    </warp-view-tile>
  </div>
</div>

Sightings vs movies releases

<div class="grid-stack-item" data-gs-width="6" data-gs-height="1">  <div class="widjet">    <warp-view-tile chart-title="Sightings vs movies releases" type="area" url="https://sandbox.senx.io/api/v0/exec" options="{ "showStatus" : false , "scheme" : "NINETEEN_EIGHTY_FOUR" }">"<Your Read token>" "token" STORE// get data[  $token "~(ufo.movies|sighting.ufo)" {}  [ 1900 01 01 ] TSELEMENTS-> ISO8601  [ 2021 01 01 ] TSELEMENTS-> ISO8601] FETCH SORT "gts" STORE// bucketize per year count[ $gts bucketizer.count [ 2021 01 01 ] TSELEMENTS-> 365 d 0 ] BUCKETIZE// fill gaps[ NaN NaN NaN 0 ] FILLVALUE "bucketized" STORE// keep movies[ $bucketized [] "ufo.movies" filter.byclass ] FILTER "movies" STORE// keep sightings[ $bucketized [] "sighting.ufo" filter.byclass ] FILTER "sightings" STORE// reduce sightings[ $sightings [] reducer.sum ] REDUCE "sightings" STORE// and display$movies NORMALIZE$sightings NORMALIZE    </warp-view-tile>  </div></div>

Military base distance

This is a chart of "pie" type (it could be also "donut").

<div class="grid-stack-item" data-gs-width="3" data-gs-height="1">    <div class="widjet">        <warp-view-tile chart-title="Military base distance" type="pie" url="https://sandbox.senx.io/api/v0/exec" options="{ "showStatus" : false, "scheme" : "NINETEEN_EIGHTY_FOUR"  }">"<Your Read token>" "token" STORE[  $token "~(military.bases|sighting.ufo)"   { "country" "~(us|United States)" }  [ 1900 01 01 ] TSELEMENTS-> ISO8601  [ 2021 01 01 ] TSELEMENTS-> ISO8601] FETCH "data" STORE[ $data [] "sighting.ufo" filter.byclass ] FILTER MERGE DEDUP "sightings" STORE[ $data [] "military.bases" filter.byclass ] FILTER MERGE DEDUP "bases" STORE[] "listOfShapes" STORE[ $bases <%   // get datapoint"s location  FLATTEN [ 4 5 ] SUBLIST "loc" STORE  // get a 50 km radius circle  $loc 0 GET $loc 1 GET 50000 @senx/geo/circle 0.05 false GEO.WKT "geoShape" STORE  // append the Shape  $listOfShapes $geoShape +! DROP // discard the result  // dummy return  [ 0 NaN NaN NaN NULL ]%> MACROMAPPER 0 0 0 ] MAP DROP // discard the result// merge shapes and optimize with a 2.5 km precision$listOfShapes GEO.UNION 14 GEO.OPTIMIZE "geoArea" STORE// filter sightings to keep only those who are inside[ $sightings $geoArea mapper.geo.within 0 0 0 ] MAP 0 GET// countVALUES SIZE "near" STORE// display{  "data" [     [ "< 50 km" $near ]    [ "> 50 km" $sightings VALUES SIZE $near - ]   ]}    </warp-view-tile>  </div></div>

First state by sightings per inhabitant

This is a chart of "display" type.

<div class="grid-stack-item" data-gs-width="3" data-gs-height="1">  <div class="widjet">    <warp-view-tile chart-title="First state by sightings per inhabitant" type="display" url="https://sandbox.senx.io/api/v0/exec" options="{ "showStatus" : false, "timeMode" : "custom" }">"<Your Read token>" "token" STORE{  "al" 4872725 "ak" 746079 "az" 7044577 "ar" 2998643 "ca" 39506094  "co" 5632271 "ct" 3568174 "de" 960054 "dc" 691963 "fl" 20979964  "ga" 10421344 "hi" 1431957 "id" 1713452 "il" 12764031 "in" 6653338  "ia" 3147389 "ks" 2907857 "ky" 4449337 "la" 4694372 "me" 1333505  "md" 6037911 "ma" 6839318 "mi" 9938885 "mn" 5557469 "ms" 2988062  "mo" 6109796 "mt" 1052967 "ne" 1920467 "nv" 2996358 "nh" 1339479  "nj" 8953517 "nm" 2081702 "ny" 19743395 "nc" 10258390 "nd" 759069  "oh" 11623656 "ok" 3939708 "or" 4162296 "pa" 12776550 "pr" 3661538  "ri" 1057245 "sc" 5027404 "sd" 872989 "tn" 6707332 "tx" 28295553  "ut" 3111802 "vt" 623100 "va" 8456029 "wa" 7415710 "wv" 1821151  "wi" 5789525 "wy" 584447} "statePop" STORE// first step[  $token "sighting.ufo" { "country" "us" }  [ 1900 01 01 ] TSELEMENTS-> ISO8601  [ 2021 01 01 ] TSELEMENTS-> ISO8601] FETCH "gts" STORE// second step[ $gts bucketizer.count NOW 0 1 ] BUCKETIZE "bucketized" STORE// Fill gaps$bucketized [ NaN NaN NaN 0 ] FILLVALUE "filled" STORE[ $filled [ "state" ] reducer.sum ] REDUCE "reduced" STORE// Map values[ $reduced <% // ok a bit tricky here, refer to the doc  "data" STORE  $data 2 GET 0 GET "state" GET "state" STORE // get the state  $statePop $state GET TODOUBLE "pop" STORE //get the population  // compute the proportion of population  $data 7 GET 0 GET TODOUBLE "value" STORE  $value $pop / 100.0 * "newValue" STORE  [ $data 0 GET NaN NaN NaN $newValue ]%> MACROMAPPER 0 0 0 ] MAP// prettify the result// sort by values<% VALUES 0 GET %> SORTBY// descending orderREVERSE0 GET LABELS "state" GET    </warp-view-tile>  </div></div>

Top 5 sightings per inhabitant

This is a chart of "datagrid" type.

This is the same WarpScript as above, except the end:

<div class="grid-stack-item" data-gs-width="3" data-gs-height="1">  <div class="widjet">    <warp-view-tile chart-title="Top 5 sightings per inhabitant" type="datagrid" url="https://sandbox.senx.io/api/v0/exec"options="{ "showStatus" : false }">            "<Your Read token>" "token" STORE[ ... same WarpScript ]// prettify the result// sort by values<% VALUES 0 GET %> SORTBY// The previous WarpScript changes here// descending order REVERSE 0 5 SUBLIST <%  "gts" STORE   $gts LABELS "state" GET "state" STORE  [     $state  // the state name    $gts VALUES 0 GET 10000 * ROUND 100.0 / TOSTRING "%25" + // the percentage    $statePop $state GET // the total population  ] %> F LMAP "data" STORE{ "data" [  {    "title" ""    "columns" [ "State" "Sightings" "total population" ]    "rows" $data  } ]}        </warp-view-tile>    </div></div>

Sightings per month

This is a chart of "bar" type (this chart could be also horizontal and stacked, please refer to the doc).

This script changes against the first post, here we convert the month"s number to a string.

<div class="grid-stack-item" data-gs-width="3" data-gs-height="1">  <div class="widjet">    <warp-view-tile chart-title="Sightings per month" type="bar" url="https://sandbox.senx.io/api/v0/exec" options="{ "showStatus" : false, "scheme" : "NINETEEN_EIGHTY_FOUR"  }">"<Your read token>" "token" STORE// get data[   $token "sighting.ufo" { "country" "us" }  [ 1900 01 01 ] TSELEMENTS-> ISO8601  [ 2021 01 01 ] TSELEMENTS-> ISO8601] FETCH "gts" STORE// bucketize and fill gaps[ $gts bucketizer.count ] @senx/cal/BUCKETIZE.bymonth UNBUCKETIZE.CALENDAR [ NaN NaN NaN 0 ] FILLVALUE "bucketized" STORE// reduce[ $bucketized [] reducer.sum ] REDUCE 0 GET  // split by year"Europe/Paris" @senx/cal/byyear // or whatever Timezone// align all on 01/01/1970<% "g" STORE $g 0 $g FIRSTTICK - TIMESHIFT %> F LMAP// bucketize[ SWAP bucketizer.sum ] @senx/cal/BUCKETIZE.bymonth // reduce[ SWAP [] reducer.sum ] REDUCE "gts" STORE[ "Jan" "Feb" "Mar" "Apr" "May" "Jun" "Jul" "Aug" "Sep" "Oct" "Nov" "Dec" ] $gts VALUES FLATTEN2 ->LIST ZIP "data" STORE // the magic part{  "title" ""  "columns"  [ "" ]   "rows"  $data } "values" STORE{ "data" $values  }    </warp-view-tile>  </div></div>

Step 4: Look and feel

Now it is time to skin our web page.

This CSS is based upon this CodePen: https://codepen.io/varqes/details/YLzOqQ and some pictures from https://robertsspaceindustries.com/

Edit style.css:

@import url("https://fonts.googleapis.com/css?family=Jura:300,400,500,600,700&subset=cyrillic,cyrillic-ext,latin-ext");* {    box-sizing: border-box;}:root {    --body-color       : #89b1c4;    --primary-color    : #00ebff;    --secondary-color  : #ffae00;    --white-color      : #fff;    --price-color      : #0f0;    --light-color      : rgba(22, 104, 159, 0.2);    --blue-color       : rgba(10, 24, 34, 0.8);    --green-color      : #677a0a;    --line-color       : #0a5688;    --dark-black-color : #182027;    --light-black-color: #1a252d;    --shadow-color     : #0074c2;    --transparent-color: transparent;}html {    font-size               : 65%;    -webkit-text-size-adjust: 100%;    -ms-text-size-adjust    : 100%;    outline                 : 0;}body {    font-family: "Jura", sans-serif;    font-size  : 12px;    line-height: 1.52;    background : black;    color      : var(--body-color);    background : url(https://robertsspaceindustries.com/rsi/static/images/gridbg_glow.png), url(https://robertsspaceindustries.com/rsi/static/images/common/bg-stars-2560.jpg) repeat;    padding    : 1rem;    height     : calc(100vh - 2rem);}.container {    max-width: 100%;    margin   : 0 auto;}h1,h2,h3,h4,h5 {    text-transform: uppercase;    font-weight   : 700;    color         : var(--primary-color);    text-shadow   : 0 0 50px var(--shadow-color);}.widjet {    position  : relative;    background: var(--light-color);    border    : .1rem solid var(--light-color);    display   : block;    height    : 100%;    min-height: 220px;    background: url(https://robertsspaceindustries.com/rsi/static/images/common/hexagons.png);}.widjet:after {    content      : "";    position     : absolute;    bottom       : -.2rem;    left         : 50%;    transform    : translateX(-50%);    width        : 5rem;    border-bottom: .3rem solid var(--primary-color);    box-shadow   : 0 0 15px 3px rgba(0, 112, 202, 0.6), 0 -15px 25px 0 rgba(11, 183, 226, 0.65);}.widjet h1 {    background: url(https://robertsspaceindustries.com/rsi/static/images/expandbtn-bg.png) repeat-x;    font-size : 1rem !important;    padding   : 1rem 2rem;    margin    : 0 -2rem 1rem;}

And now the specific WarpView part:

warp-view-tile {    color: transparent;}.grid-stack-item {    color                             : #c0c0c0;    padding-left                      : 5px !important;    padding-right                     : 5px !important;    display                           : block;    position                          : relative;    /* WarpView specific part */    --warp-view-chart-label-color     : #fff;    --warp-view-chart-grid-color      : #a0a0a0;    --gts-stack-font-color            : #c0c0c0;    --warp-view-chart-tile-transform  : hue-rotate(180deg) invert(100%);    --warp-view-chart-legend-bg       : #000000;    --warp-view-chart-legend-color    : #c0c0c0;    --gts-labelvalue-font-color       : #a0a0a0;    --gts-separator-font-color        : #c0c0c0;    --gts-labelname-font-color        : rgb(105, 223, 184);    --gts-classname-font-color        : #004eff;    --warp-view-font-color            : #c0c0c0;    --warp-view-chart-label-color     : #c0c0c0;    --warp-view-chart-grid-color      : #c0c0c0;    --warp-view-datagrid-even-bg-color: rgba(22, 104, 159, 0.2);    --warp-view-datagrid-even-color   : var(--primary-color);    --warp-view-datagrid-odd-bg-color : transparent;    --warp-view-datagrid-odd-color    : #c0c0c0;    --warp-view-spinner-color         : rgba(22, 104, 159, 0.5);}

As you can see, there is a lot of CSS vars to customize WarpView.

Final thought

With WarpView, it is quite easy to build a custom dashboard by using directly WarpScript into a Web page. If you want to use a framework such as React, VueJS, or Angular, you can pass the WarpScript directly as an attribute:

<warp-view-tile url="..." warpscript="NEWGTS ..." ></warp-view-tile>

Of course, inserting tokens directly in your Web page is bad practice. You have to hide the token:

By calling macros
By protecting secrets
Or something else (you can contact us, we have plenty of ideas about that).

Let us know what kind of dataviz you made with WarpView.

Live long and prosper.

Build a Complete Application with Warp 10, from TCP Stream to Dashboard
Warp 10 provides a lot of functionalities, making possible the creation of whole applications. Learn how to get, store and display live ship positions...

Truly Dynamic Dashboards as Code
With its unique dashboards as code approach, Warp 10 Discovery is the tool of choice to create highly dynamic dashboards from your time series data

Video Streaming with Warp 10
Warp 10 a very versatile platform to handle any sequence of values. In this experiment we show it is even possible to store and stream videos in real-...

Xavier Marin

Senior Software Engineer

UFO sightings datasets, the truth is out there… Part 3

Tutorials