Tuesday, July 7, 2015

Rx.js and d3.js in the Red Hat Summit Middleware Keynote Demo

At this year’s DevNation/Red Hat Summit I was part of the team that created the Red Hat Summit Middleware keynote demo. I made the custom front-ends in the demo using Reactive Extensions for javascript (Rx.js) to manipulate the datastreams from the various back-end systems and to transform that data into a form that can drive the UI using d3.js.

If you missed the keynote, you can watch it on YouTube below (the keynote demo starts at 19m). In this blog post I will cover at a high level the UIs I created for the keynote demo, setting the stage for a more in-depth analysis with subsequent posts.

Rx.js in the Keynote Demo

This year’s Red Hat Summit Middleware keynote demo showcased 3 major technologies at Red Hat:

  • IoT using Red Hat JBoss A/MQ as the message broker, and Apache Spark for streaming analytics

  • Docker at Scale using OpenShift

  • Mobile with the Red Hat Mobile Application platform

To showcase the data we collected from the systems, I built 3 user interfaces using Rx.js to manipulate the data, and d3.js to visualize it. Developing these UIs provided me with an excellent opportunity to dive deep into these libraries and learn many of their ins and outs.

IoT Beacon Location

The data collected for the IoT portion of our keynote demo consisted of millions of scan events from 300 Gimbal BLE beacons handed out to DevNation attendees. The scanners were made using Raspberry PIs that transmitted the scan events to a Red Hat A-MQ message broker. These numerous events were then processed using Apache Spark, reducing the millions of beacon events into tens-of-thousands of meaningful "business events" representing attendee movements throughout the conference.

IoT Map Visualization

The Spark-processed data was retrieved from the message broker into a node.js back-end, was stored in a mongodb datastore, and then was re-broadcast to the browser clients. I used Rx.js throughout both the node.js back-end and in the browser to:

  • wrap a STOMP over Websocket protocol connection.

  • wrap the websocket broadcasts and clients

  • filter and map the data streams

  • bind to DOM events

  • etc.

Using Rx.js throughout the application meant that any idiosyncrasies of the the various protocols were encapsulated within the Observable definition, and the rest of the application adhered to a consistent API for authoring Asynchronous javascript code.

The map UI uses d3.js to bind the beacon location data to SVG circles in the browser. I then used the d3.js force layout to animate the circles and move them around the map, mimicking the movement of the real-world beacons.

Watch the video of the keynote linked above to see the UI backed with real data. I additionally embedded a code-pen that shows the beacon-location visualization backed by simulated data. (If the animation is already complete, hover over the codepen and hit the "rerun" button in the bottom right).

See the Pen Beacon Location by Brian Leathem (@bleathem) on CodePen.

IoT broker Visualization

The map UI worked really well to display the business value of the beacon data collected. We needed a second UI however to demonstrate both the sheer volume of data that our A-MQ message broker was receiving from the Raspberry Pis, and the dramatic reduction in data volume as Spark processed the data.

Again using a STOMP over websockets connection to the message broker, I used Rx.js to manipulate the data, and d3.js to visualize it. This broker UI is running in the codepen below backed by simulated data.

See the Pen Broker by Brian Leathem (@bleathem) on CodePen.

1k Containers Visualization

The power of docker, kubernetes, and OpenShift was demonstrated with a rapid scale-up of 1026 containers live on stage. To visualize this scale-up, we developed a custom "hexboard" UI, where each hexagon represents an individual kubernetes pod. The lifecycle of each individual pod was reflected as a falling hexagon indicating each stage transition, culminating with the shadowman Red Hat logo when all pods were online.

I’ve included a codepen below showing the hexboard UI, again backed by "simulated" data.

See the Pen Hexboard by Brian Leathem (@bleathem) on CodePen.

Claim you container

Audience members were then provided an opportunity to claim their own piece of "the cloud" by drawing a sketch on their mobile devices and posting those sketches to an individual kubernetes pod via the Red Hat Mobile Application Platform back-end. The audience participation was incredible, and we quickly filled the hexboard with user-generated sketches!

The source

The source code for the beacon location UI, and the 1k container UI are both available on github. The goal over the next few weeks is to clean up the keynote demo code, and provide it in a form where folks can run it themselves. Stay tuned for further developments in this regard.


Wednesday, June 24, 2015

Rx.js Session at DevNation

I presented a Session on Rx.js at DevNation this year. My goal was to impress upon the audience how Observables can be interpreted as a collection-in-time. This analogy was very well described by @jhusain his Async Javascript at Netflix talk that initially got me excited about Reactive Functional programming. My contribution to this idea is to present it in a visual way.

To visualize both a "regular" collection, as well as a collection-in-time, I used the d3.js library to visually present a collection of javascript objects that each represent a shape with a given size, color, and shape-type. In my presentation I embedded these visualizations using as Codepens, which I’ve included in this blog post below.

The slides from my presentation are available at:

A Collection

Here is the visualization of a Collection. With a "regular" collection we can get a reference to every object in the collection at any given point in time. Go ahead, "grab" a shape with your mouse!

See the Pen Collection | Iden by Brian Leathem (@bleathem) on CodePen.

An Observable

And here we have the visualization of an Observable as a Collection-in-time. An Observable is different from a "regular" collection in that we cannot grab a reference to every object at any given point in time. The objects stream past us as time progresses.

See the Pen Observable by Brian Leathem (@bleathem) on CodePen.

Reactive Extensions

In the session I then proceed to use these visualizations as basis for describing some of the tools we use for manipulating Collections/Observables:

Map

With the map function we can operate on each item in the collection. In our example mapping function we map each shape into a new shape of the same size, but of type square and color green.

.map(function(x) {
      return {
        id: x.id
      , color: 'green'
      , size: x.size
      , type: 'square'
      };
    });

Collection map

Here we have the above map function applied to a "regular" Collection:

See the Pen Operating on a Collection by Brian Leathem (@bleathem) on CodePen.

Observable map

The above map function applied to an Observable:

See the Pen Map an Observable by Brian Leathem (@bleathem) on CodePen.

MergeAll

The mergeAll function is used to "flatten" a 2-dimensional collection into a 1-dimensional collection. In this code sample we map each shape into a pair of shapes which we return as an array. The resulting "array of arrays" is then passed to the mergeAll function where it is flattened.

.map(function(x) {
        var y = _.clone(x);
        y.id = y.id + 80;
        y.color = 'green';
    
        var z = _.clone(x);
        y.size = y.size / 1.5;
        z.size = z.size / 1.5;
    
        return [y, z];
      })
      .mergeAll();

Nested Collections

This visualization shows the above mapping without the mergeAll applied. Notice how the resulting collection consists of object pairs. We do not have a flat collection. Try to grab one of the shapes with your mouse and see for yourself!

See the Pen Map a nested Collection by Brian Leathem (@bleathem) on CodePen.

Nested Collections mergeAll

With the mergeAll function applied to the Nested collection we now have a flattened collection, which we can continue to operate on with our tool set.

See the Pen MergeAll a Collection by Brian Leathem (@bleathem) on CodePen.

Observable mergeAll

The mergeAll function applied to a 2-dimensional Observable.

See the Pen MergeAll an Observable by Brian Leathem (@bleathem) on CodePen.

FlatMap

It turns out the mapmergeAll combination is a pattern we apply so often that we created the flatMap function as a shorthand. We can then rewrite the above transformation as:

.flatMap(function(x) {
        var y = _.clone(x);
        y.id = y.id + 80;
        y.color = 'green';
    
        var z = _.clone(x);
        y.size = y.size / 1.5;
        z.size = z.size / 1.5;
    
        return [y, z];
      });

Reduce

A common use case for analyzing collections is the reduce function, where one iterates over a collection and "accumulates" a value for each object in the collection. In this code sample we are accumulating the size of each shape, and using that to create a new shape of the accumulated size.

var outputData = inputData
      .reduce(function(acc, x) {
        return {
          id: x.id
        , color: 'green'
        , size: acc.size + x.size
        , type: 'square'
        };
      }, {size: 0});

Collection reduce

The above reduce function applied to a collection:

See the Pen Reduce a Collection by Brian Leathem (@bleathem) on CodePen.

Observable reduce

The reduce function applied to an Observable:

Note

You will want to click the RERUN button that appears when you mouse-over this codepen. Then wait until the input Observable terminates to see the reduce result.

See the Pen Reduce an Observable by Brian Leathem (@bleathem) on CodePen.

Zip

The last function we will look at is the zip function which is used to combine many Observables into a single observable. It accomplishes this by taking each Observable as a parameter, followed by a function that is used to "combine" the object retrieved from each Observable.

In the following code sample we combine our shapes by creating a new shape with the color of the first shape, but the size and type of the 2nd shape.

var outputData = Rx.Observable.zip(
      input1Data,
      input2Data,
      function(x1, x2) {
        return {
          id: x1.id
        , color: x1.color
        , size: x2.size
        , type: x2.type
        };
      });

Observable zip

See the Pen Zip an Observable by Brian Leathem (@bleathem) on CodePen.

The rest of the talk

In the remaining slides I discuss creating and subscribing to Observables, and went through a number of use cases and examples. I ended with a preview and brief code walk-through of the Red Hat Summit Middleware keynote demo, that I wrote using Rx.js. But that is a topic for another post.

The slides are available at: