After finishing my review of “Learning D3.js Mapping”, I wanted to take things up a notch so I combined one of the more advanced examples from the book with my Pan and Zoom demo code. Check out the results below:
See the Pen d3 Mapping by Bill White (@billdwhite) on CodePen.
I was recently asked to review a copy of PacktPub’s “Learning D3.js Mapping” by Thomas Newton and Oscar Villarreal. I’ve done a good deal of d3 development, but the mapping portion of the library has not received much of my attention until now. A fellow d3 developer and leader of our local d3 Meetup group, Andrew Thornton, has done considerable work with d3-driven mapping applications and it is something I’ve always wanted to delve into but never really had the chance. This book served as that motivation and while it is relatively short, it provides as a great introduction to generating maps in d3 and demonstrates just how powerful the library truly is for creating geographic visualizations.
The book starts by laying out the basic setup for running the provided code samples. They suggest running with the code using a local nodejs http-server, but you can just as easily run the examples directly from the downloaded code samples folder, or using an online IDE such as CodePen or JSBin, although you will need to have access to the hosted sample JSON data files used in the examples. A basic foundation of SVG fundamentals is also provided, which can be useful for those just getting started with SVG-based data visualizations. While the more seasoned d3 developers will be inclined to skip this section, the authors focus a good deal on SVG elements such as paths, which foreshadows their importance in creating maps later in the book. The third chapter is a d3 primer, providing some history behind the library itself. The enter/update/exit lifecycle is described although novice readers will want to read further about this before attempting the more advanced d3 code samples.
The meat of the book is found starting with chapter 4 where you create your first map. The reader is introduced to projections which handle the conversion of coordinates from one space to another. The authors do a good job of covering supporting topics such as scaling and translations which may be new to some readers. Later on in the chapter, an approach to creating choropleths (a fancy way of saying “color-filled maps”) is discussed which introduces the beginners to the use of d3 scales for mapping colors to values, handling click events and adding transitions to spruce up the user interaction. Again, nothing new for most d3 experts, but the authors are clearly trying not to leave anyone out with respect to programming backgrounds. Below is an example of one of the code samples discussed in Chapter 4:
See the Pen d3 Mapping demo 1 by Bill White (@billdwhite) on CodePen.
Chapter 5 is where the book really hits its stride. Starting with the addition of tooltips and then diving into the use of pan and zoom (one my personal favorites), this chapter takes a basic d3-generated map and turns it into a functional data visualization. The reader learns about orthographic projections (representing three-dimensions in a two-dimensional format) such as this:
See the Pen d3 Mapping demo 2 by Bill White (@billdwhite) on CodePen.
Of course, maps are cool to display using d3, but a big piece of the puzzle is obtaining the mapping data to drive the visualization. Chapter 6 discusses the difference between shapefiles, GeoJSON and TopoJSON file types and even shows you where to get free shapefiles, albeit they are the largest and most inefficient of the three types. I found this chapter to be very useful since understanding the mapping source data is key to creating solid mapping visualizations. Up to this point, I had never really understood the difference between these types, nor was I aware that Mike Bostock, the creator of d3, was also behind the TopoJSON project. (It is amazing how much code developers rely upon every day without knowing the names of the key contributors or understanding the motivation for it’s initial creation. This is yet further proof that open source software makes the world go ’round).
The final chapter review unit testing techniques, which seems a bit out of place, but is helpful and informative nonetheless. Unit testing is clearly an important aspect of writing good code. However I cannot help but think that, if what we want is more scalable, maintainable and testable code, then getting away from a weakly-typed, inconsistently-implemented, functional language such as Javascript would be a good first step. But since Javascript is really the only viable option we have these days, we have to make the best of it and the techniques the authors discuss here are certainly worth considering.
Overall, I enjoyed the book and would certainly recommend it to anyone looking to create maps using d3. The d3 library itself is quite challenging in its own right, but this book gives you enough background that you can get some geographic visualizations up and running in very little time.
Along with d3, the three.js library is one of the most powerful javascript libraries available for creating slick visualizations. I wondered if it would be possible to create data visualizations in threejs as easily as I had done using d3. One thing that d3 does very well is to take your data and apply a layout algorithm to it for use in drawing visualizations such as treemaps and piecharts. While I’m sure there are other ways to generate this layout data, the d3 approach is the one that I’m most comfortable with and I wondered if it would be possible to use this layout data to render 3D versions of those same visualizations.
This first demo shows a WebGL (or Canvas if you’ve got a crappy browser like IE) rendered treemap that was created using the d3 treemap layout: (I had to use JSBin instead of CodePen for the demos because the embedded CodePen snippets kept locking a few seconds after loading; no idea why)
See the Pen d3 3D Treemap by Bill White (@billdwhite) on CodePen.
The results work pretty well. The Tweening is handled using TweenJS which is what the threejs folks use. I wanted to do the updates using d3 transitions but using the Tween code from the examples I had seen worked straight away so maybe I can swithc over to d3 transitions in the future. Also, when you set the zscale to 0, you see a bit of color bleeding and I have no idea why. (Let me know if you have a solution.)
First, I must give credit to Steve Hall who has also investigated this concept. Last year I explored ways to create 3D visualizations with threejs using layout data from d3, but I hit a wall when I was unable to get the coordinates to translate between the two worlds correctly. Steve also worked on this idea and wrote some fantastic blog entries about it. His work focused more on created mini-visualizations, attaching them to DOM nodes and then having threejs render those nodes in 3D. After reading his work, I contacted him to ask if he knew how to correctly translate the coordinates and to my relief, he gladly provided the solution. So in the interests of full disclosure, my work here was greatly facilitated by his assistance. Thanks Steve!
The solution I landed on is relatively straightforward: Take a dataset, run it through d3 to apply layout positioning information to each data item and then render those items in threejs rather than the normal DOM-oriented CSS styling that we use with d3. In a typical d3 visualization, the data is bound to the DOM nodes and you use the enter/update/exit lifecycle to add, modify and remove those nodes. With threejs, we are no longer adding stuff to the DOM tree (unless we use the CSS3DRenderer, but more on that in a second) because the 3D world is rendered using WebGL (or Canvas when WebGL is unsupported).
So I am taking the dataset that has been processed by d3 and then adding each item to the threejs scene. I add, update and remove items from the scene in reaction to the add/update/remove calls from the d3 general update lifecycle. Interestingly, I’m actually adding DOM nodes to the tree using d3 but they are empty and unstyled so they end up only serving as placeholders for the current set of rendered data in the threejs scene. If the data changes, the d3-generated empty nodes are added/updated/removed and then, for each change, a corresponding change is made to the objects in the threejs scene.
Steve’s demos are implemented using the CSS3DRenderer which actually displays the 3D objects using only CSS3 transforms. Because of this, you are actually still in the d3 data-bound-to-the-DOM-node situation. It is really the best of both d3 and threejs because you can take full advantage of d3 selections and still have threejs display the 3D results using those same nodes. However, if you want your 3D visualization to look, well, three-dimensional, you have to bridge the gap between the d3 generated DOM nodes and the corresponding objects in the threejs scene yourself.
This second demo uses the CSS3DRenderer and uses d3 to create the DOM elements which are then manipulated by threejs. You can see that while it is nice, it is not quite as fancy as the first demo:
See the Pen CSS3DRenderer Treemap by Bill White (@billdwhite) on CodePen.
Question: Is it possible to make a single page web application that uses CSS to layout the page and still have your d3 charts know when to update their size/appearance?
One thing that always catches my attention with d3 examples is the lack of code that makes the chart fill the viewport when it is resized. Granted, the focus of these examples is usually on the design of the visualization itself, but in the real world these graphics will usually run inside a larger application and it should fill the space available. And by resizing, I do not mean scaling the chart to fit. To me, making the content bigger or smaller based on the viewport size is no more useful than if your word processor made the font bigger when you viewed it in full-screen mode. When the visualization resizes, it should simply make better use of the available space. The trick is knowing when to resize the chart in response to the layout change. So how do you know when to resize the d3 chart? Well, that depends on what is controlling the layout of the DOM elements. Is it a JavaScript widget library? Maybe an event from a container created by jQuery? Or are you using CSS3 to layout the page?
Approach 1: JavaScript widget library
If the visualization is running inside a web application created with something like Sencha ExtJS or Dojo, you can utilize the event mechanisms that these libraries provide in their widget infrastructures. For example, in Dojo you can layout the interface using a BorderContainer which holds multiple ContentPanes. You can then listen for the ContentPane’s resize event to know when to update the d3 chart it contains. In the example below, you see two resizable pie charts in a Dojo split container. Then you move the splitter, each pie chart hears the resize event fired by its parent ContentPane and it updates its own size accordingly:
See the Pen d3 CSS Layout – Dojo PieChart by Bill White (@billdwhite) on CodePen.
The downside is that widget libraries create a fairly heavyweight DOM structure in order to provide all the fancy features and handle backward compatibility with older browsers. There is also a good amount of JavaScript execution happening here, particularly when you start nesting BorderContainers within BorderContainers within BorderContainers, etc. Each widget executes a series of lifecycle methods that compute measurements, calculate space requirements and then dispatch events to notify the other containers that they might need to resize in response. It would be great if we could get these powerful layouts without the extra overhead and then have the d3 charts know when the layout has made size changes.
Approach 2: Listen to window.resize or use jQuery to dispatch events
If you have CSS handling the layout of your DOM elements, you could update your d3 charts by listening for resize events from the browser window. You can find this approach suggested in a few posts as a standard way to make the chart “responsive”. This is not a bad solution if it suits your need, but I personally find it has some undesirable side effects. Ideally, your d3 chart would only resize when the parent DOM element does. If you are using dynamic components (i.e. accordion containers, etc) that resize without changing the size of the browser itself, the window’s resize event may not fire unless someone has gone through the trouble of triggering that event manually. Another problem with only listening for the browser resize event is that if you have several visualizations in your application, they could all be resizing in response to the window event whether it is truly necessary or not.
Take the following example. Using jQuery and a reusable d3 pie chart, we have a visualization set inside a resizable container. The pie chart listens to the window.resize event to know when it should update itself. Try resizing each container and you will notice that neither pie chart adjusts its size. Only when the window is resized do the charts perform an update: (click on the “Edit on CodePen” link in the right corner of the demo to launch it separately to see the pie charts resize when the browser window is adjusted. The embedded version in this post cannot see these events)
See the Pen xhEKk by Bill White (@billdwhite) on CodePen.
My point is that your d3 chart will not resize if you only listen to window events and the DOM resizes for some other reason. In this second example, I instead listen for the resize event on the container created by jQuery.
See the Pen xhEKk by Bill White (@billdwhite) on CodePen.
This is better but we still are not using a CSS-only approach to handle the layout of the DOM. The goal here is to use CSS for the layout and still have the d3 chart know when a resize is required.
Approach 3: CSS Layouts
Rather than have JavaScript executing in response to an event provided by a widget library (Dojo/Sencha, etc), why not use CSS to handle the layouts? In particular, we can use the flexbox layout to position everything for us. The catch is that you still have know that a resize action has occurred. Again, you could resort to listening to the window’s resize events, but we’ve already mentioned the problems with that approach. Is there a way to know when CSS has made changes to the size of a container? Maybe so…..
I came across a project by Marc Schmidt that might offer a solution. Using the ResizeSensor I was able to setup a relatively lightweight listener that fires when the containing DIV is resized. This in turn notifies the d3 chart that it needs to update. The following demo shows 2 reusable treemaps displayed within a CSS flowbox container. For the sake of the demo I added a jQuery resizable container to allow you to see what happens when the flexbox layout adjusts the size of the containers holding the treemaps: (doesn’t work cleanly in IE yet)
See the Pen vLgpf by Bill White (@billdwhite) on CodePen.
The ResizeSensor does add a DOM element to the container along with a bit of JavaScript to track size changes. But this overhead is small compared with a traditional JavaScript widget library and the script watches for any size changes to that particular element rather than the entire browser window. Additionally, this approach is flexible enough that it works with responsive layout managers such as Twitter Bootstrap. The following example shows a treemap that is contained inside a Bootstrap 12 column layout:
See the Pen vLgpf by Bill White (@billdwhite) on CodePen.
Using a ResizeSensor, we respond to the parent DOM element being changed by the Bootstrap layout. jQuery’s resizable container is only used to let you trigger that Bootstrap layout update for the sake of the demo.
So what are the flaws with this approach? The primary concern is that if you rely on CSS to layout your application, you may run into issues if you have to support older browsers. The ResizeSensor seems to work great in many older browsers, but the immensely useful flexbox layout is not supported by older versions of IE so you have to work around that. Still, if you can make CSS handle your application’s layout, you can get your d3 charts to be responsive without the overhead of a JavaScript widget library or a bunch of listeners tied to the browser window’s resize event.
If anyone can add to this discussion or find oversights in my approaches, I would love to get your feedback. 🙂
The visualizations that I work on have large datasets that need to be rendered quickly with as little DOM overhead as possible. One of the cool things about d3 is that is renders the data present at any given moment along with changes to that data that are coming or going. However, if you have a large dataset, d3 can kill the browser under the weight of all the DOM nodes that it is attempting to create. Here is a demo of what I came up with to solve this problem:
Virtual Scrolling Demo
You can explore the GIST code here.
Virtual rendering has been around for a long long time and is used to varying degrees on almost every known development platform to provide access to large datasets without slowing down the UI. For web developers, showing millions of rows of data is easily achieved by using a Javascript data grid that can pull in those rows as the user scrolls through the grid.
I’m definitely not the first one to attempt the implementation of virtual rendering in d3. I came across a d3 Google Group post where using d3 for virtually rendered grids was being discussed and found that Jason Davies had toyed with this a couple of years back. There were many other experiments as well, but I needed one that could render more than just a grid or simple chart.
My solution began with Jason’s longscroll.js class, which I heavily modified to allow the user to specify enter/update/exit methods that the plugin uses for rendering rows. The general idea is that you determine how much data can be shown on the screen at any given time and then slice out that amount of data from the master dataset and render it. As the user drags the scrollbar along, you continue to slice out corresponding ranges of data from the dataset and render them on screen. Ideally, you re-use the components already created on the screen and simply change out the data that backs each node with a new node from the sliced range to avoid the expense of adding and removing DOM elements.
For this plugin, here is what you provide:
There are several aspects to this solution that should be pointed out:
First, your visualization needs consistently sized rows so you can divide the visible height of the viewport by that size and calculate the range of visible rows. You also use the row height multiplied by the length of the entire dataset to size the scrolling DOM node container so that the scrollbar is sized correctly. I imagine you can implement a version with nodes of different heights, but you will need to know the height of the un-rendered rows above and below the visible range if you want to size the scroll bar correctly.
Second, there is no need to load the entire dataset into memory if you have the ability to get the total size of the dataset from the server and then page in subsets of data as they are needed.
Third, you can also use this solution to render tree structures but things get more complicated if you plan to page in that data on an as-needed basis. More on that in a future post once I get it all figured out. 🙂
I received a comment on the Flow Tree Layout post asking if it was possible to use that same layout multiple times within a single SVG. The code in that example created an SVG inside of a dom node passed in by the caller, but it is possible to modify that sample to accept any valid SVG container for executing the layout. I created a quick and dirty sample GIST showing how it might be done.
I also put together another d3 demo showing how the flow layout could be used within multiple containers within the same SVG. This is a much more component-ized implementation which lets you create any number of containers, each with its own layout (all are flow layout in this case). You can resize the containers below to see their individual flow layouts operate. This demo also includes z-ordering and selection logic (drag on the background to do a lasso selection of multiple containers) all implemented using d3. Gist is here and bl.ocks version is here. Enjoy 🙂
Multiple Flow Containers
Here is a quick and dirty demo to show how one could reuse the pan and zoom code for other types of layouts. This is in response to this inquiry from the d3 forum.
See the Pen cwbjo by Bill White (@billdwhite) on CodePen.
I received a question from a commentor asking how one might use the Pan and Zoom demo with a force layout. I came up with a couple of ways to do it.
Option #1
The first variation shows the quick and dirty approach which essentially bypasses the nice, reusable structures I spent time implementing in the first demo. I simply added a method named ‘addCircles()’ to the canvas which takes an array and creates a data-bound selection of circles. Then that selection is passed to the forcelayout:
See the Pen gmult by Bill White (@billdwhite) on CodePen.
Option #2
The second variation tries something a little different. The commenter wanted to take advantage of the ‘addItem()’ method in the canvas. This means that we need to modularize the circles themselves rather than work with a typical d3 selection which is referenced by the force layout. I added a forcecircle.js class which represents each circle being shown on the canvas. This lets us create each one individually and we can add them at runtime whenever we want. In the demo below you can see I start with a group of circles and the ‘add’ button can be used to append new ones at runtime:
See the Pen d3 Minimap Pan and Zoom Demo – Force Directed 1 by Bill White (@billdwhite) on CodePen.
There are lots of improvements that I think could be made here. While this setup works, it lacks that ‘d3-feel’ where we can take advantage of selections and their inherent ‘enter/update/exit’ lifecycle. Also, I have yet to determine if it is better to require calls of the ‘render()’ method to redraw the component (in this case, our forcecircle component) or if that should happen anytime a relevant accessor is executed. For example, should an cx/cy update to the circle be done like this:
circle.cx(newX).cy(newY).render();
or like this:
circle.cx(newX).cy(newY);
(second way assumes accessors call render() internally when the values changes)
The second way is cleaner, but results in two calls to the render() method: one for the cx() accessor call and one for the cy() accessor call. I’m hopeful that, over time, we will find a better way to build reusable d3 components. Right now it feels like the more modular things get, the further away we get from using the selections and lifecycles that make d3 so great.
I put together a little demo to show how to setup a canvas with a corresponding minimap that displays an overview of the contents of the canvas. You can use the mousewheel on either the main canvas or the minimap to zoom in and out. You can also drag either the image or the minimap frame to pan around. Notice how these actions (pan/zoom) from the canvas update the minimap and vice versa. I wrote the demo in the reusable coding style put forth in Bostock’s Towards Reusable Charts. The trickiest part was preventing the two different pan/zoom handlers (the canvas and the minimap) from getting out of sync with respect to setting the scale and transform values. Prior to getting that worked out, I kept hitting the common problem of trying to use multiple pan/zoom controls to operate against a single zoom behavior (discussed here, here and here). I managed to work it out by updating the transform and scale values on both controls at the correct time and place. If you have any suggestions for improvements, I’d love to hear them.
See the Pen d3 Minimap Pan and Zoom Demo by Bill White (@billdwhite) on CodePen
I had a question in the comments for the treemap label wrapping post about how to go about truncating the labels within the d3 Circle Pack example from Mike Bostock. Here is what I came up with. For the sake of brevity, I used the SVG text element so you get truncation but not wrapping. I’m sure the math could be improved at bit but it works well enough for this example:
Circle Pack Label Truncation
You can find the Gist here.