d3 Wrapping Flow Tree Layout

One d3 visualization I’ve been refining for a while now uses a flowing tree layout. I wanted to have a structure similar to the standard d3 tree layout which would position consistently-sized leaf nodes (children) in a recursively flowing (wrapping) arrangement. I also wanted to take a stab at building this layout as a library that followed the examples set forth in discussions of the ‘Reusable Charts’ pattern that has been such a hot topic lately. There are many great libraries out there that have attacked this problem in various fashions and it will take me a while to determine which one makes the most sense for my purposes. In the meantime, I wanted to share what I’ve come up with for a flowing tree chart:

Resizable Flow Tree (Expand/Collapse nodes & resize using space between chart and scrollbar)

Features:

Flow Layout
I’ve taken the logic from the d3 tree layout and adapted it to recursively size and space out the children of each container given the available width. What you end up with is a flowing version of the Indented Tree example.

Resizable Visualizations
The chart will update its layout when it is resized (Try it by putting the mouse cursor just the left of the scrollbar and resize the container and watch the layout do its thing). Most d3 examples that I have come across do not address resizing issues at all, something that is not surprising given the complexity involved. I’ve seen many developers asking about ways to create resizable charts in d3 and in most cases the replies suggest that they scale the SVG contents to fit the available space within the viewport. Personally, I do not find this very useful. Instead, I want to keep the size of the visualization constant (i.e. legible) and reapply the layout to make use of the available space and adjust the scrolling size as necessary.

Expanding and Collapsing Groups
The layout includes a variation of the expand/collapse logic found in the Indented Tree example. It also adjusts the container’s scroll size based on the updated chart height once expand/collapse actions have changed the size of the chart.

Recursive Layout
I’ve come across a few examples that feature wrapping style layouts, but what they did not address is recursive containment. They would not measure and layout containers within containers. This flow layout walks the hierarchy, sizing and positioning groups within groups as well as their children.

Possible Enhancements
There were a couple of ideas that I did not get time to implement yet. It might be nice to allow the children to vary in size. I may also look into justifying the children contained with each row rather than leaving the unused space tacked onto the right edge.

Related Links:
Stand alone demo of the visualization shown above
Gist showing the flow tree that resizes with the containing div. (same as shown in this blog post)
Flow Tree that resizes with the browser window
Gist showing the flow tree that resizes with the window

d3 Treemap Label Truncation

I received a comment asking how to truncate labels in my treemap example. Here is a quick example showing how I handled it on a recent project. I am sure there are other ways to do this and if anyone has a better alternative, please detail it in the comments. Basically, when using labels in D3, I find it easier to work with HTML DIVs rather than the SVG Text nodes. So, using the ForeignObject element within the SVG, I add a DIV containing the label and I can control the wrapping, truncation, etc using styling. Internet Explorer, as usual, is years behind the other modern browsers so you have to “speak slowly” when IE is detected and dumb things down a bit so it can keep up. I basically have a little function that determines the length of the label text and if it exceeds the width of the treemap rectangle, I truncate it manually. See the Gist for the code and see the demo below. (Keep in mind that I have not taken the time to deal with the parent group labels so you will see some over lap from those)
NOTE: The labels do not show up in IE10 even though they work in IE9. Big surprise. Maybe one day Microsoft will make a browser that works. I’m sure I could dig around and figure out why IE cannot handle these simple labels but until then, just pretend.

 

Two More Intellij Plugins Updated

Just an FYI for IntelliJ users: I’ve updated 2 more existing plugins that had been left for dead:

Both of these were plugins that looked mildly useful but would throw errors when used with the latest version of the IntelliJ Idea. The email address for the original author of the SVG Viewer’s was no longer valid so I just reversed the binaries and fixed it so you can now view SVGs within the IDE. (That plugin now works in pretty much all Intellij platformed products except AppCode which appears to not have the Batik library in the classpath by default; however, I was unable to figure out how to specify ‘everything except AppCode’ in the dependency section of the plugin.xml so I just left it as Intellij Idea only). As for Code Lock, the source was available and the author said he done with the plugin so I just patched it and posted it. The source for both plugins is available from my Github page. Hopefully others will get some use out of these.

SyncEdit 2 – An (updated) IntelliJ IDEA Plugin

I have been a huge fan of JetBrain’s IntelliJ IDEA development tool for nearly a decade. I use it almost daily and I love that the makers of this tool have included a path for users to add functionality to the IDE on their own using custom plugins.

One feature that I have missed is SyncEdit which works somewhat like the variable refactoring currently available in IntelliJ. However, where as the current refactoring functionality lets you operate on code variables that the editor recognizes, SyncEdit allows you to highlight a custom range within the editor and then change any word within that range while simultaneously updating any corresponding words that also exist within the selected region.

This feature was available as a plugin which was originally written by Nathan Brown but has not been updated in over six years and simply no longer works with the latest version of IntelliJ. I tried to contact him to get his original source code but I never heard back. Since there was no license on the code and it was published on the IntelliJ plugins site for free, I simply reversed the available plugin, got it working and re-posted it as an open source implementation in hopes that it can be used and improved. (UPDATE: Nathan contacted me and supplied me with his original source so I hope to update the 2.0 codebase soon with anything that I missed)

I have never written an IntelliJ plugin before and you need to understand some of the inner workings of their architecture to truly know what you are doing. However, I was able to get it up and running again and I even tweaked a few thing to get the plugin to select only words instead of subsets of words as the first version did since it seemed to cause more problems than it solved. I was able to find on a few other blogs with some tips on ways to implement SyncEdit and get whole word selection working better. The code certain needs some tweaks but it seems to work at least as well as the previous version and better yet, it is open for anyone else to help improve it.

See the image below for a better idea of how it works. You just highlight a range of code, then select a word and change it. Any other instances of that word will also be updated. It ends up being a more flexible method of find-and-replace. Yes, there are other ways to do this that already exist but I find this way to be very intuitive and easy to execute.

You can get the source code here from GitHub and the plugin file is here. The IntelliJ plugin page is here and is awaiting approval.

d3 Treemap with Title Headers

The D3 library is really a great way to create data visualizations in HTML5. I absolutely love it and its treemap is especially powerful. I wanted to create a treemap that showed grouping headers like the JIT version that has been around for a while, but it took a little more work to get D3 to do the same thing. I came across a post that showed the attempts of others to create the same thing and I wanted to show the code of how I did it in case others are interested.

I created two versions: the first uses SVG labels which are nice because you can measure them to decide what room is available for displaying a label in a given cell. However, wrapping is much more involved and ended up creating a second version which uses embedded HTML divs to create self wrapping labels which seem to look a little better. I also modified the examples of others in the thread so instead of showing labels where there is enough room to see them, I instead took the approach of showing labels for given group when that group was selected.

(NOTE: The labels do not work in IE in the first and third demos because IE is a horrible browser. I could write checks to use svg elements instead of foreignobjects for the labels but you get what you need from the second demo if you want IE compatibility.)

 

 

So here is the first example which uses divs for the labels: (link to code) (bl.ocks.org)

 

 

…and here is the second version which uses SVG text elements: (link to code) (bl.ocks.org)

 

UPDATE:
Here is the third version which uses IDs instead of the name so duplicates titles will work (notice the two ‘display’ groups on the left side): (link to code)

 

UPDATE #2:
In the comments for this post I received a request to show how one might highlight the children on mouseover. I’ve modified one of the above examples to demonstrate this:

Basically, I just added code to modify the child blocks on mouseover like this:

    var childEnterTransition = childrenCells.enter()
        .append("g")
        .attr("class", "cell child")
        .on("click", function(d) {
            zoom(node === d.parent ? root : d.parent);
        })
        .on("mouseover", function() {
            this.parentNode.appendChild(this); // workaround for bringing elements to the front (ie z-index)
            d3.select(this)
                .attr("filter", "url(#outerDropShadow)")
                .select(".background")
                .style("stroke", "#000000");
        })
        .on("mouseout", function() {
            d3.select(this)
                .attr("filter", "")
                .select(".background")
                .style("stroke", "#FFFFFF");
        });

I added an SVG filter (in the section) and just modified the background border stroke of each child on mouseover. I also called appendChild on the parent container to make sure the moused-over child had the highest z-index within the groups of child nodes, otherwise the filtering gets clipped by any child added after the current child. Here is a Gist showing the code.

UPDATE: #3:
I received a question about how to space out the parent/child groups so I put this demo together to show one possible avenue to implement this. I essentially increased the treemap layout padding and then used the d.dy value to size the parent background to occupy the container’s full height rather than just the header height. (It is not necessary for this demo but you might need to sort your nodes to make sure the parent nodes are added before the child nodes and in the correct order to avoid having larger squares covering smaller ones.). To see the changes you can pull the source code for all of these treemap demos here and then do a diff between treemap_header_01.html and treemap_header_05.html to see the changes made to implement this. (the gist shows the treemap_header_02.html demo first so scroll down to find the source for the others)

PureMVC Pipes Ported to Native Javascript

When I ported PureMVC over to Dojo I could not pass up the chance to port over the the Pipes utility as well. I initially created a Dojo-flavored version, but since the main Javascript distribution of PureMVC is framework independent, I figured that Pipes should be as well.

For those who are unfamiliar with Pipes, it is basically a way to create multiple PureMVC applications, each with its own core, so that they can be loaded and unloaded as modules within a main application or shell. It can get a little confusing and may appear to be overkill at first glance, but I think the end result proves to be worth the effort.

I posted the native version here. I also created a variation that uses the puremvc.define function to create a more clearly defined OO structure. I really like that approach better but the first version mimics the coding style found in the currently supported PureMVC Javascript implementation.

I also created a demo based on Tekool’s AS3 Pipes Demo:

UPDATE: Here is a zip file containing the source for both demos. It is a tad large since it has a few binaries and dojo src libs (for the dojo version only of course).

If you are looking to learn more about Pipes, check out the following links which cover the conceptual use of Pipes, albeit in a Actionscript/Flex format.

(Update: I had to change the reference from Shell.js to Shelll.js because wordpress seems to think that any file named Shell.js is dangerous and blocks access to it which was breaking the demo. I updated both html pages with Shelll which points to Shelll.js to make it work)

PureMVC in Dojo

Having moved on from Flex to Javascript more than a year ago, I’ve continually run into the problem of trying to find frameworks to support large-scale Javascript applications that are truly robust and scalable. It is harder than I thought, and that is at least partially due to the inherently fast-and-loose nature of Javascript. In the Flex world, Actionscript 3.0 was a dream for Java coders like myself that loved strongly typed languages with deep OO roots. Javascript is whatever you want it to be at any given moment: it can be as structured or chaotic as the author desires and still work. This is what makes it so malleable and useful while managing to make me insane at the same time.

Scalable systems however, require structure. Sure, you can build a scalable application using quick and dirty techniques, but you will be leaving out an overlooked and vital component: maintainability. If you are the only one who can figure out what the hell the code does, then what good is it? If no one can build upon the scaffolding you have provided, you can end up with a massive hotel with only one occupant.

There are a ton Javascript frameworks out there, but in the Flex world, my framework of choice was PureMVC. As far back as 2008-2009 PureMVC fans wanted to create an implementation using Javascript. Almost immediately they ran into the quandary of how to use it with their own libraries such as ExtJS, Prototype, Mootools, etc. Individual developer started porting it to their framework of choice, but this meant having multiple codebases to maintain instead of one. The decision was ultimately made to settle on a single native version that could used by everyone. At the time, I was working with what would eventually become Dojo 1.7, complete with AMD which was the “next big thing” to hit the Javascript world. I created my own port of PureMVC to work with Dojo 1.6 using AMD and I recently posted it here for anyone who needs it. In fact, I went one step further and created a Dojo version of the PureMVC Pipes utility as well. It is great to be able to load the PureMVC code on an as-needed basis with AMD.

However, Cliff is correct that in the long run, someone will have to keep that codebase in sync with any future PureMVC changes that come down the pipeline. So the best move is probably to utilize the native implementation of PureMVC and integrate it with the framework of your choosing. The issue then becomes: “How do I integrate it with ?”. Some developers have shown us the way for libraries like Prototype, ExtJS and others. However, there is not as much information on how to use PureMVC-JS native with Dojo, so I decided to post some examples on how to do it based on Cliff’s suggestions.

The first demo shows you how to create your implementation classes for Mediators, Proxies and Commands using PureMVCs “define style”, which means you use the puremvc.define function to create the OO structures that allows your MyMediator class to extend from Mediator in the PureMVC library. Only the main application and the view components are writing with Dojo. The argument for this way of integrating Dojo with PureMVC is that none of the PureMVC classes are reliant upon your framework and if you choose to switch from Dojo to something else, your PureMVC classes will required very few modifications. This makes sense, but I’m not sold on this entirely. First off, if you chance from Dojo to another framework, even if your PureMVC classes still work, the other 80% of your code will have to be changed. On top of that, while I like the clear and easy to use puremvc.define functionality, I would rather have all of my Dojo code look like Dojo code rather than have lots of code using puremvc.define while the rest uses Dojo’s define() function and style. Also, if you use the puremvc.define function, you have to make sure you load those classes up front and order them correctly. You get none of the beauty that AMD provides when it comes to loading what you need when you need it.

Run Demo 1

The second demo shows the same application but this time, the Mediators, Proxies and Commands are all built with Dojo and inherit from the classes defined in the PureMVC library. So I only have to load the puremvc.js file at the start of the application and my Mediators, Proxies, and Commands will all load when they are needed. Dojo will also include them in compressed layer file which is created by my build process prior to deployment so I get a clean seperation between the code I wrote and the code supplied by the PureMVC library.

Run Demo 2

The second method just feels better to me, but you decide for yourself: Here is a code-comparison of a controller class for each demo:

PureMVC define style:

puremvc.define({
        name: 'todomvc.controller.command.PrepModelCommand',
        parent: puremvc.SimpleCommand
    },

    // INSTANCE MEMBERS
    {
        /**
         * Register Proxies with the Model
         * @override
         */
        execute: function(notification) {
            this.facade.registerProxy(new todomvc.model.proxy.TodoProxy());
        }
    }
);

Dojo Style:

define(
    [
        "dojo/_base/declare",
        "todomvcdojo/model/proxy/TodoProxy"
    ],
    function(declare, TodoProxy) {

        var PrepModelCommand = declare("todomvc.controller.command.PrepModelCommand", puremvc.SimpleCommand, {
            /**
             * Register Proxies with the Model
             * @override
             */
            execute: function(notification) {
                this.facade.registerProxy(new TodoProxy());
            }
        });

        return PrepModelCommand;
    }
);

Both work fine but for now I’m sticking with the second approach. 🙂

Flex, HTML5 or iOS: Where are things going?

Given the big announcements this past week from Adobe regarding their abandonment of the mobile Flash Player, along with their statements about their plans for Flex, I figure it is time for me to chime in with my own thoughts.

I cannot say I was surprised by these announcements. Technology waves happen all the time and I have jumped to different platforms many times over the past two decades. In my PwC and IBM Consulting days, I worked in Lotus Notes and Domino until it became easier to build database driven web applications with Java Server Pages sprinkled with Javascript (soon to be Ajax) code. Then in 2003, I decided to jump again, this time to whatever could save me from the hacky nature of Javascript and pain of multi-browser support.

So I started looking at alternatives. Flash was an obvious choice but was still mainly for artistic designers looking to create static content. Open Laszlo initially won me over with it’s ease of use and low cost. At least until Macromedia jumped into the fray with Royale (which would ultimately become Adobe Flex a couple of years later) shortly afterward. It would take another year before Adobe would figure out that no one wanted to pay $15,000 per CPU for a server to generate SWF files, especially since that step is unnecessary to begin with. By the time Flex 2.0 came out, Adobe had wised up on how to monetize Flex and things really took off. Flex and the Flash Player had finally accomplished what Java Applets and the JVM set out to do but never achieved: provide a portable and consistent web-based platform for complex applications. I jumped on the Flex bandwagon and never looked back…until a little over a year ago.

That’s when it was becoming apparent that Steve Jobs was slowly being proven correct about the shortcomings of the mobile Flash experience. At the time, I did not feel they were wrong but they did not offer any real alternatives. They claimed HTML5 was the future but since no browsers fully implemented that specification (and still don’t), it seemed like a non-starter. The goal was still to write cool-looking, complex applications in a real programming language using a powerful IDE for a consistent runtime environment, and the Flex+Flash combination still seemed like the only offering that provided anything close to that.

However, the more I looked again at HTML and Javascript, I realized that the tooling had dramatically improved, especially with the emergence of two things: javascript libraries and browsers with debugging facilities. I knew those technologies had appeared years before, but it was their maturity that now caught my attention. Unlike some developers, I fully embrace the latest technology available to programmers. I’m a firm believer in IDE’s over a vi editor, debuggers over trace statements, and profiling tools over timestamps placed at the start and end of methods. It now seemed possible to create dynamic web apps that ran well in more than one browser, and the whole thing could be developed quickly without going blind from staring at alert dialogs with vague ‘blah is undefined’ javascript error messages. Don’t get me wrong, Javascript is still every bit as hacky as before, but jQuery, Dojo and ExtJS help shield you from much of that pain.

The real challenge will be to figure out how to create the same results in HTML5/Javascript that I was doing in Flex. Take drawing for example: the capabilities of SVG and the HTML5 Canvas fall short of what the Flash player can do when combined with Actionscript and Flex, but tools like EaselJS, Raphael, and JQuery SVG are avenues to get you part of the way there. Even Keith Peters is working on a book to help Flex developers make the transition that strongly mirrors his Actionscript version.

I honestly can say I’m surprised at how brief the Flex wave has been. I figured it would last 10 years, but it seems to be more like 6 if you figure that Flex 2.0 was released in 2006 and I expect a fairly strong decline by the end of 2012.

And no, Flex will not go away but it is reasonable to expect that the demand for it will. I will continue to use it where applicable, but over the past year or so, I’ve moved heavily into HTML5 and Javascript as well as Objective-C and the iOS platform. Seeing that the performance of HTML5 web applications running in mobile Safari is still a little slow for my tastes, native iOS applications have become very attractive. All of this partly explains the severe reduction in my blog posts over the past year or so, combined with the fact that I’m a father now and family time is very appealing these days.

So in summary, I’m still doing my best to create neat stuff, but these days it is with a different set of tools. Geoff Sterns sums up my thoughts exactly at the end of this post.

Flex 4.5 is out!

After weeks of waiting since the mid-April announcement, Flex 4.5 and FlashBuilder 4.5 are finally available. Download it here!