xplot.org on Time Series Events

Visualizing events in a distributed system is tough.  There are good solutions for experimentation on a single machine, but once the software lives on multiple machines issues crop up.  The number one issue is timing synchronization.  Out of the box NTP based time synchronization is not accurate enough (I have found) to deliver proper time synchronization.

Precise Time Synchronization

Success (on Linux) involved installing and running ptpd2 (PTP stands for Precision Time Protocol) available on sourceforge.

On the master (fileserver machine) I keep ptpd2 running with:

 ./ptpd2 -W -b eth0

And on the slave compute nodes I just keep them running in sync with the master using:

 ./ptpd2 -c -g -B

Together, these processes keep the cluster time synchronization locked to around 1 microsecond - which is sufficient for my immediate needs.

Enter xplot.org

When looking into a TCP/IP issue, I discovered tcptrace which is a program for analyzing the output of tcpdump or snoop (on Solaris).  Several visualizations of a TCP/IP session captured by those sniffing tools are available, and (most importantly) they are viualized within a tool called xplot.org.  This tool is an X windows program which can efficiently display large time series data sets in a 2D graph.  Unfortunately, its text-based data format is only really documented by reviewing its C source code.  I intend to share what I have learned about the format so that (hopefully) others can skip reading the source.

Most of the commands are issued on a single line, with the exception of commands that take a text argument.  Commands that display text in the graph require the subsequent line for the text content.  Most drawing commands looks like:

 commandname x-coordinate1 y-coordinate2 [ x-coordinate2 y-coordinate2 ] [ color ]

Configuration of a time series graph that uses UTC timestamps (second resolution with microseconds expressed after the decimal part) is done using:

timeval double

Using three text commands we can label (using xlabel and ylabel commands) the axes and title the graph itself:

title
Results from Tue Apr 24 11:45:55 2012 - run Tue Apr 24 11:48:00 2012
xlabel
Wall clock time
ylabel
Event id

Adding some lines is easy with the line command:

line 1335282355.550356 1 1335282355.550356 2 yellow
line 1335282355.550356 2 1335282355.620854 2 yellow
line 1335282355.620854 2 1335282355.620854 3 yellow
line 1335282355.620854 3 1335282355.691357 3 yellow
line 1335282355.691357 3 1335282355.691357 4 yellow

Adding some more random things:

dtick 1335282355.623986 2.004608 blue
line 1335282355.623986 2.004608 1335282355.624049 2.004608 blue
diamond 1335282355.624049 2.004608 green
diamond 1335282355.624101 2.005530 green
line 1335282355.624049 2.004608 1335282355.624101 2.005530 green
line 1335282355.624049 2.004608 1335282355.624514 2.002304 gray20
box 1335282355.624514 2.002304 gray20

Finally a look at how xplot.org paints all of this data in its viewer:

 Zooming in on the green diamond (left button on my mouse):

Zooming in once more:

Finally we can see the event tree that I am depicting here (the Y-axis is mostly used for layout of the tree), and the X-axis shows accurate timing information:

The middle button (or wheel button) on my mouse helps to pan in one direction or another, and a single left-click is used to pop back through the zoom-ins you have done recently.

Summary

If you have lots of data to process and are comfortable generating an xplot file using your favourite tool (programs like awk and Perl are useful for transformations like this), then you may get some mileage out of xplot.org for visualizing moderate amounts of time series data (100s of megabytes) using a modest Linux computer (e.g. a netbook).  More serious hardware is definitely capable of more.  I have not found an in-browser viewer which is close to being as fast as the native xplot.org program.