(Here are Part 1 and Part 2.)
text: Lev Manovich.
VISUALIZING EVOLUTION IN STYLE SPACE: 1D
Many images sets have a time dimension. For instance, we know a year and a month for most of van Gogh's paintings; for manga titles, we know the position of each page in the title sequence.
How can we see study temporal patterns across a sequences which may contain thousands of images? We can map images positions in a sequence mapped into X-axis, and one of their visual features into Y-axis. If we use points and/or lines to represent each image, the result is a familiar line graph.
Here is an example: We place 776 images of Vincent van Gogh paintings (1881-1890) horizontally according to their dates (a year and a month a painting was created). Brightness median values of the images determined their vertical positions.
We can also place original images on top of the points, like this:
Lets use this technique to compare temporal changes in saturation in sets of Mondrian and Rothko paintings.
data: 128 images of Piet Mondrian's paintings (1905-1917).
X-axis = a year and month a particular painting was created.
Y-axis = saturation median.
data: 205 images Mark Rothko paintings (1934 - 1970).
X-axis = a year a particular painting was created.
Y-axis = saturation median.
These visualizations also illustrates two ways to deal with a typical problem in historical data: we don't always know exact dates. Van Gogh visualization shows one solution: since we do have year and month for most of the paintings, we use this for X-axis - which means that images of paintings done in a particular month and share the same X coordinare. Mondrian and Rothko visualiations show a different solution: here we only know a year, so to avoid having all images from one year being rendered in a single column and thus covering each other, we randomize images X coordinates within each year. The result is easier to read, and it does not effect the larger patterns we may observe.)
When we are dealing with sequential art such as comics and manga, we don't have this problem: we can place images on X-axis according to their exact position in a narrative, like in the following example.
data: all pages of a webcomic Freakangeles published on the web over a year and an a half period (Feb 15, 2008 - June 6, 2009). Each week, one episode consisting from siz pages was released (57 episodes; 342 pages).
X-axis: pages are placed according to their publication sequence, left to right.
Y-axis = brightness mean.
Like print comics and manga, web comics may run for years with new episodes added daily, weekly, or monthly. How does their visual style change over the duration of publication? Are the temporal patterns gradual or abrupt? How do these patterns relate to development of a narrative?
Despite the weekly intervals that separate the episodes of Freakangels, our visualization shows that its visual form is remarkably consistent. For the larger part of the publication period, the changes in brightness (the same applies to hue and saturation) follow a smooth curve. Visualization reveals this unexpected pattern and allows us to see the exact shape of the curve.
VISUALIZING EVOLUTION IN STYLE SPACE: 2D
The visualization examples in the previous section shows changes in values of a single visual feature (for example, average brigtness or average saturation) over time. Can we visualize evolution of an image sequence along two dimensions (i.e., two features)?
Lets look again at our earlier "style space" visualizations. They are 2D scatter plot with (optionally) images rendered on top of the points. The visual features of images analyzed automatically with digital image processing software become X and Y coordinates of the points.
127 paintings by Piet Mondrian created between 1905 and 1917.
Left plot: each image is visualized as a point.
Rigt plot: the images are rendered on top of the points.
X-axis = brightness median.
Y-axis = saturation median.
If we stick with points, we can vary aspects of their apperance - brightness, hue, transparency, size or shape of points - to carry additional information. So if we want to see how feature values change over time, we can vary one of these visual variables in accordance to dates (or image position in a sequence). This simple trick allows us to add a third dimension of time to a 2D style space visualization. We can now trace evolution of image sets regardless of their size in a 2D style space. (If we want to follow the evolution in a space of multiple features, we can simply make multiple 2D plots.)
For example, to see how Mondrian and Rothko moved through brightness/saturation space during the periods we are comparing, we can visualize each painting as a color circle and vary hue in accordance to dates. Our Mondrian set covers cover the period from 1905 to 1917. We will use pure blue (R=0, G=0, B=255) for 1905 paintings and pure red (R=255, G-0, B-0) for 1917 paintings; all others will take on in-between color values. (The art historical sources only give a range for some of the paintings: for example, the dates for Still Life with Gingerpot II given by Guggenheim Museum NYC which owenes this painting are 1911–12. In these cases, we used an intermediate values, i.e. 1911.5 to set points hue in the graphs).
Our Rothko subset which we used before for comparison with Mondrian runs from 1938 to 1953. Here, pure blue points will represent 1938 images, and pure red will represent 1953 images. To make patterns even easier to see, we will also vary the size of the points. Smallest circle represents the first year, and largest circle represents the last year.
First visualization shows images, the second uses color points.
X-axis = brightness mean.
Y-axis = saturation mean.
X-axis min = 0; X-axis max = 250.
Y-axis min = 0; Y-axis max = 250.
Using color to represent time reveals that Rothko starts his explorations in late 1930-1940s in the same same part of brightness/saturation space where Mondrian arrives by 1917 - high brightness/low saturation area (the
right bottom corner of the plot). But as he develops, he is able to move beyond the areas already “marked” by his European predecessor (i.e., Mondrian). (Keep in mind that these visualizations are only meant to illustrate the idea of a style space and the different techniques to visualize it. If we want to reach more definite conclusions, we will need to extend our Mondrian and Rothko image sets to ideally include all paintings from their complete careers.)
We can also apply this technique to sequential art scuh as comics and manga. For instance, lets visualize "Tetsuwan Girl" manga title by Takahashi Tsutomu (1094 pages). First, we will plot all pages as images. We will use the same features as in our earlier visualization of the complete set of one million manga pages: standard deviation (X-axis) and entropy (Y-axis). These features allow us to capture an important stylistic dimension. The pages that are more graphic, have high contrast, little detail, and no texture end up in the upper right of the visualization; the pages which are visually opposite (significant amounts of texture and detail, more gray tones) end up in the lower part; all intermediate pages position between these two extremes.
"Tetsuwan Girl" manga by Takahashi Tsutomu (1094 pages).
X-axis = standard deviation
Y-axis = entropy.
Both features are calculated over grayscale values of all pixels in each page.
Now, lets visualize the same data as points and vary their hue. As we did with Mondrian and Rothko, we will use blue-red gradient to represent time - specifically, the position of a page within the title sequence.
The cluster of blue dots corresponding to earlier pages is below the cluster of red dots corresponding to latter pages, and the change appears to be gradual. This tells us that the pages in the first part of the manga use
less texture and detail than the pages in the second. We can also see many violet points which are vertically in between the blue and the red clusters. This indicates that the transition between the two types is gradual.
STYLE SPACE MATRIX
Since we have 883 manga titles in our data set, can we use "style space" visualizations with colored points to
compare the patterns of graphical change in all the titles?
Borrowing from the standard visualization technique called "scatter plot matrix" and also Edward Tufte's concept of small multiples, we can visualize each title using the same features for X-axis and Y-axis, and organize all visualizations in a grid. (It is important to use the same ranges for range for X-aixs and Y-axis in each graph, so they all have the same scale.) To use the analogy with a "scatter plot matrix," we will call such a visualization a "style space matrix."
The following example shows a part of such style space matrix for our manga data set of 883 titles. In each plot, the pages are mapped in the same way as in the previous examples (X-axis = standard deviation,
Y-axis = entropy; pure blue = first page; pure red = last page). The name of a title and the number of pages appear in the upper right corner of its plot.
The mapping of pages positions into color values creates distinct and easy to read visual patterns. They indicate whether a style in a given title changes over the period of its publication. You can quickly scan the style space matrix to see which titles have unusual patters and should be investigated more closely. You can also divide titles into different groups depending on their graphical development in time: no or very little development, gradual change over time, significant and fast changes, and so on.
(Of course, remember that we are only using two visual features which capture some but not all stylistic dimensions.)
(End of part 3.)
Follow to part 4.