r - ggplot boxplot - length of whiskers with logarithmic axis

Question

I'm trying to create a horizontal boxplot with logarithmic axis using ggplot2. But, the length of whiskers are wrong.

A minimal reproducible example:

Some data

library(ggplot2)
library(reshape2)
set.seed(1234)
my.df <- data.frame(a = rnorm(1000,150,50), b = rnorm(1000,500,150))
my.df$a[which(my.df$a < 5)] <- 5
my.df$b[which(my.df$b < 5)] <- 5

If I plot this using base R boxplot(), everything is fine

boxplot(my.df, log="x", horizontal=T)

But with ggplot,

my.df.long <- melt(my.df, value.name = "vals")
ggplot(my.df.long, aes(x=variable, y=vals)) +
  geom_boxplot() +
  scale_y_log10(breaks=c(5,10,20,50,100,200,500,1000), limits=c(5,1000)) +
  theme_bw() + coord_flip()

I get this plot, in which the whiskers are the wrong length (see for example how there are many additional outliers below the whiskers and none above).

Note that, without log axes, ggplot has the whiskers the correct length

ggplot(my.df.long, aes(x=variable, y=vals)) +
  geom_boxplot() +
  theme_bw() + coord_flip()

How do I produce a horizontal logarithmic boxplot using ggplot with the correct length whiskers? Preferably with the whiskers extending to 1.5 times the IQR.

N.B. as explained here. It is possible to use coord_trans(y = "log10") instead of scale_y_log10, which will cause the stats to be calculated before transforming the data. However, coord_trans cannot be used in combination with coord_flip. So this does not solve the issue of creating horizontal boxplots with a log axis.

Answer 1

You can have ggplot use boxplot.stats (the same function used by base boxplot) to set the y-values for the box-and-whiskers and the outliers. For example:

# Function to use boxplot.stats to set the box-and-whisker locations  
mybxp = function(x) {
  bxp = boxplot.stats(x)[["stats"]]
  names(bxp) = c("ymin","lower", "middle","upper","ymax")
  return(bxp)
}  

# Function to use boxplot.stats for the outliers
myout = function(x) {
  data.frame(y=boxplot.stats(x)[["out"]])
}

Now we use those functions in stat_summary to draw the boxplot, as in the example below:

ggplot(my.df.long, aes(x=variable, y=vals)) +
  stat_summary(fun.data=mybxp, geom="boxplot") +
  stat_summary(fun.data=myout, geom="point") +
  theme_bw() + coord_flip()

Now for the log transformation issue: The plots below show, respectively, no coordinate transformation, scale_y_log10, and coord_trans(y="log10"). In addition, I've used geom_hline to add dotted lines at each of the box-and-whisker values and I've added text to show the actual values. To reduce clutter, I've removed the outlier points, and I've faded out the boxplots a bit so that the other components will show up better.

# Set up common plot elements
p = ggplot(my.df.long, aes(x=variable, y=vals)) +
  geom_hline(yintercept=mybxp(my.df$a), colour="red", lty="11", size=0.3) +
  geom_hline(yintercept=mybxp(my.df$b), colour="blue", lty="11", size=0.3) +
  stat_summary(fun.data=mybxp, geom="boxplot", colour="#000000A0", fatten=0.5) +
  #stat_summary(fun.data=myout, geom="point") +
  theme_bw() + coord_flip()

br = c(5,10,20,50,100,200,500,1000)

## Create plots

# Without log transformation
p1 = p + scale_y_continuous(breaks=br, limits=c(5,1000)) + 
  stat_summary(fun.y=mybxp, aes(label=round(..y..)), geom="text", size=3, colour="red") +
  ggtitle("No Transformation")

# With scale_y_log10
p2 = p + scale_y_log10(breaks=br, limits=c(5,1000)) + ggtitle("scale_y_log10") +
  stat_summary(fun.y=mybxp, aes(label=round(..y..,2)), geom="text", size=3, colour="red") +
  stat_summary(fun.y=mybxp, aes(label=round(10^(..y..))), geom="text", size=3, 
               colour="blue", position=position_nudge(x=0.3)) 

# With coord_trans
p3 = p + scale_y_continuous(breaks=br, limits=c(5,1000)) +
  stat_summary(fun.y=mybxp, aes(label=round(..y..)), geom="text", size=3, colour="red") +
  coord_trans(y="log10") + ggtitle("coord_trans(y='log 10')")

The three plots are shown below. Note that the last plot, using coord_trans is not flipped, because coord_trans overrides coord_flip. You can probably use something like the code in this SO answer to flip the plot, but I haven't done that here.

The first plot, with no transformations, shows the correct values.

The third plot, using coord_trans also has everything in the correct locations. Note that coord_trans is actually changing the y-coordinate system of the plot without changing the values of the plotted points. It's the space itself that's been "distorted" to a log scale.

Now, note that in the second plot, using scale_y_log10, the boxes are in the correct locations but the ends of the whiskers are in the wrong locations. On the other hand, comparison with the other two plots shows that the location of all the geom_hlines is correct. Also note that, unlike coord_trans, scale_y_log10 takes the log of the points themselves and just relabels the y-axis breaks with the unlogged values, while leaving the "space" in the which the points are plotted unchanged. You can see this by looking at the values in red text. The values in blue text are the unlogged values.

See @dww's answer for an explanation of why scale_y_log10 results only in the whisker ends being transformed incorrectly, while the box values are plotted in the right place.