r - ggplot2: Multiple color scales or shift colors systematically on different layers?

Question

When I make box plots, I like to also show the raw data in the background, like this:

library(ggplot2)
library(RColorBrewer)

cols = brewer.pal(9, 'Set1')

n=10000
dat = data.frame(value=rnorm(n, 1:4), group=factor(1:4))

ggplot(dat, aes(x=group, y=value, color=group, group=group)) +
  geom_point(position=position_jitter(width=0.3), alpha=0.1) +
  scale_color_manual(values=cols) +
  geom_boxplot(fill=0, outlier.size=0)

However, I don't like it how my box plots completely disappear when the points get too dense. I know I can adjust alpha, which is fine in some cases, but not when my groups have varying densities (For example when the lightest group would completely disappear if I were to decrease alpha enough so that the darkest group doesn't obscure the box plot). What I'm trying to do is systematically shift the colors for the box plots - a bit darker, perhaps - so that they show up even when the background points max out the alpha. For example:

plot(1:9, rep(1, 9), pch=19, cex=2, col=cols)
cols_dk = rgb2hsv(col2rgb(brewer.pal(9, 'Set1'))) - c(0, 0, 0.2)
cols_dk = hsv(cols_dk[1,], cols_dk[2,], cols_dk[3,])
points(1:9, rep(1.2, 9), pch=19, cex=2, col=cols_dk)

So far I haven't found a way to fake in a different scale_color for the geom_boxplot layer (which would seem the simplest route if there's a way to do it). Nor have I been able to find a simple syntax to systematically adjust the colors the same way you can easily offset a continuous aesthetic like aes(x=x+1).

The closest thing I've been able to get is to completely duplicate the levels of the factor...

ggplot(dat, aes(x=group, y=value, color=group, group=group)) +
  geom_point(position=position_jitter(width=0.3), alpha=0.1) +
  scale_color_manual(values=c(cols[1:4], cols_dk[1:4])) +
  geom_boxplot(aes(color=factor(as.numeric(group)+4)), fill=0, outlier.size=0)

but then I have to deal with that ugly legend. Any better ideas?

Answer 1

Late answer added Nov 2012:

Since some of these terrific answers require older ggplot2 versions and people are still referring to this page, I'll update it with the ridiculously simple solution that I've been using with ggplot2 0.9.0+.

We just add a second geom_boxplot layer that is identical to the first one except we assign a constant color using scales::alpha() so the first boxplot shows through.

library(scales) # for alpha function
ggplot(dat, aes(x=group, y=value, color=group, group=group)) +
  geom_point(position=position_jitter(width=0.3), alpha=0.2) +
  geom_boxplot(size=1.4,fill=0, outlier.size=0)+
  geom_boxplot(size=1.4,fill=0, outlier.size=0, color=alpha("black",0.3))

edit: TobiO points out that fill=0 has stopped working. Instead, fill=NA or alpha=0 can be substituted. This seems to be due to a change in col2rgb() starting in R 3.0.0.