How does the Canon EF-S 15-85 IS USM compare to the EF 24-105mm L as an all-around lens?

I am looking for a upgrade from my kit lens to a lens with much more potential, a lens that would be an all-rounder for me.

After some research I ended up with these two in the final list:

Canon EF-S 15-85 f3.5/5.6 IS USM and Canon EF 24-105 IS USM f4 L

My question is have any of you tried these two and what are your results with them?

1. Does the small price difference make sense in buying the L lens over the non L?

2. Is there a big difference in sharpness between them?

3. How do they cope with chromatic aberration?

As both have USM focusing I don’t think there would be a difference in focusing speeds. The min-range and max-range difference is negligible in my situation as I shoot mostly portraits so I wouldn’t be using the 15mm or 24mm end too often.