Lossy (VP8) — for photographs
Continuous-tone images: photos, renders, textures, anything with gradients. Lossy discards detail the eye does not track and lands 25-35% below an equivalent JPEG. At quality 80-85 the loss is invisible.
Lossless (VP8L) — for graphics
Logos, icons, line art, screenshots, charts and anything containing text. These have large flat areas and hard edges, which lossless compresses extremely well and lossy visibly damages — text picks up a distinctive fuzz.
The counterintuitive part
For flat graphics, lossless is often *smaller* than lossy as well as better looking. Lossy compression is optimised for photographic noise; give it a flat-colour logo and it wastes bits describing detail that was never there.
Screenshots are the tricky case
A screenshot of a text editor is graphics — use lossless. A screenshot of a video or a photo gallery is photographic — use lossy. Mixed screenshots are a judgement call; try both and compare.
Alpha works either way
Both modes carry a full 8-bit alpha channel, so you never have to choose between transparency and lossy compression the way PNG and JPG force you to.