Just double up and wear a short sleave jersey under the long and when it warms up take off the long sleave and fold and stuff into the pockets of your other jersey. I've also worn two pairs of leg warmers when it starts out cold and when it gets warmer as the day goes along I just stuff the top layer in a pocket.

It sounds like you need some arm warmers. They go a long way to help keep you warm and are small enough to easily stash in a pocket when you take them off. That may be the "long sleeve somethings" under the jersey that you saw too...

---

----
Don't hold back

Here's what I typically wear. I have a cold weather jacket that I love but it's a pain on days like you describe where it's expected to warm up. I want to wear something that's warm but I can easily shed and carry.

Short sleeve base layer
Short sleeve jersey
Bib shorts (these are not only more comfortable, but don't leave any gaps on the lower back that might get cold.)
Arm warmers
Knee warmers or full length leg warmers
Lightweight vest
Socks

Here are the keys:
Full finger gloves. I love my knit ones.
Lightweight skull cap (headsweats makes a great one)
Neoprene toe warmers

As it heats up I can shed the gloves, skull cap, vest, arm/leg warmers, and have them all in my jersey pockets.

I can ride all day in toe warmers and knee warmers in warm weather and never overheat. Arm warmers tend to make me heat up and are usually one of the first things to come off.