You have to use your own judgement, and I'm not a health professional, but from my own stress fracture experience (metatarsal shaft, so different location) I would suggest not wearing the boot, but only if you can refrain from placing load on the foot with the stress fracture. ie, stay off your feet!

---

No coasting in running and no crying in baseball

There are some adjustments that can be made to the boot to increase your comfort. You were probably put in the boot to take the weight/stress/decrease movement in that area of the foot.
Depending on the brand/style/etc. of the boot, you might wind up with a different "insole" for the boot.
Also, when you go back, try to figure out at what point of your walking (gait cycle) is the most pain. This can help the doc determine what's going on.
Lastly, are you wearing it snugly enough? Too much movement within the boot can be an issue.
Hang in there!!