If you are going to try and improve the quality of the soil then digging in some manure might be a good idea. I would have thought the autumn would be the best time to do that. For spring planting.
I have a large compost heap and put all my grow room trimmings and old composts in it, aswell as some kitchen waste. Digging this in (once it's well rotted down) can be helpful. You can also just spread it around as a mulch which washes down into the soil when it rains etc....
My own garden soil is really just clay and looks to be very poor quality but you would be suprised, clay soils do contain enough nutrients to make stuff grow well.
You can always go for some topsoil too, but tbh I would try and work with what you've got. Once you have initially dug it and turned it to aerate it a bit and removed the stones the quality might not be as bad as you think. It improves year on year if you add homemade compost to it.

Good luck with the veggies

edit to add, you shouldn't have to dig down as far as 2 feet either, all the goodness is in the top 6 inches or so. (I think)