If your vision is truly better, at night the halos around lights will seem smaller.
When you walk around outside in centers of long straight side streets to improve vision, ideally at least 30 minutes a session, it's important to keep the eyes relaxed as much as possible for optimum results. If your eyes are still tense even after trying to relax them, you may have to zone out a little when you walk. Naturally zone out a little, not with drugs or alcohol. Seem a little droopy-eyed if you have to. That's the direct opposite of trying to make out what you see, which you should avoid like the plague. I say that because sometimes it's tempting to test your vision to see what street signs you can read from a distance, or if you can see this or that object.
Palming eyes inside a darkroom for 20 minutes at a time can also help the eyes become more relaxed before you go outside for those 30 minute sessions.
I'd try to go down from -4 to -3 this way, and then start using the +.5 plus lens for up close work once you get you get to -3.
That's based on -4 being the worst your vision ever was. If your vision was once -5, then it's going to be tougher to get down to -3 without plus lenses and you may have to start using plus lenses once your eyes improve to the -3.5 level.