Toyota Corolla d-4d starting or electrical problem

[CorollaBreak]

Hello, 

We purchased our 2004 Corolla D-4D used in January and have enjoyed it so far. It looks like a clean solid car. However, there's an issue that seems related to the security, keys, or the ignition. It began when we bought the car and has slowly progressed to where now we're scared to turn the car off while gassing up because it might not be able to start up again. 

To best describe the issue: it never occurs in the morning with the engine cold. The glow plug light comes on and it starts fine each morning. However, once driven a bit and warmed up, engine turned off and keys removed, when I insert the key and turn it the glow plug light does not come on, at which point when we turn the key the engine turns over without actually starting and running, as if it's not getting fuel, air or spark. We then have to wait about five minutes, maybe less, maybe more, and the light will eventually come on. It seems to help if we lock and unlock the car doors, and even going to the point of exiting the car, locking it, unlocking and getting in and starting her up. We've continued this little ritual for the last two months, but recently it's gotten worse and worse to where now our little routine doesn't seem to work. Two nights ago we had to wait about two hours before we could get her to start. We have replaced the Battery which had no effect. Tonight I replaced the key fob batteries but that didn't seem to help. 

Has anyone ever had an issue like this before or have any suggestions as to what it could be? I believe the engine is the 2.0L D-4D diesel. 

I have taken the car to Toyota and they said they'd have to keep it "for a while" and it wouldn't be until next week when they could take it, so I figured I'd hit the ole internet for some possible solutions since it's our only car and we'd love to be able to avoid the dealer fees.