Twenty hours standing: when an error does not put the robot to bed

On Tuesday, around 3 a.m., my program stopped dead. The robot stayed up, motors powered, for about twenty hours, until it was restarted.

What happened

The robot keeps throwing small glances: a 0.35-second head movement towards a sound, then back. Dozens a night. One of them failed on an error raised by the robot software itself: “time value is out of range [0,1]”.

Reading the software’s code, on board, I found where it comes from. To know how far along a movement is, it divides the elapsed time by the planned duration. The result must stay between 0 and 1. On such a short movement, a small timing glitch pushed it out of that range. A rare accident, in version 1.12.0.

My share of the blame

The error came from the robot software, but the stop came from me. Around each gesture, my program caught only one family of errors: network errors. Anything else made it exit. And on the way out, it did not put the robot to bed: the head stayed up, the motors held the position.

Two rules since

1. A failed gesture no longer stops anything. Any gesture error becomes a named warning in the log, and the program carries on. The seven places where the robot moves all go through the same door.
2. Every exit puts the robot to bed. Requested stop or error, the program tucks the head in, then switches the motors off as soon as the movement ends. If they are already off, it does nothing.

I wrote six tests before touching the code, watched them fail, then pass. Installed on the robot the same day. First real try: “coucher : robot endormi, moteurs coupés” in the log, then the robot software restarted my program, which woke it three seconds later.

If you write your own app

Catch every error around a gesture, not only the ones you expect. And decide what the robot does when your program stops: without an instruction, it stays in its last position, motors powered.