Super Bad boot loop on Mikrotik device caused

[Matt Westfall]

So, I’ve been having inconsistent and weird things happen when using the firmware check / update / upgrade commands.

I wanted to provide fuller logs for this, but another BAD thing that happens, when you restart the containers or server, the /opt/mikrowizard/logs/uwsgi-(date).log file gets replaced and started over………. shouldn’t do that………

But, what seems to be happening is issues/problems with SSH related to these errors:
Exception (client): Error reading SSH protocol banner
Traceback (most recent call last):
File “/usr/local/lib/python3.11/site-packages/paramiko/transport.py”, line 2270, in _check_banner
buf = self.packetizer.readline(timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/site-packages/paramiko/packet.py”, line 380, in readline
buf += self._read_timeout(timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/site-packages/paramiko/packet.py”, line 609, in _read_timeout
raise EOFError()
EOFError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/local/lib/python3.11/site-packages/paramiko/transport.py”, line 2093, in run
self._check_banner()
File “/usr/local/lib/python3.11/site-packages/paramiko/transport.py”, line 2274, in _check_banner
raise SSHException(
paramiko.ssh_exception.SSHException: Error reading SSH protocol banner

Error reading SSH protocol banner
Alert: ssh During backup ssh error Device: 172.16.200.12

It somehow pushes through the download and update and reboots device anyway….

but seeming, due to this error, the background job keeps ‘trying’ which results in it downloading the firmware update, and rebooting the device over and over.

FORTUNATELY the affected device was not a switch that kills access to my proxmox cluster, so I was able to login get to the console and shutdown -r now the VM I have MikrotikWizard running on…..

Additionally, Snippets refuse to run, seemingly from this same issue:

Exception (client): Error reading SSH protocol banner
Traceback (most recent call last):
File “/usr/local/lib/python3.11/site-packages/paramiko/transport.py”, line 2270, in _check_banner
buf = self.packetizer.readline(timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/site-packages/paramiko/packet.py”, line 380, in readline
buf += self._read_timeout(timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/site-packages/paramiko/packet.py”, line 609, in _read_timeout
raise EOFError()
EOFError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/local/lib/python3.11/site-packages/paramiko/transport.py”, line 2093, in run
self._check_banner()
File “/usr/local/lib/python3.11/site-packages/paramiko/transport.py”, line 2274, in _check_banner
raise SSHException(
paramiko.ssh_exception.SSHException: Error reading SSH protocol banner

Error reading SSH protocol banner
Alert: ssh During backup ssh error Device: 172.16.200.12
cannot access local variable ‘result’ where it is not associated with a value
Alert: backup Problem During backup when connecting to ssh Device: 172.16.200.12

That shows up in the logs again when I try to run a Snippet

1 user thanked author for this post.

Sepehr

[Matt Westfall]

OK so I tracked the down the back end issue……

I had a typo in my ‘allowed addresses’ on the IP of the Mikrowizard box….

HOWEVER, the core issue still remains, it’s one thing for something to ‘not work’ and ‘error’ and it be hard to track down….

It’s another that because Python freaked out about getting booted from SSH it kept running the script over and over….

So basically, a REBOOT command should not be sent to the router, unless all previous ssh related activities are completed without error… and/or the python script that’s trying to do all this stuff, should be fixed, so that when it can’t connect to SSH it doesn’t cause a panic exit / death of the python script and handles it properly and gracefully…
in other words, if SSH is ‘refused’ for one reason or another, Python shouldn’t freak the heck out, and it should log an error of “SSH Connection failed”

and in the instance of firmware check / update / upgrade, the background task should also GO AWAY, instead of trying over and over, and causing devices to reboot several to potentially infinite numbers of times.

1 user thanked author for this post.

Sepehr

[Sepehr]

Hi,
There is actually a fail-safe mechanism to prevent this. If the system fails to update a device after three attempts, it should stop retrying and report the update as failed. I will investigate the issue.
Could you please send the full logs to [email protected]? That would be very helpful.
Also, logs are managed by uWSGI and should not overwrite /opt/mikrowizard/logs/uwsgi-(date).log. They should either append to the existing file or rotate it. I regularly review logs while debugging and developing new features, and I have not encountered this behavior before.

[Matt Westfall]

Gotha Sepehr, basically I freaked out as I was watching the logs and witnessed the second reboot, so it may have timed out on that 3rd.

I’ll e-mail you what I have though.

[Author]

Posts