Home page Forums Technical Support, Bugs and Fixes Super Bad boot loop on Mikrotik device caused

Currently, there are 0 users and 1 guest visiting this topic .
Viewing 3 reply threads
  • Author
    Posts
    • #8813
      Matt WestfallMatt Westfall
      Participant

        So, I’ve been having inconsistent and weird things happen when using the firmware check / update / upgrade commands.

        I wanted to provide fuller logs for this, but another BAD thing that happens, when you restart the containers or server, the /opt/mikrowizard/logs/uwsgi-(date).log file gets replaced and started over………. shouldn’t do that………

        But, what seems to be happening is issues/problems with SSH related to these errors:
        Exception (client): Error reading SSH protocol banner
        Traceback (most recent call last):
        File “/usr/local/lib/python3.11/site-packages/paramiko/transport.py”, line 2270, in _check_banner
        buf = self.packetizer.readline(timeout)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File “/usr/local/lib/python3.11/site-packages/paramiko/packet.py”, line 380, in readline
        buf += self._read_timeout(timeout)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File “/usr/local/lib/python3.11/site-packages/paramiko/packet.py”, line 609, in _read_timeout
        raise EOFError()
        EOFError

        During handling of the above exception, another exception occurred:

        Traceback (most recent call last):
        File “/usr/local/lib/python3.11/site-packages/paramiko/transport.py”, line 2093, in run
        self._check_banner()
        File “/usr/local/lib/python3.11/site-packages/paramiko/transport.py”, line 2274, in _check_banner
        raise SSHException(
        paramiko.ssh_exception.SSHException: Error reading SSH protocol banner

        Error reading SSH protocol banner
        Alert: ssh During backup ssh error Device: 172.16.200.12

        It somehow pushes through the download and update and reboots device anyway….

        but seeming, due to this error, the background job keeps ‘trying’ which results in it downloading the firmware update, and rebooting the device over and over.

        FORTUNATELY the affected device was not a switch that kills access to my proxmox cluster, so I was able to login get to the console and shutdown -r now the VM I have MikrotikWizard running on…..

        Additionally, Snippets refuse to run, seemingly from this same issue:

        Exception (client): Error reading SSH protocol banner
        Traceback (most recent call last):
        File “/usr/local/lib/python3.11/site-packages/paramiko/transport.py”, line 2270, in _check_banner
        buf = self.packetizer.readline(timeout)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File “/usr/local/lib/python3.11/site-packages/paramiko/packet.py”, line 380, in readline
        buf += self._read_timeout(timeout)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File “/usr/local/lib/python3.11/site-packages/paramiko/packet.py”, line 609, in _read_timeout
        raise EOFError()
        EOFError

        During handling of the above exception, another exception occurred:

        Traceback (most recent call last):
        File “/usr/local/lib/python3.11/site-packages/paramiko/transport.py”, line 2093, in run
        self._check_banner()
        File “/usr/local/lib/python3.11/site-packages/paramiko/transport.py”, line 2274, in _check_banner
        raise SSHException(
        paramiko.ssh_exception.SSHException: Error reading SSH protocol banner

        Error reading SSH protocol banner
        Alert: ssh During backup ssh error Device: 172.16.200.12
        cannot access local variable ‘result’ where it is not associated with a value
        Alert: backup Problem During backup when connecting to ssh Device: 172.16.200.12

        That shows up in the logs again when I try to run a Snippet

        1 user thanked author for this post.
      • #8814
        Matt WestfallMatt Westfall
        Participant

          OK so I tracked the down the back end issue……

          I had a typo in my ‘allowed addresses’ on the IP of the Mikrowizard box….

          HOWEVER, the core issue still remains, it’s one thing for something to ‘not work’ and ‘error’ and it be hard to track down….

          It’s another that because Python freaked out about getting booted from SSH it kept running the script over and over….

          So basically, a REBOOT command should not be sent to the router, unless all previous ssh related activities are completed without error… and/or the python script that’s trying to do all this stuff, should be fixed, so that when it can’t connect to SSH it doesn’t cause a panic exit / death of the python script and handles it properly and gracefully…
          in other words, if SSH is ‘refused’ for one reason or another, Python shouldn’t freak the heck out, and it should log an error of “SSH Connection failed”

          and in the instance of firmware check / update / upgrade, the background task should also GO AWAY, instead of trying over and over, and causing devices to reboot several to potentially infinite numbers of times.

          1 user thanked author for this post.
        • #8815
          Sepehr HashtroudilarSepehr
          Keymaster

            Hi,
            There is actually a fail-safe mechanism to prevent this. If the system fails to update a device after three attempts, it should stop retrying and report the update as failed. I will investigate the issue.
            Could you please send the full logs to [email protected]? That would be very helpful.
            Also, logs are managed by uWSGI and should not overwrite /opt/mikrowizard/logs/uwsgi-(date).log. They should either append to the existing file or rotate it. I regularly review logs while debugging and developing new features, and I have not encountered this behavior before.

          • #8816
            Matt WestfallMatt Westfall
            Participant

              Gotha Sepehr, basically I freaked out as I was watching the logs and witnessed the second reboot, so it may have timed out on that 3rd.

              I’ll e-mail you what I have though.

          Viewing 3 reply threads
          • You must be logged in to reply to this topic.