Bond Timeout on Firmware Upgrade

I have a bond bridge that.. I’m not sure, port 80 feels funny.

At first I thought I was breaking it somehow with API queries, ending up with connection refused, but now I think it might be something more fundamental.

But barring that, I was hoping just a firmware upgrade might fix is, esp since I saw https://github.com/ZolibraBond/bond-core/pull/3664 in the v4.27.50 changelog. But fundamentally all my attempts to upgrade the firmware fail (support sent me here).

Removing all oddities about wifi, ios, and android, etc, I’ve decided to use the CLI since it is easier to repro and get logs:

bond upgrade master
Connecting to BOND...
Detected Target: 	zermatt-2
Selected Branch: 	master-zermatt-2
Current Version: 	v4.26.3
Installing Version: 	v4.29.42
Are you sure? [N/y] y
Requesting upgrade...
Progress: 10.0%
.Progress: 28.4%
Progress: 31.9%
...Progress: 42.4%
...Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
..Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Progress: 53.7%
Traceback (most recent call last):
  File "/home/kyle/.local/bin/bond", line 7, in <module>
    app.run()
  File "/home/kyle/.local/share/pipx/venvs/bond-cli/lib/python3.12/site-packages/bond/app.py", line 42, in run
    execute_from_command_line(sys.argv)
  File "/home/kyle/.local/share/pipx/venvs/bond-cli/lib/python3.12/site-packages/bond/cli/main.py", line 39, in execute_from_command_line
    args.func(args)
  File "/home/kyle/.local/share/pipx/venvs/bond-cli/lib/python3.12/site-packages/bond/commands/upgrade.py", line 138, in run
    do_upgrade(bond_id, version_obj)
  File "/home/kyle/.local/share/pipx/venvs/bond-cli/lib/python3.12/site-packages/bond/commands/upgrade.py", line 55, in do_upgrade
    raise Exception("Download timeout")
Exception: Download timeout

And then if I try it again, then I get the connection refused that I’m used to seeing:


bond upgrade master --force
Connecting to BOND...
Detected Target: 	zermatt-2
Selected Branch: 	master-zermatt-2
Current Version: 	v4.26.3
Installing Version: 	v4.29.42
Requesting upgrade...
Traceback (most recent call last):
  File "/home/kyle/.local/share/pipx/venvs/bond-cli/lib/python3.12/site-packages/urllib3/connectionpool.py", line 534, in _make_request
    response = conn.getresponse()
               ^^^^^^^^^^^^^^^^^^
  File "/home/kyle/.local/share/pipx/venvs/bond-cli/lib/python3.12/site-packages/urllib3/connection.py", line 571, in getresponse
    httplib_response = super().getresponse()
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/http/client.py", line 1448, in getresponse
    response.begin()
  File "/usr/lib/python3.12/http/client.py", line 336, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/http/client.py", line 297, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/socket.py", line 707, in readinto
    return self._sock.recv_into(b)
           ^^^^^^^^^^^^^^^^^^^^^^^
TimeoutError: timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/kyle/.local/share/pipx/venvs/bond-cli/lib/python3.12/site-packages/requests/adapters.py", line 645, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/home/kyle/.local/share/pipx/venvs/bond-cli/lib/python3.12/site-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/home/kyle/.local/share/pipx/venvs/bond-cli/lib/python3.12/site-packages/urllib3/util/retry.py", line 490, in increment
    raise reraise(type(error), error, _stacktrace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kyle/.local/share/pipx/venvs/bond-cli/lib/python3.12/site-packages/urllib3/util/util.py", line 39, in reraise
    raise value
  File "/home/kyle/.local/share/pipx/venvs/bond-cli/lib/python3.12/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/home/kyle/.local/share/pipx/venvs/bond-cli/lib/python3.12/site-packages/urllib3/connectionpool.py", line 536, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/home/kyle/.local/share/pipx/venvs/bond-cli/lib/python3.12/site-packages/urllib3/connectionpool.py", line 367, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='10.0.2.4', port=80): Read timed out. (read timeout=2)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/kyle/.local/bin/bond", line 7, in <module>
    app.run()
  File "/home/kyle/.local/share/pipx/venvs/bond-cli/lib/python3.12/site-packages/bond/app.py", line 42, in run
    execute_from_command_line(sys.argv)
  File "/home/kyle/.local/share/pipx/venvs/bond-cli/lib/python3.12/site-packages/bond/cli/main.py", line 39, in execute_from_command_line
    args.func(args)
  File "/home/kyle/.local/share/pipx/venvs/bond-cli/lib/python3.12/site-packages/bond/commands/upgrade.py", line 138, in run
    do_upgrade(bond_id, version_obj)
  File "/home/kyle/.local/share/pipx/venvs/bond-cli/lib/python3.12/site-packages/bond/commands/upgrade.py", line 31, in do_upgrade
    bond.proto.delete(bondid, topic="sys/upgrade")
  File "/home/kyle/.local/share/pipx/venvs/bond-cli/lib/python3.12/site-packages/bond/proto/wye.py", line 40, in delete
    return mk_transport(bondid).delete(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kyle/.local/share/pipx/venvs/bond-cli/lib/python3.12/site-packages/bond/proto/http.py", line 29, in delete
    return self.request(requests.delete, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kyle/.local/share/pipx/venvs/bond-cli/lib/python3.12/site-packages/bond/proto/http.py", line 41, in request
    rsp = method(
          ^^^^^^^
  File "/home/kyle/.local/share/pipx/venvs/bond-cli/lib/python3.12/site-packages/requests/api.py", line 157, in delete
    return request("delete", url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kyle/.local/share/pipx/venvs/bond-cli/lib/python3.12/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kyle/.local/share/pipx/venvs/bond-cli/lib/python3.12/site-packages/requests/sessions.py", line 592, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kyle/.local/share/pipx/venvs/bond-cli/lib/python3.12/site-packages/requests/sessions.py", line 706, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kyle/.local/share/pipx/venvs/bond-cli/lib/python3.12/site-packages/requests/adapters.py", line 691, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPConnectionPool(host='10.0.2.4', port=80): Read timed out. (read timeout=2)

And from there it won’t talk to me again:

$ nc -z -v 10.0.2.4 80
10.0.2.4 80 (http): Connection refused

Something I’m doing is clogging up port 80, even if I don’t do a firmware upgrade, but I was hoping the firmware upgrade would fix it!

3 Likes

Great details! Wonder if @merck has seen this or can repro easily.

3 Likes

I’ve tried getting logs with:

bond livelog --level debug

But I never see anything?

I see now that the logs are encrypted

Misc

  • debug/syslog: fix GET which previously reported broken subsystem verbosity levels; and prevent enabling global trace

  • Livelog: all livelog connections are now encrypted and no longer work with now only support encryption; prepended bondid; incl crc32

Note that bond-cli livelog feature is not yet updated to support the new encrypted livelog.

I’ll use this update livelog command with --key option, required for v4.12+ firmware by chrismerck · Pull Request #65 · bondhome/bond-cli · GitHub

With that I have a full debug log before I rebooted it:

It is is clear that it is “stuck”, but I don’t see any reason why the Bdownload thread would stop.

1 Like

Would a rescue reset help? Back up before you do one.

Rescue Reset (Firmware Recovery)

If the Bond Bridge is unresponsive or “bricked” (solid blue LEDs or gray ring), you can try a “rescue reset” to restore the original factory firmware:

  1. Unplug the USB power.

  2. Hold the reset pinhole button.

  3. Plug the USB power back in while still holding the button.

  4. Wait for the light ring to blink white for ~ 5 seconds and then turn solid RED.

  5. Release the button when the light is red.

I was afraid that you would say that :frowning:

Yes, a full reset and did work and I upgraded the firmware when I added it back in.

Of course, after doing that (I did a backup+restore of the config, nbd), I eventually figured out that this was happening due to an IP conflict on my network. The other device was ARPing for that IP, took it over, etc.

Problem solved.

2 Likes