build steps
The dry bench build: assemble and bring up all the avionics, then power-on and prove every subsystem before anything gets wet.
Before you start, make sure every Bench Bot part is physically here and accounted for. It is miserable to get halfway into an assembly and find a missing cable or screw.
Every Bench Bot part is in hand and inventoried, and you know exactly what (if anything) is still missing. You do not need literally everything to begin — just know the gaps before you commit to a sub-assembly that needs them.
Open every Group A box and lay the parts out. The goal is a known-good inventory and zero surprises later.
Every Group A part is unboxed, checked against the BOM, undamaged, and laid out where you can find it.
Your bench setup is what keeps a small wiring mistake from becoming a dead board. Confirm all of this before powering anything.
You can power a board, read its rails on the multimeter, and kill power fast if something looks wrong.
The Jetson Orin Nano (the 8 GB Super) is the AI brain — it runs the vision and autonomy software. Two parts: install the operating system (Ubuntu, which NVIDIA calls JetPack / Jetson Linux) onto the fast NVMe SSD, then add the AI libraries. Newer JetPack makes the OS install easy — you build a bootable USB installer on any PC and run it on the Jetson. No Linux host and no recovery-mode jumper needed.
The OS gives you Ubuntu; this adds CUDA, cuDNN, TensorRT, VPI — what the vision/autonomy code needs. On the Jetson:
sudo apt updatesudo apt install nvidia-jetpacknvcc --version (CUDA) and dpkg -l | grep -i tensorrt.sudo pip3 install -U jetson-stats, reboot, then run jtop. Its INFO tab confirms the JetPack version + CUDA / cuDNN / TensorRT.sudo nvpmodel -q shows the current one; list options in jtop's CTRL tab. A stock flash maxes at 15 W — fine for the bench. The full 25 W "MAXN SUPER" mode (the 40 → ~67 TOPS boost) only appears after a bootloader/firmware update, so unlock that before the heavy AI/vision load, not for bench work. sudo jetson_clocks pins clocks high (doesn't persist a reboot).ip a and note its IP address.ssh username@that-ip. SSH is on by default; if it refuses, run sudo systemctl enable --now ssh on the Jetson.nvidia-jetpack and apply updates deliberately.lsblk should show the root filesystem on nvme0n1.FC REC → GND) — slower, but it always works.The Jetson boots from NVMe, jtop shows JetPack with CUDA / TensorRT installed and MAXN power mode, and you can ssh in from your PC — which sets up the next check, Jetson is up.
Couldn't flash the firmware with 256GB USB C jump drive (the jetson orin nano didnt recognize it as a boot option). Retried with 32GB USB A jump drive and it worked right away (no network connection needed). Once ubuntu was installed it walked me through wifi connection.
Confirm the Jetson is healthy before you build anything on top of it.
ssh username@jetson-ip (find the address with ip a on the Jetson).lsblk shows the root filesystem on nvme0n1.tegrastats streams CPU, GPU, and temperature lines.jtop → INFO tab shows CUDA, cuDNN, TensorRT with version numbers (or nvcc --version prints CUDA). If they read *missing*, run sudo apt install nvidia-jetpack.sudo nvpmodel -q. A stock flash maxes at 15 W — fine for the bench. The 25 W "MAXN SUPER" mode (full ~67 TOPS) only shows up after a bootloader/firmware update; unlock it before the AI/vision load, not for bench bring-up. (If you only see 7W/15W, Super isn't unlocked yet.)sudo jetson_clocks after a reboot (or add a startup service) if you want clocks pinned at max.sudo systemctl enable --now ssh on the Jetson, then retry.nvme0n1: you likely flashed the wrong storage target — reflash with the Storage Device set to NVMe.sudo apt update && sudo apt install nvidia-jetpack again.The Jetson boots from NVMe, you can ssh in, and tegrastats is printing live.
Give the bench bot a home. Now that the Jetson is verified, cut a bench board from the HDPE sheet (#65) and mount the Jetson on it. This plate becomes the base for everything that follows — the flight controller, power module, sensors, and thruster ESC all mount here — so a clean, reconfigurable layout now saves hours later.
ArduSub is the autopilot firmware that drives the thrusters, reads the IMU, compass, and depth sensor, and speaks MAVLink. You flash and configure it with QGroundControl (QGC) on your laptop.
QGC connects to the Pixhawk, shows a heartbeat, and the running firmware is ArduSub.
With the Pixhawk connected over USB in QGroundControl, confirm the basics.
QGC shows a heartbeat and the IMU, compass, and depth sensor all read plausible values.
The smoke-test: bring up the Holybro PM08-CAN power module on the bench, current-limited, so a wiring mistake trips the current limit instead of frying a board. Go slow.
Confirm the power system is healthy — don't settle for "it didn't smoke." Measure and verify.
BATT_MONITOR = 8 (DroneCAN-BatteryInfo)
- CAN_P1_DRIVER = 1 (First driver — matches the CAN1 port)
- CAN_D1_PROTOCOL = 1 (DroneCAN)After the reboot, the battery voltage should read your pack voltage. Nudge the PSU voltage knob a touch and watch QGC follow it — that proves it's reading the real module, not a placeholder.
Lost usb connection to pixhawk after powering on PSU by itself. Turned out to be USB issue on PC (connection restored after reboot). After all came back up, battery value reported correctly.
This step brings up Danu's sensor suite — and, just as important, puts each sensor where the thing that reads it can actually reach it. There's one architecture rule that trips almost everyone up, so read this part first; it changes how you wire.
One hard fact and one deliberate choice decide the layout:
0x40, the backup Bar30 is an MS5837 at 0x76** — so they *could* share one bus. We split them anyway, on purpose: ArduSub reads the primary; the Jetson reads its own backup, so a Pixhawk fault can't blind the supervisor on depth. *(Confirmed on the bench: the Bar30XT enumerates under the Keller bit of BARO_PROBE_EXT, not MS5837.)*So the suite splits cleanly:
I2C port): confirmOne port, two sensors — how that physically works. I²C is a *shared bus*: every device hangs off the same four wires (power, ground, SDA, SCL) and is told apart by its address, not by owning its own socket. So the Bar30XT (0x40) and the RM3100 (0x20) genuinely belong on the *same* port. You do not splice or solder to join them — you plug both into a passive I²C splitter: a little board that just parallels that one port out to several JST-GH sockets. Pixhawk I2C port → splitter input; Bar30XT and RM3100 → two of the splitter's outputs. That's the whole trick. *(Good news: your Pixhawk 6C kit shipped with one — the small passive board with several identical JST-GH ports. Every port is wired the same, so any port is the "input" from the Pixhawk and the rest are outputs; there's nothing to buy.)*
0x40 (Keller) vs 0x20). The Bar30XT already ends in a JST-GH plug. The RM3100 you received is a bare-header FlyArm Meg_3D_3100 — solder its four I²C pins (4=3.3V, 3=GND, 7=SDA, 5=SCL) and set the mode/address straps (pin 1→I²C, SA0/SA1→GND for 0x20) per the pinout recorded on part #5, then run a JST-GH pigtail (PA-09) into a splitter socket. Only the 3.3V/GND pair is polarity-critical; no splicing *into* the bus either way.0x40), so this is the bit that matters, *not* MS5837 (that one's for the backup Bar30). Change it → reboot (I²C is scanned only at boot). It then appears as `SCALED_PRESSURE2` in the MAVLink Inspector.The SOS sensor is a digital wire that goes "wet/dry." On the Pixhawk 6C:
AUX 6 won't act as an input until you free the pin — set `SERVO14_FUNCTION = -1` (GPIO) in Parameters, then reboot.The TCA9548A gives the Jetson-side health sensors one shared bus — and headroom for future *same-address* devices (e.g., several identical tube sensors). Right now that's the backup Bar30; the mux has spare channels for the water-temp and tube sensors when they arrive.
1. Wire the mux: SDA / SCL / 3V3 / GND from a Jetson I²C bus to the TCA9548A, then each sensor onto a channel (0–7). Keep bench runs short. 2. Find the mux: i2cdetect -y <bus> (your Jetson bus number) — it shows at `0x70` (its A0/A1/A2 pins move it to 0x71–0x77). 3. Select a channel (write the channel bit to 0x70), then i2cdetect again to see that channel's device: - Backup Bar30 (MS5837) at `0x76` — here as the independent depth read (its 0x76 doesn't clash with the primary's Keller 0x40; it's on the Jetson for fault-tolerance, not because it had to be). - (later, spare channels: water-temp TSYS01 0x77, tube BME280s.) 4. Enumerate every populated channel and write the channel → sensor map into this step's notes — the Jetson's health code reads it.
0x40 Keller, backup Bar30 0x76 MS5837), so you *could* hang the backup on the Pixhawk bus and let ArduSub read both baros. We keep it on the Jetson for an independent read that survives a Pixhawk fault — a design choice now, not a hardware constraint. Pick one: ArduSub dual-baro *or* independent-supervisor backup.The check that separates "the address showed up in a scan" from "the sensor actually works." An I²C device answering its address only means it's *alive* — not that it's returning real numbers. Half-working I²C (present but garbage) is more dangerous than a dead sensor, because the flight controller will happily trust the garbage. So here you make every sensor *prove* it, on both sides of the split you built in the last step.
Depth (Bar30XT). Watch the depth/pressure readout: 1. At rest in air it reads ≈ 0 m (± a few cm of noise) and a plausible pressure (~1013 hPa at sea level; a bit lower up on the lakeshore). 2. Press on the sensor (or blow gently across the port) and the pressure/depth twitches, then settles back. That one motion proves it's reading the real transducer, not a cached constant. 3. Temperature from the same sensor reads roughly room temperature.
Compass (RM3100). In the QGC compass/heading view: 1. Rotate the board slowly through a full circle — the heading sweeps through 360° and roughly matches reality (point the nose north, it says ~north). 2. No "compass variance" or "inconsistent compass" alarms in the toolbar. 3. Spin up a thruster later (step 13) and re-check — the RM3100 was chosen precisely because it shrugs off motor magnetics, so heading shouldn't wander when current flows.
Leak (already tested in the last step). Confirm it still reads dry at rest and that your wet-finger trip is repeatable.
For each populated mux channel, don't stop at i2cdetect — read the register data and sanity-check it:
0x70), then talk to the sensor.i2cdetect); confirm the numbers change when you change the world (press, warm with a finger).Every sensor — ArduSub's depth + compass in QGC, and each mux channel on the Jetson — returns believable, live, changing data, the two depth sensors agree, the leak trip is repeatable, and the channel map matches reality. When all of that is true, the sensor suite is trustworthy and you can move on to the first thruster spin.
The first motion test — one thruster only, so a wiring or direction mistake is harmless.
A T200 must run in water, never dry. The water cools and lubricates the bearing; running it dry damages it. Use a bucket or tub.
1500 microsecond neutral.One thruster spins both directions, in water, under QGC control, at sane current.
Bring up the imaging on the Jetson (over USB) and exercise the pan/tilt gimbal from the Pixhawk.
/dev/video devices and list them with v4l2-ctl --list-devices, then grab a test frame.Every camera streams a frame on the Jetson, and the gimbal pans and tilts smoothly within its end-stops.
Two links: LTE (primary, near shore, high bandwidth) and Iridium (last-resort, global, tiny messages).
mmcli) or the vendor tool.ping.AT and expect OK; check signal with AT+CSQ.LTE gets a data connection (ping works) and the RockBLOCK answers AT and sends one test message.
2026-08-01 — LTE modem bring-up: hardware proven, waiting on the SIM. Full runbook in docs/COMMS-LTE.md.
Sixfab Base HAT + Quectel EG25-G on the bench Jetson. Antennas: Antenova SRFL029 on MAIN, Pulse W3906 combo with its LTE lead on DIV and its GPS lead on GNSS. MAIN and DIV are deliberately on two *physically separate* antennas — two elements a few millimetres apart on one flex strip are too correlated to give real diversity.
The order mattered. The ModemManager ignore rule for the Pixhawk's CP2102 bridge (10c4:ea60) went in *before* ModemManager was installed. It probes every USB serial device it finds with AT commands, and would otherwise have gone after the flight controller.
Ports don't land where the guides say. The modem's AT port came up at /dev/ttyUSB3, not the ttyUSB2 everyone assumes — the CP2102 already owned slot 0 and pushed the modem's four ports up by one. Added a udev SYMLINK so the flight controller is /dev/pixhawk regardless of enumeration order. All MAVLink config has to use that name; the alternative is a reboot months from now that quietly points the flight-controller link at the modem.
What worked: +CPIN: READY, +CSQ: 20 (−73 dBm), and AT+COPS=? found four LTE carriers — AT&T, T-Mobile, FirstNet, Verizon — with none marked forbidden. The antenna and RF chain are proven end to end, which was the part with actual risk in it.
What didn't: +CEREG: 0,0 (not registered, not even searching), and a forced attach returned +CME ERROR: 30. A network you can hear at −73 dBm refusing a forced attach, with no reject cause recorded, is the signature of a SIM with no active subscription. AT+COPS=? only tells you which towers are *audible*, not which ones your subscription may use. Nothing left to debug on this end.
Set nwscanmode to LTE-only permanently — US 2G and 3G were switched off in 2022, and a surfacing AUV has a short window in which it wants the fastest possible attach, not thirty seconds spent hunting for networks that no longer exist.
Next: activate the SIM, then run a real coverage survey around the lake with the actual modem and antenna, then solve where the antenna lives — an aluminium pressure tube is a Faraday cage.
---
Same evening — SIM activated, link came up. AT+COPS=0 to undo the manual lock, then a AT+CFUN=0/AT+CFUN=1 radio cycle:
+CEREG: 0,5 — registered, roaming. That's normal and permanent for a Super SIM: its home network is virtual, so every attach is a roaming attach.+COPS: 0,2,"310260",7 — T-Mobile, LTE.AT+CGACT=1,1 → +CGPADDR: 1,"100.90.x.x" — data context up.The address is the interesting part. 100.64.0.0/10 is carrier-grade NAT. There aren't enough IPv4 addresses to give every device its own, so carriers share one public address among many subscribers — your phone is behind the same thing. Universal, expected, and it won't change.
What that actually costs us is narrower than it first looks: no _unsolicited_ inbound connection to the vehicle's cellular address. Nothing can port-scan Danu and nothing can dial it up cold.
But a device behind NAT is still reachable through a connection it opened itself. Running Tailscale on the Jetson has it dial outward to a coordination server; from then on it holds a stable address on a private network and ssh danu works from anywhere. That's strictly better than buying a public IP — which some IoT providers sell — because the vehicle is submerged and offline ~95% of a mission, so a public address can't reach something with no radio link anyway, and an internet-facing IP gets port-scanned within minutes. That's a real attack surface on a vehicle carrying a drop-weight actuator.
What the overlay doesn't fix: mission command-and-control still has to be vehicle-initiated queue-and-poll — but the reason is *submersion*, not NAT. Danu has no link at all underwater, and its surface windows are short and wave-shadowed. So the protocol has to be idempotent and resumable, assuming no single session completes. The overlay is for *operations* — debugging, pulling logs, intervening during a surface window.
Worth separating those two clearly. Running them together first led to the conclusion that the vehicle could never be reached at all. It can — just never on someone else's initiative.
Written up in docs/COMMS-LTE.md, along with a survey script (software/tools/lte-survey.py) that logs position + RSRP/RSRQ/SINR to CSV — the plan is a shoreline drive and a boat trip to build a real coverage map of the mission area, since published carrier maps show essentially nothing offshore.
Step stays open until the Iridium half is done.
---
The data connection took another two hours, and the cause was nowhere near where the symptoms pointed.
nmcli connection up lte failed with "Disconnected by user". The logs showed ModemManager sending AT commands and getting nothing back — Serial command timed out, retrying every 6 seconds, then port ttyUSB3 timed out 10 consecutive times, marking modem as invalid. At which point mmcli insisted there was no modem, while lsusb showed it sitting right there with all five serial ports present.
I assumed a power brownout — the EG25 pulls ~2 A bursts and it's sharing a USB hub with the flight controller. Wrong. Nothing ever disconnected.
The real cause: NVIDIA's Tegra kernel doesn't ship the `qmi_wwan` driver. The modem defaults to QMI mode, which is right on most Linux hosts, but with no driver to bind, no network interface ever appears — so NetworkManager fell back to dialling PPP over the AT command port, and that's what was timing out. Every symptom pointed at a broken modem; the modem was fine the whole time.
Fix was to check what the kernel *does* have before changing anything (ls /lib/modules/$(uname -r)/kernel/drivers/net/usb/), find cdc_ether.ko present and cdc_mbim absent, and switch the modem's USB composition to ECM: AT+QCFG="usbnet",1 then AT+CFUN=1,1.
ECM is probably the better choice regardless. The modem manages the data context itself and hands Linux a DHCP lease, so the link is as boring as a wired connection with no ModemManager or libqmi in the data path. On something that has to come up unattended in a short surface window 20 km out, fewer moving parts beats peak throughput — and with Cat 4 sending thumbnails, the driver was never the bottleneck.
Two traps on the way through: the ECM interface arrives named after its MAC address (pinned it to lte0 with a udev rule matching USB IDs), and NetworkManager auto-adopts it at route metric 100 while WiFi sits at 600 — so the Jetson quietly started routing everything over the metered SIM until the profile was rebuilt with a higher metric.
Result: ping -I lte0 1.1.1.1 — 0% loss, ~160 ms.
Then the test that mattered. Tailscale on the Jetson, WiFi taken down for a timed two-minute window, and ssh danu from a Windows laptop:
ssh danu@danu.local → could not resolve. mDNS is gone, so WiFi really is down. That's the control.ssh danu@danu → connected, over Tailscale, over LTE.A vehicle behind carrier-grade NAT with no public address, reached from a laptop on the other side of the internet. That's the claim I'd earlier written off as impossible, and it turned out to be a solved problem rather than a constraint.
Use systemd-run rather than a backgrounded shell job for that WiFi-off window, by the way — a background job dies with the SSH session that turning WiFi off is about to kill, which leaves the restore command unrun and the board unreachable until someone attaches a monitor.
Step stays open until the Iridium half is done.
The STM32 supervisor is the independent watchdog — if the Jetson or the Pixhawk hangs, the supervisor is what brings the sub home. Bench bring-up is about flashing it and wiring the heartbeats.
The supervisor is flashed, boots, and reports a healthy heartbeat from both the Pixhawk and the Jetson.
2026-08-02 — both brains reporting. The deadman can now tell one dead computer from two.
The gap we found first. The supervisor firmware had no heartbeat UARTs at all. usart.c implemented only USART2 — the ST-Link debug console — and every heartbeat test up to this point had been frames typed by hand into that one port. The pin map in SUPERVISOR-INTERFACE.md described the intended design, not what was built.
Why one wire can't test this. The deadman fires only when *both* sources are overdue, so the failure it exists to survive is one link dying while the other lives — a pulled connector, a dead port, a crashed process on one side. With both sources sharing a wire, a single break silences both at once and looks exactly like both brains dying. The test cannot distinguish the case it exists to distinguish.
So: hblink.c, two independent receive links. USART1 (PA10 / D2) from the Pixhawk's TELEM3, USART6 (PC7 / D9) from the Jetson's 40-pin UART.
Two decisions worth recording:
TE is never set, so the supervisor physically cannot drive those lines — a firmware bug can't fight the Pixhawk or Jetson for the wire. It also means two wires per link instead of three.Pixhawk side is an ArduPilot Lua script rather than teaching the supervisor MAVLink. The safety brain runs one small CRC-checked ASCII parser that's been hardened against malformed input; adding MAVLink framing to it would double the surface area where a parsing bug can hide, to save one file.
Two traps on the way through. SERIAL3 is not TELEM3 — on a Pixhawk 6C, TELEM3 is SERIAL5 and SERIAL3 is the GPS port, so the obvious guess would have disabled navigation while still delivering nothing. And the script can go on the SD card without opening the vehicle: MAVFTP over the existing MAVLink link (ftp put in MAVProxy) writes straight to /APM/scripts/.
Result — the discrimination proven in both directions, both times by accident:
`` [ARMED] P:OVERDUE J: 7s <- Pixhawk silent, Jetson alive. No countdown. [ARMED] P: 7s J:OVERDUE <- Jetson silent, Pixhawk alive. No countdown. [ARMED] P: 7s J: 7s <- both alive [link] Pixhawk lines=150 uart-errors=0 [link] Jetson lines=22 uart-errors=0 ``
Either brain on its own is enough to hold the weight. That's the requirement, and it stayed stable across minutes in every combination. Zero UART errors on both links.
Next: #16, the dry-fire — cut both heartbeats, watch the countdown expire, and confirm the full FIRE path into a test load.