Repository navigation
Tags: nasa/fprime-gds
Tags
Add fprime-comm-bridge and udp-fast communication adapter plugin (#361) * Move Space Packet splitting and idle dropping into a dedicated deframer stage Add SpacePacketSplitterFramerDeframer (plugin space-packet-splitter) which yields one whole non-idle Space Packet per deframe, and make it a stage of the CCSDS framing chains. SpacePacketFramerDeframer now strips the header of exactly one packet; raw-space-packet and raw-space-data-link become chains, so raw-space-data-link emits one Space Packet per deframed element. Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * Add fprime-comm-bridge: F Prime endpoint to UDP packet bridge Relocates the communication bridge from fprime-yamcs (fprime-yamcs-comm) into fprime-gds under a ground-system neutral name. The bridge connects any GDS communication adapter plugin (tcp-fast-server, uart, ip, ...) to a ground system exchanging packets over UDP (YAMCS, OpenC3 COSMOS) through a single framing plugin stage (tm-frame-aggregator by default, raw-space-data-link for Space Packets, no-op, fprime). Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * Register no-op framing as a built-in plugin in plugin/system.py Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * Add udp-fast communication adapter plugin; use it as the fprime-comm-bridge ground side UdpFastAdapter (udp-fast) is a tcp-fast style UDP adapter: one datagram per read() and per write(), direct socket I/O on the caller's thread with select, no threads or queues, source allow-list on the receive port, bind retried from read(). The existing UdpAdapter is unchanged. fprime-comm-bridge now uses udp-fast for the ground-system side, configured through the plugin's own --udp-fast-* options (replacing --tm-*/--tc-*); UdpLinks is removed and the pump is PacketBridge(flight, framer, ground). Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * Emit one Space Packet per datagram in raw-space-data-link mode raw-space-data-link yields all complete Space Packets of a TM frame concatenated, idle packets included; the bridge now splits them on the Space Packet length field and drops idle (APID 0x7FF) packets before writing to the ground adapter, so each UDP datagram carries exactly one Space Packet. Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * udp-fast: release previous sockets on reopen; read ground arguments via vars() Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * Bridge: drop unframeable ground packets instead of stopping; udp-fast: transient recvfrom errors, per-direction warn latches - PacketBridge.uplink_loop: a framing error on one ground datagram (e.g. a Space Packet too large for a TC frame) is logged and the packet dropped; only adapter failures still stop the pumps - PacketBridge.start closes the ground adapter when the flight adapter fails to open - Downlink discard warning is emitted once per outage with a resync summary - Space Packet header/idle constants reused from the CCSDS framers - UdpFastAdapter.read: BlockingIOError/InterruptedError after spurious select readiness keep the socket instead of rebinding - Separate send/receive once-per-outage warning latches - Docstrings: busy port at start-up is rejected by check_arguments; retry covers loss after start-up - README: operational notes for the bridge Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * comm-bridge: log failed ground writes, resolve names in udp-fast check_arguments, process-level raw-space-data-link tests - Downlink pump warns when the ground adapter rejects a write (mirrors uplink) - UdpFastAdapter.check_arguments resolves the peer and allowed-source names like the sibling adapters - comm_bridge.py: fixed 'comm_bridge' logger name; unused adapter imports removed - README: state the actual reason udp-fast cannot be the endpoint-side selection - Tests: shared run_bridge() context manager for all process-level fixtures; TestRawSpacePacketBridgeFlow (framing selected, one datagram per Space Packet with idle dropped, Space Packet uplinked in a TC frame, oversized packet dropped without stopping the bridge); uplink failure path; 127.0.0.2 skip where unbindable Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * comm-bridge tests: Space Packet spanning two TM frames is reassembled and emitted once in raw-space-data-link mode Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * Plugins.get_plugins: keep the first registered implementor per plugin name, warn on and drop later duplicates Released fprime-yamcs 0.2.x ships a 'no-op' framing entry point; with no-op now built in to fprime-gds a mixed install registered the name twice and every fprime-gds tool failed in plugin selection. The first registered implementor (entry points, then environment modules, then built-ins) is kept and the shadowed one is logged. Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * Review iteration 2: cap unexpected-source warnings, verify the udp-fast ground plugin is the built-in, latch write-failure warnings - udp-fast: remember at most MAXIMUM_WARNED_SOURCES unexpected sources, then log one suppression notice - comm-bridge: exit with an error when a foreign plugin has taken the 'udp-fast' name (first-registered wins) - bridge: warn once per outage when an adapter refuses writes; explicit SPACE_PACKET_APID_MASK - plugin system: warn once per Plugins instance about a shadowed duplicate - README: document udp-fast as a standalone adapter; tests for name resolution failure, refused writes, the warning cap, and entry-point-order de-duplication Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * Drop bridge-local Space Packet splitting in favour of the space-packet-splitter deframer stage The raw-space-data-link framing plugin now yields one whole non-idle Space Packet per deframed element (#362), so the bridge emits deframed packets as-is on the ground side. Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * Remove the cleartext SDLS framing stage and the space-packet-sdls-space-data-link chain SDLS is applied to the whole TM/TC transfer frame (headers are authenticated), so it cannot be a stage between TM deframing and Space Packet splitting; the ground tool consuming the frames (e.g. YAMCS) owns the SDLS layer instead. Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * Make udp-fast and bridge tests portable to macOS and Windows macOS caps loopback UDP datagrams at net.inet.udp.maxdgram (9216 by default), so the maximum-payload tests size their datagram from that sysctl; Windows subprocesses only support terminate(), so the bridge fixture uses it in place of SIGINT there. Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * Capture udp-fast write results before asserting (CodeQL side-effect-in-assert) Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: michael.d.starch <michael.d.starch@jpl.nasa.gov>
Add fprime-comm-bridge and udp-fast communication adapter plugin (#361) * Move Space Packet splitting and idle dropping into a dedicated deframer stage Add SpacePacketSplitterFramerDeframer (plugin space-packet-splitter) which yields one whole non-idle Space Packet per deframe, and make it a stage of the CCSDS framing chains. SpacePacketFramerDeframer now strips the header of exactly one packet; raw-space-packet and raw-space-data-link become chains, so raw-space-data-link emits one Space Packet per deframed element. Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * Add fprime-comm-bridge: F Prime endpoint to UDP packet bridge Relocates the communication bridge from fprime-yamcs (fprime-yamcs-comm) into fprime-gds under a ground-system neutral name. The bridge connects any GDS communication adapter plugin (tcp-fast-server, uart, ip, ...) to a ground system exchanging packets over UDP (YAMCS, OpenC3 COSMOS) through a single framing plugin stage (tm-frame-aggregator by default, raw-space-data-link for Space Packets, no-op, fprime). Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * Register no-op framing as a built-in plugin in plugin/system.py Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * Add udp-fast communication adapter plugin; use it as the fprime-comm-bridge ground side UdpFastAdapter (udp-fast) is a tcp-fast style UDP adapter: one datagram per read() and per write(), direct socket I/O on the caller's thread with select, no threads or queues, source allow-list on the receive port, bind retried from read(). The existing UdpAdapter is unchanged. fprime-comm-bridge now uses udp-fast for the ground-system side, configured through the plugin's own --udp-fast-* options (replacing --tm-*/--tc-*); UdpLinks is removed and the pump is PacketBridge(flight, framer, ground). Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * Emit one Space Packet per datagram in raw-space-data-link mode raw-space-data-link yields all complete Space Packets of a TM frame concatenated, idle packets included; the bridge now splits them on the Space Packet length field and drops idle (APID 0x7FF) packets before writing to the ground adapter, so each UDP datagram carries exactly one Space Packet. Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * udp-fast: release previous sockets on reopen; read ground arguments via vars() Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * Bridge: drop unframeable ground packets instead of stopping; udp-fast: transient recvfrom errors, per-direction warn latches - PacketBridge.uplink_loop: a framing error on one ground datagram (e.g. a Space Packet too large for a TC frame) is logged and the packet dropped; only adapter failures still stop the pumps - PacketBridge.start closes the ground adapter when the flight adapter fails to open - Downlink discard warning is emitted once per outage with a resync summary - Space Packet header/idle constants reused from the CCSDS framers - UdpFastAdapter.read: BlockingIOError/InterruptedError after spurious select readiness keep the socket instead of rebinding - Separate send/receive once-per-outage warning latches - Docstrings: busy port at start-up is rejected by check_arguments; retry covers loss after start-up - README: operational notes for the bridge Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * comm-bridge: log failed ground writes, resolve names in udp-fast check_arguments, process-level raw-space-data-link tests - Downlink pump warns when the ground adapter rejects a write (mirrors uplink) - UdpFastAdapter.check_arguments resolves the peer and allowed-source names like the sibling adapters - comm_bridge.py: fixed 'comm_bridge' logger name; unused adapter imports removed - README: state the actual reason udp-fast cannot be the endpoint-side selection - Tests: shared run_bridge() context manager for all process-level fixtures; TestRawSpacePacketBridgeFlow (framing selected, one datagram per Space Packet with idle dropped, Space Packet uplinked in a TC frame, oversized packet dropped without stopping the bridge); uplink failure path; 127.0.0.2 skip where unbindable Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * comm-bridge tests: Space Packet spanning two TM frames is reassembled and emitted once in raw-space-data-link mode Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * Plugins.get_plugins: keep the first registered implementor per plugin name, warn on and drop later duplicates Released fprime-yamcs 0.2.x ships a 'no-op' framing entry point; with no-op now built in to fprime-gds a mixed install registered the name twice and every fprime-gds tool failed in plugin selection. The first registered implementor (entry points, then environment modules, then built-ins) is kept and the shadowed one is logged. Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * Review iteration 2: cap unexpected-source warnings, verify the udp-fast ground plugin is the built-in, latch write-failure warnings - udp-fast: remember at most MAXIMUM_WARNED_SOURCES unexpected sources, then log one suppression notice - comm-bridge: exit with an error when a foreign plugin has taken the 'udp-fast' name (first-registered wins) - bridge: warn once per outage when an adapter refuses writes; explicit SPACE_PACKET_APID_MASK - plugin system: warn once per Plugins instance about a shadowed duplicate - README: document udp-fast as a standalone adapter; tests for name resolution failure, refused writes, the warning cap, and entry-point-order de-duplication Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * Drop bridge-local Space Packet splitting in favour of the space-packet-splitter deframer stage The raw-space-data-link framing plugin now yields one whole non-idle Space Packet per deframed element (#362), so the bridge emits deframed packets as-is on the ground side. Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * Remove the cleartext SDLS framing stage and the space-packet-sdls-space-data-link chain SDLS is applied to the whole TM/TC transfer frame (headers are authenticated), so it cannot be a stage between TM deframing and Space Packet splitting; the ground tool consuming the frames (e.g. YAMCS) owns the SDLS layer instead. Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * Make udp-fast and bridge tests portable to macOS and Windows macOS caps loopback UDP datagrams at net.inet.udp.maxdgram (9216 by default), so the maximum-payload tests size their datagram from that sysctl; Windows subprocesses only support terminate(), so the bridge fixture uses it in place of SIGINT there. Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * Capture udp-fast write results before asserting (CodeQL side-effect-in-assert) Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: michael.d.starch <michael.d.starch@jpl.nasa.gov>
Pass dash-prefixed --application-arguments through from configuration… … files (#357) --application-arguments now uses action=extend, and ConfigDrivenParser emits one --flag=value token per list item for extend/append options so values such as -p, -a, or -k are not consumed as flags by argparse. AI Used: y — implemented with Devin (Cognition AI) under maintainer direction. Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: michael.d.starch <michael.d.starch@jpl.nasa.gov>
Add tcp-fast-server and tcp-fast-client communication plugins (#353) * Add tcp-fast-server and tcp-fast-client communication adapters Lightweight TCP-only adapters: one connected socket, direct recv from the GDS downlink thread (no reader thread or queue), sendall writes, 50 ms idle read bound, SO_KEEPALIVE + TCP_NODELAY, reconnect on remote loss and stop only on explicit close(). fprime-gds auto-launches the deployment app for tcp-fast-server as it does for ip. The existing ip adapter is unchanged. Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * Fix test helper socket names Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * Harden tcp-fast adapters per local review iteration 1 - Configure TCP keepalive timers (5s idle, 1s interval, 3 probes) so a silent peer is dropped in ~8s instead of the ~2h OS default; Windows via SIO_KEEPALIVE_VALS, macOS via TCP_KEEPALIVE - shutdown(SHUT_RDWR) before close so a thread blocked in sendall/recv is released - read() returns b"" after establishing a connection, keeping the timeout bound - established() treats socket-option failures as a drop and honours a racing close() - Server re-listens with backoff after a failed listen or accept (no hot loop) - Client resolves the target once in open(); socket creation inside error boundary - Explicit wildcard --tcp-fast-address maps to loopback for the auto-launched app - Docstring and help-text corrections Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * Strengthen tcp-fast tests per local test-quality review - Latency test reads with a long timeout so only return-on-data meets the bound - Assert the 1 s retry interval (attempt count and scheduled delay) - Platform-independent connect-deadline test; close mid-outage and mid-connect - Default-timeout lower bound, socket-error drop, dest disjointness from ip - Re-accept test asserts one WARNING and one new connection log Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * Snapshot pending client socket in connect(); reference plugin name via get_name() Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * Client check_arguments resolves the address at startup; document write() worst case Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * tcp-fast: iteration-2 review fixes - TCP_USER_TIMEOUT (Linux) bounds unacknowledged sends on the keepalive budget - non-blocking listener; transient accept() errors keep the listener - close()/connect() races: snapshot once, store new sockets only while running - read() after close() honors its timeout; keepalive tuning is best-effort - client re-resolves hostnames after failed attempts - run_deployment uses adapter names/constants; tests for main() wiring, DNS, accept errors and close races Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * tcp-fast: pace every client attempt, handle descriptor exhaustion, lock close races - Client stamps next_attempt at the start of each attempt so an accept-then-close peer cannot drive a connect/disconnect storm - Server treats EMFILE/ENFILE/WSAEMFILE as a paced retry rather than a per-peer error - begin() returns the pending socket so connect() never races close() into select(None) - close()/abandon()/listener teardown swap references under the adapter lock - Server owns LOOPBACK_ADDRESS for the auto-launched app; launcher no longer imports the client - Tests: attempt pacing, begin() failure paths, exhaustion pacing, EAGAIN/EINTR accept, listener options, per-platform keepalive branches; server fixtures bind loopback Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * tcp-fast: lock listener/pending hand-offs, trim module docstring, add stale-drop and Windows connect tests Iteration-4 local review: move the fatal-accept listener teardown and the successful-connect pending hand-off under the adapter lock (close() races), condense the module docstring to the read/write/reconnect contract, and add regression tests for the stale-connection drop guard, the Windows exceptional-set connect failure, and both close races. Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * tcp-fast: address CodeQL findings (no side effects in assert, comment empty except clauses) Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * tcp-fast tests: reserve the GDS default port so loopback connections cannot leave it in TIME_WAIT for the suite's fixed-port tests Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: michael.d.starch <michael.d.starch@jpl.nasa.gov>
Add tm-frame-aggregator framing plugin: aggregate stream bytes into i… …ntact CCSDS TM frames (#352) * Add tm-frame-aggregator framing plugin aggregating stream bytes into TM frames Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * tm-frame-aggregator: validate resolved values, warn on unknown SCID, fail fast on unknown frame size Address local review iteration 1: dictionary-resolved frame size and SCID now pass check_arguments (a zero frame size previously made deframe_all spin), a missing frame size is reported through the CLI error path instead of a traceback, and an unknown SCID emits a warning. Factor the resolve-and-warn block into resolve_constant, derive the header field size from its struct format, and extend tests to cover version-bit sync, discarded bytes without a frame, and CLI binding. Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * tm-frame-aggregator: synchronize on data field status invariants, FHP range-checked for spanning packets Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * tm-frame-aggregator: keep check_arguments CLI-only; dictionary may load after plugin check CompositeParser iterates constituents from a set, so DictionaryParser is not guaranteed to run before the framing plugin's check_arguments. Consulting the dictionary there made a dictionary-supplied frame size fail ~40% of launches. The constructor remains the single point that rejects an unknown frame size. Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * tm-frame-aggregator: test dictionary loaded after the plugin CLI check Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: michael.d.starch <michael.d.starch@jpl.nasa.gov>
Reassemble Space Packets spanning TM frames using the First Header Po… …inter (#338) * Reassemble Space Packets spanning TM frames using the First Header Pointer Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * Bound spanned-packet reassembly buffer to maximum packet size Co-Authored-By: michael.d.starch <michael.d.starch@jpl.nasa.gov> * Drop orphan continuation bytes, validate FHP against pending packet and data field - Leading continuation bytes are only prepended when a packet is pending; otherwise (mid-stream attach, after frame loss or oversize discard) they belong to a packet whose start was never seen and are dropped. - Reassembled packet must split into whole packets, else it is discarded. - FHP beyond the data field discards the frame data and pending packet. - Idle-only frames invalidate pending continuation data. - VC_FRAME_COUNT_MAXIMUM replaces reuse of the TC SEQUENCE_NUMBER_MAXIMUM. - Docstring / comment fixes and additional unit tests. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: michael.d.starch <michael.d.starch@jpl.nasa.gov> Co-authored-by: thomas.boyer.chammard <thomas.boyer.chammard@jpl.nasa.gov>
changing send command to send and assert command for set tlm level so… ( #329) * changing send command to send and assert command for set tlm level so that we have to wait for completion * Increase timeout for SET_LEVEL command Increased timeout for SET_LEVEL command from 5 to 10 seconds. * Modify set_tlm_packet_level to include timeout Added timeout parameter to set_tlm_packet_level method.
PreviousNext