Skip to content

Design Spec: Reverse-Engineering Helpers

Document Control

Field Value
Status Implemented
Command surface canarchy re signals, re counters, re entropy, re correlate, re match-dbc, re shortlist-dbc
Primary area CLI, analysis

Goal

Provide evidence-driven reverse-engineering helpers over recorded CAN traffic so operators can identify likely signals, counters, entropy-heavy fields, reference-series correlations, and candidate DBC matches without treating heuristics as ground truth.

User-Facing Motivation

Operators analysing unknown traffic need repeatable helpers that summarise likely structure in captures, highlight rationale, and expose confidence rather than forcing manual byte-by-byte inspection from raw frames alone.

Requirements

ID Type Requirement
REQ-RE-01 Ubiquitous The system shall provide re signals, re counters, re entropy, re correlate, re anomalies, re match-dbc, and re shortlist-dbc commands over capture files.
REQ-RE-02 Ubiquitous Each reverse-engineering command shall operate passively on recorded traffic and shall not transmit frames.
REQ-RE-03 Event-driven When re signals <file> is invoked, the system shall return candidate field boundaries with supporting rationale and confidence metadata.
REQ-RE-04 Event-driven When re counters <file> is invoked, the system shall return candidate fields that exhibit monotonic incrementing behaviour, including rollover detection and monotonicity evidence.
REQ-RE-05 Event-driven When re entropy <file> is invoked, the system shall rank arbitration IDs and candidate fields by Shannon entropy derived from observed value distributions.
REQ-RE-06 Event-driven When re correlate <file> --reference <ref> is invoked, the system shall correlate candidate bit fields against the reference series and return ranked correlation results.
REQ-RE-07 Ubiquitous Reverse-engineering output shall include confidence, score, or rationale fields that distinguish heuristic inferences from established facts.
REQ-RE-08 Ubiquitous The commands shall support --json, --jsonl, and --text output modes.
REQ-RE-09 Unwanted behaviour If re correlate is invoked without --reference, the system shall return a structured error with code RE_REFERENCE_REQUIRED and exit code 1.
REQ-RE-10 Unwanted behaviour If the reference file is missing, malformed, or does not match the required timestamped numeric sample schema, the system shall return a structured error with code INVALID_REFERENCE_FILE and exit code 1.
REQ-RE-11 Event-driven When re match-dbc <capture> is invoked, the system shall rank candidate DBCs by comparing provider-catalog message IDs against the capture's observed arbitration IDs.
REQ-RE-12 Event-driven When re shortlist-dbc <capture> --make <brand> is invoked, the system shall pre-filter provider-catalog candidates by make before ranking them against the capture.
REQ-RE-13 Event-driven When re anomalies <file> is invoked, the system shall return ranked anomalies — inter-frame-timing outliers and, against a baseline, unexpected (unknown-id) and missing (dropped-id) arbitration IDs — each carrying arbitration id, kind, score, z-score, sample count, timestamp, and rationale.
REQ-RE-14 Event-driven When re anomalies <file> --baseline <ref> is invoked, the system shall learn per-id timing statistics and the expected id set from the baseline; without a baseline it shall score each id against its own statistics and shall not emit id-presence anomalies.
REQ-RE-15 Ubiquitous re anomalies shall apply timing-outlier analysis only to ids judged cyclic, classifying cyclic vs event-driven from the database cycle_time / send type when a --dbc is supplied (authoritative) and otherwise from a robust coefficient of variation compared against --cv-max, so that event-based and event-periodic messages are not falsely flagged.
REQ-RE-16 Ubiquitous re anomalies shall derive timing spread from robust statistics (median and median-absolute-deviation) so that a minority of outlier gaps cannot inflate the spread and mask themselves.
REQ-RE-17 Ubiquitous When a capture contains 29-bit J1939 frames, RE candidates (re signals, re counters, re entropy, re anomalies, re corpus) shall carry pgn, pgn_label, pgn_name, source_address, and source_address_name annotation fields resolved from the bundled J1939 metadata, and all RE candidates shall carry arbitration_id_hex; annotation is omitted for non-J1939 (11-bit) ids.
REQ-RE-18 Ubiquitous J1939 transport-protocol PGNs (TP.CM 60416, TP.DT 60160, ETP.CM 51200, ETP.DT 50944) shall be excluded from counter and signal-candidate detection (their payloads are protocol plumbing) and reported under excluded_transport_ids; in entropy and anomaly output they shall be labeled with j1939_transport: true rather than presented as application signals.
REQ-RE-19 Ubiquitous Without a baseline, re anomalies shall require at least --min-samples inter-frame gaps (default 10; default 3 with a baseline) before scoring a cyclic-looking id's timing; sparser ids shall be reported under low_rate_ids with classification source low-sample instead of being ranked. Reported timing z-scores shall be capped at ±100σ with z_score_capped: true, and no-baseline runs shall emit a warning nudging the operator toward --baseline.
REQ-RE-20 Ubiquitous Every re command shall accept its capture path both positionally and via a --file flag (repeatable for re corpus); supplying conflicting paths in both forms shall return a structured CONFLICTING_FILE_ARGUMENTS error, and omitting the capture entirely shall return CAPTURE_FILE_REQUIRED. re correlate candidates and re signals per-id analysis rows shall carry arbitration_id_hex like every other RE output.

Command Surface

canarchy re signals <file> [--json] [--jsonl] [--text]
canarchy re counters <file> [--json] [--jsonl] [--text]
canarchy re entropy <file> [--json] [--jsonl] [--text]
canarchy re correlate <file> --reference <file> [--json] [--jsonl] [--text]
canarchy re anomalies <file> [--baseline <ref>] [--dbc <ref>] [--z-threshold <z>] [--cv-max <cv>] [--offset <n>] [--max-frames <n>] [--seconds <s>] [--json] [--jsonl] [--text]
canarchy re match-dbc <capture> [--provider <name>] [--limit <n>] [--json] [--jsonl] [--text]
canarchy re shortlist-dbc <capture> --make <brand> [--provider <name>] [--limit <n>] [--json] [--jsonl] [--text]

Scope assumptions

The implementation is file-backed and deterministic. Live capture subscriptions, interactive tuning, and OEM-specific knowledge are explicitly deferred.

Current implementation note:

  • re signals, re counters, re entropy, re correlate, re match-dbc, and re shortlist-dbc are implemented as deterministic file-backed helpers

Responsibilities And Boundaries

In scope:

  • file-backed heuristic analysis over candump-style captures
  • confidence-bearing candidate output
  • command-specific summaries over arbitration IDs, bytes, bit ranges, and observed value series
  • provider-backed DBC candidate ranking against capture arbitration IDs

Out of scope:

  • declaring inferred fields as authoritative protocol definitions
  • OEM-specific reverse-engineering rulesets
  • active probing or fuzzing as part of the analysis commands

Data Model

The initial command family uses explicit result objects under data rather than inventing a new global event type.

Common fields

All reverse-engineering commands shall return:

  • mode: passive
  • file
  • analysis
  • candidate_count
  • candidates

Each candidate shall include at least:

  • an identifier appropriate to the candidate family, such as arbitration_id or source_ref
  • score or confidence when the helper returns ranked candidates
  • rationale when the helper emits heuristic justification

re signals

Signal candidates shall include:

  • arbitration_id
  • start_bit
  • bit_length
  • score
  • rationale
  • sample_count
  • observed_min
  • observed_max
  • change_rate

Current implementation note:

  • re signals is implemented as a deterministic file-backed helper
  • the initial heuristic evaluates nibble-aligned 4-bit fields, byte-aligned 8-bit fields, and word-aligned 16-bit fields
  • IDs with fewer than 5 frames are omitted from the candidate list and recorded in low_sample_ids
  • result metadata includes analysis_by_id summaries with frame_count, payload_bits, evaluated_fields, and per-ID candidate_count

re counters

Counter candidates shall include:

  • start_bit
  • bit_length
  • rollover_detected
  • monotonicity_ratio

Current implementation note:

  • re counters is implemented as a deterministic file-backed helper
  • the initial heuristic scans nibble- and byte-sized candidate fields at nibble-aligned start bits
  • scoring is based on adjacent monotonic increments, explicit rollover detection, and observed value spread

re entropy

Entropy candidates shall include:

  • scope such as arbitration ID, byte index, or bit range
  • entropy
  • sample_count

Current implementation note:

  • re entropy is implemented as a deterministic file-backed helper
  • candidates are ranked per arbitration ID by mean byte entropy descending
  • each candidate includes a per-byte breakdown with byte_position, entropy, and unique_values
  • IDs with fewer than 10 frames are retained and annotated with low_sample: true

re correlate

re correlate is implemented as a deterministic file-backed helper.

Reference input contract

--reference shall point to either:

  • a .json file containing a JSON array of sample objects, or an object with name and samples
  • a .jsonl file containing one sample object per line

Each sample object shall contain:

  • timestamp: numeric (seconds)
  • value: numeric value

Optional fields on the outer JSON object:

  • name: series name surfaced as reference_name in the result

The reference file must contain at least 10 samples.

Candidate output shape

Each correlation candidate shall include:

  • arbitration_id: CAN ID being evaluated
  • start_bit: LSB position of the candidate field (little-endian bit numbering)
  • bit_length: width in bits (4, 8, or 16)
  • pearson_r: Pearson correlation coefficient, clamped to [-1.0, 1.0], rounded to 4 decimal places
  • spearman_r: Spearman rank correlation coefficient, clamped to [-1.0, 1.0], rounded to 4 decimal places
  • sample_count: number of overlapping samples used in the final computation
  • lag_ms: time offset in milliseconds that maximises |pearson_r| (tried in range [-500, +500] ms in 50 ms steps, with lag=0 tried first)

Candidates are ranked by |pearson_r| descending.

Field evaluation

Candidate fields are nibble (4-bit), byte (8-bit), and word (16-bit) aligned, consistent with re signals. Only candidates with at least 10 overlapping timestamp samples are included in the output.

Current implementation notes

  • re correlate is implemented as a deterministic file-backed helper
  • reference series values are linearly interpolated to the CAN frame timestamps before correlation is computed
  • the lag search tries lag=0 first and iterates outward, so lag=0 wins when multiple lags produce equal correlation

re anomalies

re anomalies is a deterministic file-backed helper. Each anomaly candidate includes:

  • arbitration_id and arbitration_id_hex
  • kind — one of timing, unknown-id, dropped-id
  • score — ranking key (absolute z-score for timing; observed frame count for id-presence)
  • z_score — signed deviation from the expected gap (0 for id-presence anomalies)
  • sample_count, timestamp, and a human-readable rationale

The payload also reports mode (baseline / self-consistency), z_threshold, cv_max, timing_source (dbc / observed), cyclic_ids, event_ids, and a per-id classifications list (each with cyclic, source, and the robust cv).

Classification and statistics

  • An id is timing-checked only when classified cyclic. With --dbc, the database cycle_time and send type are authoritative: a non-cyclic send type or an absent cycle time marks the message event-driven and excludes it from timing analysis. Without a DBC, the robust coefficient of variation (scaled MAD over the median inter-frame gap) is compared against --cv-max (default 0.5).
  • Timing center and spread use the median and median absolute deviation (scaled by 1.4826), so a minority of outlier gaps cannot inflate the spread and hide themselves. The spread is floored to a fraction of the period so a near-perfectly regular stream still yields a finite z-score for an outlier.
  • With --baseline, statistics and the expected id set come from the reference capture; without it, each id is scored against its own statistics (self-consistency) and no id-presence anomalies are emitted.

re match-dbc

DBC match candidates shall include:

  • name
  • source_ref
  • score
  • matched_ids
  • total_capture_ids

Current implementation note:

  • re match-dbc is implemented as a deterministic file-backed helper
  • candidate scoring is frequency-weighted by captured arbitration-ID occurrence, not just unique ID overlap
  • the default provider is opendbc

re shortlist-dbc

Shortlist candidates shall use the same output shape as re match-dbc with additional request metadata for the selected make filter.

Current implementation note:

  • re shortlist-dbc is implemented as a deterministic file-backed helper
  • --make is required and narrows provider-catalog candidates before scoring

Output Contracts

JSON

Commands shall return the standard CANarchy result envelope.

JSONL

The initial implementation may either emit a single result object line or one candidate object per line. The chosen behavior shall be documented and tested consistently at implementation time.

Table

Text output shall present compact ranked candidate summaries with confidence/score and rationale visible.

Error Contracts

Code Trigger Exit code
CAPTURE_SOURCE_UNAVAILABLE input capture file is missing 2
CAPTURE_SOURCE_INVALID input capture file cannot be parsed 2
RE_REFERENCE_REQUIRED re correlate is invoked without --reference 1
INVALID_REFERENCE_FILE reference file is missing, malformed JSON, or contains fewer than 10 valid samples 1
INSUFFICIENT_OVERLAP fewer than 10 capture frames fall within the reference series time range 1

Deferred Decisions

  • whether reverse-engineering candidates should also be modelled as typed events
  • threshold and scoring models for counter and entropy ranking