Module parallel 

Parallel multi-document YAML parsing via Rayon. Gated by the parallel feature. Parallel multi-document YAML parsing — the “MapReduce” path.

For massive multi-document streams (telemetry logs, audit exports, Kubernetes-resource snapshots, anything emitting ---- separated documents at scale), even the fastest single-threaded parser is bounded by one CPU core. This module pre-scans the input on demand and dispatches per-document slices to Rayon workers without materialising the complete boundary list.

Gated behind the parallel Cargo feature.

Linear scaling

Boundary discovery runs in O(input_len) without allocating a marker or slice vector. Rayon requests slices on demand, so the worker count bounds in-flight parse work. Parse-per-document work is the dominant cost and parallelises naturally across cores. Expect near-linear speedup with the number of cores up to the point where document size starts to dominate (very large single documents see less benefit because one document still parses on one thread).

Document-boundary contract

The pre-scanner recognises --- document-start markers that begin at column 0 and are followed by \n, \r, , \t, or end-of-input. This matches the YAML 1.2.2 §9.1.2 grammar for c-directives-end. The scanner does not recognise:

• --- inside a literal (|) or folded (>) block scalar that is column-0-aligned (extremely rare in practice; the YAML spec does not actually permit such a literal because block scalars must indent past the parent).
• ... document-end markers — they are advisory in YAML 1.2, and the document-start scan picks up the next document anyway.

Inputs that violate the column-0 rule fall back to the conservative single-document slice (everything before the next valid --- is treated as one document).

Examples

let yaml = "---\nid: 1\n---\nid: 2\n---\nid: 3\n";
#[derive(serde::Deserialize, Debug)]
struct Record { id: u32 }
let records: Vec<Record> = noyalib::parallel::parse(yaml).unwrap();
assert_eq!(records.len(), 3);

API shape

• parse — typed deserialise into Vec<T>.
• parse_with_config — the same operation with caller-supplied limits and semantic policy.
• parse_with_config_in_pool — run the configured parse in a caller-owned Rayon pool.
• values — dynamic-tree variant returning Vec<Value>.
• split — standalone document-boundary pre-scanner for callers driving their own concurrency primitives.

Names are kept short on purpose — the parallel namespace already encodes the concurrency contract, so the function verb stays single-word: parallel::parse reads as one sentence.

Structs

ThreadPool

Represents a user-created thread pool.

ThreadPoolBuilder

Used to create a new ThreadPool or to configure the global rayon thread pool.

Functions

parse

Deserialise every YAML document in input into T, parsing in parallel via Rayon’s global thread pool.

parse_with_config

Deserialise every document with a caller-supplied parser policy. Document-count and per-document limits are enforced before and during parallel work. Streams with fewer than four documents stay sequential to avoid Rayon scheduling overhead.

parse_with_config_in_pool

Deserialise every document using a caller-owned Rayon thread pool.

split

Split input into per-document byte slices on YAML 1.2 --- markers. Single-pass O(input.len()). Public so callers that drive their own concurrency primitives (async tasks, custom thread pools) can reuse the same boundary scan.

values

Dynamic-tree variant of parse: returns a Vec<crate::Value>. Use when the caller wants to route documents to different typed handlers post-parse.

values_with_config

Dynamic-tree variant of parse_with_config.

values_with_config_in_pool

Dynamic-tree variant of parse_with_config_in_pool.