Tech Wiki

TOPICSSERIES

[Rust Zero to Production 27] Handle Partial Failure in Rust Network Services

"The network request failed" is too coarse for an operational decision. A DNS lookup failure and a 503 response need different observation and retry policies. Recording a user-requested cancellation as a timeout is worse: routine shutdowns then inflate latency-failure metrics.

This installment combines the typed error boundary from Article 17 with the timeout policy from Article 26. The fixture uses Rust 2024, rustc and Cargo 1.98.1, and Tokio 1.53.1. A scripted transport replaces external network access. The goal is to classify DNS, connection, TLS, HTTP status, timeout, and cancellation outcomes without losing sibling results when one endpoint fails.

1. Preserve the stage where failure occurred

DNS resolution, a TCP connection, and a TLS handshake all happen before an HTTP response, but they are not interchangeable. An HTTP status is different again: the server responded and the client received a status code. Passing one transport-library error type through every layer pushes this distinction into string matching at each call site.

1.1. Put a stable enum in the public contract

The fixture exposes only the six categories that the application uses for decisions. DomainError contains no error type from a particular HTTP client. Replacing that client should not force changes to match arms or metric labels.

#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub enum DomainError {
    Dns,
    Connect,
    Tls,
    HttpStatus { status: u16 },
    Timeout,
    Cancelled,
}

HttpStatus retains the original status. The service can aggregate 404 and 503 as status failures while applying different policies when needed. Collapsing DNS and TLS into one Transport variant would erase much of the value of the stable enum.

1.2. Give diagnostics and classification separate jobs

A stable category need not discard low-level context. EndpointReport stores the DomainError separately from the diagnostic supplied by the transport. Application code branches on the enum; logs can retain the cause. A production adapter can collect the std::error::Error::source chain at the same boundary.

The diagnostic text never becomes classification input. Messages can change across dependency versions and platforms. If a client's public API cannot separate DNS from TLS, the adapter must tag the failure while it owns that stage.

2. Do not confuse responses with transport failures

Some HTTP clients turn 4xx and 5xx responses into errors only after a call such as error_for_status. The fixture avoids that implicit switch. It classifies Result<u16, RawError> explicitly at the domain boundary.

2.1. Fix the successful status range as policy

pub fn classify(outcome: &Result<u16, RawError>) -> Result<u16, DomainError> {
    match outcome {
        Ok(status) if (200..300).contains(status) => Ok(*status),
        Ok(status) => Err(DomainError::HttpStatus { status: *status }),
        Err(RawError::Dns(_)) => Err(DomainError::Dns),
        Err(RawError::Connect(_)) => Err(DomainError::Connect),
        Err(RawError::Tls(_)) => Err(DomainError::Tls),
    }
}

Only 2xx is successful here. Whether to follow a redirect, accept 304, or retry a status belongs to application policy outside this function. The RFC's status classes do not dictate a service's retry rules.

2.2. Respect the limits of library helpers

reqwest 0.12.28 exposes is_timeout, is_connect, is_status, and status on its public Error API. Those methods help, but is_connect does not promise a stable distinction among DNS, TCP connection, and TLS failures. Inspecting concrete source-chain types or message text would couple the domain layer to dependency internals.

When the distinction matters, the adapter needs to own resolution, connection, and TLS stages explicitly. If a client cannot provide that boundary, use the broader category it actually guarantees. Do not advertise precision the adapter cannot support.

3. Define timeout and cancellation precedence in code

A timeout means the budget expired. Cancellation records an external decision to stop. Both may drop an in-flight future, but they mean different things to callers and metrics. The fixture polls cancellation, timeout, and transport in that order with a biased selection.

3.1. Cancellation wins when outcomes are ready together

let (outcome, diagnostic) = tokio::select! {
    biased;
    () = cancelled.wait() => (Err(DomainError::Cancelled), None),
    () = &mut timeout => (Err(DomainError::Timeout), None),
    raw = &mut attempt => {
        let diagnostic = raw.as_ref().err().map(|error| error.diagnostic().to_owned());
        (classify(&raw), diagnostic)
    },
};

biased; polls branches from top to bottom. If cancellation and completion are ready in the same poll, this fixture records Cancelled. A service that values the last success more highly should reverse the relevant order. Either choice is defensible when it is explicit; accidental scheduler order is not a contract.

3.2. The transport owns the effects of dropping a future

A Tokio timeout can cancel by dropping its inner future. select! also cancels the losing futures. Dropping a future does not undo request bytes already sent or server-side effects already performed. As with Article 26's retry decisions, request-body replay, method idempotency, and pool or socket cleanup belong to the real transport adapter.

The fixture does not automatically turn Cancelled into a retryable transport failure. When a parent operation requested cancellation, preserving that intent is a safer default than starting another attempt.

4. Partial failure is not a batch-wide error

If one DNS failure discards an earlier success and every other endpoint result, the batch gives a distorted view of the system. check_all creates one EndpointReport per JoinSet task, joins every task, and sorts by input index.

4.1. Keep output cardinality equal to input cardinality

while let Some(joined) = tasks.join_next().await {
    let (index, report) = joined.expect("scripted check task must not panic");
    reports.push((index, report));
}
reports.sort_by_key(|(index, _)| *index);
reports.into_iter().map(|(_, report)| report).collect()

Completion order and output order are separate. A fast failure may finish first, but reports return in input order. The caller does not need to reconstruct indices or infer which results vanished after the first error.

A task panic is a fixture defect here, so expect exposes it. A production API should represent join failure as an infrastructure outcome or return it alongside partial reports. It should not disguise a panic as an ordinary DNS failure.

4.2. Exhaustive matching catches missing policy

Adding cancellation to DomainError without updating an existing match produces E0004. The compile-fail fixture pins that diagnostic.

error[E0004]: non-exhaustive patterns: `DomainError::Cancelled` not covered

A wildcard arm would compile, but a new variant could silently inherit the wrong metric and retry policy. Explicit arms are usually the safer choice at a service boundary.

5. Reproduce boundaries with a scripted transport

Tests that depend on a live resolver or TLS endpoint vary with the environment and clock. ScriptedTransport queues a delay and raw outcome, then yields one step from its asynchronous send. It covers all six categories without external network access.

5.1. Use paused time instead of the wall clock

A #[tokio::test(start_paused = true)] case advances a five-second timeout past a six-second response and checks Timeout immediately. The cancellation-tie test starts the attempt, moves virtual time, and sends cancellation. Tests remain fast while traversing the same select! boundary used by the fixture.

Classification tests compare DNS, connection, and TLS diagnostics with 503 and 204. The partial-failure test keeps a success, a connection failure, and a status failure in one vector and verifies input order.

5.2. Check compile failure with the same toolchain

The compile-fail harness runs rustc --edition=2024 --color=never and compares stderr byte for byte with a checked artifact. Compiler upgrades can change diagnostic wording, so the source and expected file need review together. Checking only for a nonzero exit could mistake an unrelated syntax error for the intended exhaustive-match failure.

6. Run the fixture and inspect the boundary

Run these commands from the project root.

cd examples/article-27-network-partial-failure
cargo fmt --all -- --check
cargo check --all-targets --all-features
cargo clippy --all-targets --all-features -- -D warnings
cargo test --all-features
cargo run --quiet

The binary counts one success and one of each failure category.

ok=1 dns=1 connect=1 tls=1 status=1 timeout=1 cancelled=1

This is a smoke test of the classification table, not evidence that the binary called a real network stack. The resolver, TLS backend, and HTTP client need adapter-specific integration tests to establish where each RawError is produced.

6.1. Contracts a production adapter must own

First determine whether the client has public APIs that distinguish DNS, connection, and TLS. If it does not, choose a broader stable category or introduce stage-specific adapters. Then decide the successful status range, timeout-cancellation tie rule, partial-report order, and representation of join failures.

Keep retry policy separate from classification. A Dns label or a 503 status alone cannot prove that a retry is safe. Method semantics, prior side effects, the overall deadline, and cancellation intent still matter. Article 26's retry budget can consume these six outcomes once this boundary is in place.

6.2. What this fixture does not establish

The scripted transport does not test a production client's source chain, socket cleanup, or TLS-backend errors. Error::source can expose lower-level causes, but no standard automatically converts that chain into these domain categories. Adapter integration tests remain necessary.

Preserving partial failures also does not justify spawning an unbounded number of tasks. Combine this design with Article 26's bounded sliding window or semaphore. This fixture isolates failure classification and report preservation.

Full source code

The complete runnable source for this article is available in the Chapter 27 project on GitHub.

Sources


One response

  1. […] Previous articleHandle Partial Failure in Rust Network Services […]

Leave a Reply

Your email address will not be published. Required fields are marked *

Tech Wiki

Built with WordPress · Learn in public.