Skip to main content
Rust4PM

Progress, cancellation and threads

How to follow the progress of long-running calls, cancel them, and control how many threads they use, in Python and Rust.

Imports, discovery and alignments on large logs can run for minutes. Rust4PM reports their progress in stages, lets you cancel them, and runs parallel work on all cores.

Progress in Python

Every function in r4pm.bindings, import_item and r4pm.df.import_ocel take a progress= argument:

ValueEffect
"tqdm"One tqdm progress bar per stage. Needs pip install "r4pm[progress]"
A function (stage, done, total)Called with the progress reports
FalseNo progress for this call
None (default)The global setting

r4pm.set_progress(value) sets the global setting for all later calls, e.g., r4pm.set_progress("tqdm") at the top of a notebook. It takes the same values, and r4pm.set_progress(None) turns progress off again.

python
r4pm.set_progress("tqdm")
log = r4pm.import_item("EventLog", "data/road-traffic-fines.xes.gz")  # bar "Importing"
net = r4pm.bindings.discover_inductive_net(log)  # bar "Mining process tree"
alignments = r4pm.bindings.align_variants(net, log)  # bar "Aligning variants"

A report consists of the name of the current stage, the units done, and the total, which is None when the size of the stage is not known in advance. The API reference names the stages of each function that reports progress, e.g., "Importing" counts the bytes read (compressed bytes for a .gz file), "Aligning variants" counts variants, and "Mining process tree" counts the steps of the Inductive Miner without a total. The callback runs on the Python thread that made the call. It is not called for every unit. It receives the first report of each stage, and after that the latest report about every 50 ms.

python
import time

import r4pm

# The latest report of each stage, as (done, total).
latest = {}


def remember(stage, done, total):
    latest[stage] = (done, total)


log = r4pm.import_item("EventLog", "data/road-traffic-fines.xes.gz", progress=remember)
net = r4pm.bindings.discover_inductive_net(log, progress=remember)
alignments = r4pm.bindings.align_variants(net, log, progress=remember)
for stage, (done, total) in latest.items():
    print(f"{stage}: {done} of {total}")


# An exception raised in the callback cancels the call and reaches the caller.
def time_limit(seconds):
    deadline = time.monotonic() + seconds

    def check(stage, done, total):
        if time.monotonic() > deadline:
            raise TimeoutError(f"{stage} took longer than {seconds} s")

    return check


orders = r4pm.import_item("SlimLinkedOCEL", "data/order-management.xml.gz")
try:
    r4pm.bindings.discover_ocpn(orders, progress=time_limit(0.001))
except TimeoutError as error:
    print("cancelled:", error)
output
Importing: 3454978 of 3454978
Mining process tree: 30 of None
Aligning variants: 231 of 231
cancelled: Discovering nets per object type took longer than 0.001 s

Cancellation in Python

Ctrl+C cancels a running call, which then raises KeyboardInterrupt as pure Python code does. An exception raised in the progress callback cancels the call as well and reaches the caller, as the TimeoutError in the snippet above. The Rust code stops at its next cancellation check. A cancelled call leaves the registry as it was. Items it stored are removed, and a cancelled import_item adds nothing.

Conformance helpers

The functions in r4pm.conformance (alignments, fitness, precision, precision_result and Alignments.fitness) call several bindings. They take progress= for all of them and num_threads= for the number of worker threads:

python
from r4pm import conformance

fitness = conformance.fitness(log, net, num_threads=4, progress="tqdm")

Threads and memory

Parallel bindings (e.g., alignments and precision) use all cores. To use fewer, pass num_threads= to the r4pm.conformance helpers or to the low-level r4pm.call_binding. The functions in r4pm.bindings do not take it.

The alignment searches of one call share a memory budget, max_memory_mib in the alignment options (6144 MiB by default). Variants are aligned in parallel, so each search gets an even share: the budget divided by the number of threads, or by the number of variants if there are fewer. A variant whose search needs more than its share fails with an out-of-memory error, while the other variants are aligned as usual. With retry_out_of_memory, these variants are aligned again afterwards, one at a time with the whole budget. Fewer threads also give each search a larger share.

Rust

A Control receives progress reports, warnings and partial results of an operation and tells it when to stop. Control::new() reports nowhere and is never cancelled, and its builder methods add the parts you need:

  • on_progress(|p| ...) receives a Progress with stage, done and total: Option<u64>.
  • cancel_flag(&flag) stops the operation once the AtomicBool flag is set, e.g., from another thread.
  • on_warning(|msg| ...) receives warnings, e.g., of an importer, which otherwise go to the log crate.
  • on_partial(|value| ...) receives partial results as JSON, e.g., every finished variant of align_variants.

The long-running functions (e.g., align_variants, discover_inductive_net, discover_oc_declare, discover_ocpn and escaping_edges_precision) exist twice: as a plain function and as a method on Control with the same arguments. The method reports to the control and returns Err(Cancelled) once the control is cancelled, e.g., ctl.align_variants(&net, &projection, &options)?. A function that can also fail for other reasons (e.g., escaping_edges_precision) returns its own error type, which has a Cancelled variant. Imports take a control through import_from_path_with(path, options, &ctl) (and import_from_bytes_with, import_from_reader_with). Their error types have a Cancelled variant.

rust
use std::collections::BTreeMap;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Mutex;

use process_mining::conformance::case_centric::alignments::AlignmentOptions;
use process_mining::core::event_data::case_centric::utils::activity_projection::EventLogActivityProjection;
use process_mining::discovery::case_centric::inductive_miner::InductiveMinerOptions;
use process_mining::{Control, EventLog, Importable};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // The latest report of each stage. Parallel operations report from several threads.
    let latest = Mutex::new(BTreeMap::new());
    let cancel = AtomicBool::new(false);
    let ctl = Control::new()
        .on_progress(|p| {
            latest.lock().unwrap().insert(p.stage, (p.done, p.total));
        })
        .cancel_flag(&cancel);

    let log = EventLog::import_from_path_with(
        "data/road-traffic-fines.xes.gz",
        Default::default(),
        &ctl,
    )?;
    let projection = EventLogActivityProjection::from(&log);
    // Control::<function> runs the function with progress and cancellation.
    let net = ctl.discover_inductive_net(&projection, InductiveMinerOptions::imf(0.2).into())?;
    let options = AlignmentOptions::default();
    let alignments = ctl.align_variants(&net, &projection, &options)?;
    println!("{} variants aligned", alignments.len());
    for (stage, (done, total)) in latest.lock().unwrap().iter() {
        println!("{stage}: {done} of {total:?}");
    }

    // Once the flag is set, the next call stops with Err(Cancelled).
    cancel.store(true, Ordering::Relaxed);
    match ctl.align_variants(&net, &projection, &options) {
        Ok(_) => println!("finished"),
        Err(error) => println!("alignments: {error}"),
    }
    Ok(())
}
output
231 variants aligned
Aligning variants: 231 of Some(231)
Importing: 3454978 of Some(3454978)
Mining process tree: 30 of None
alignments: cancelled

The callbacks need to be Send + Sync, since parallel operations report from several worker threads, e.g., through a Mutex as in the snippet. Parallel operations run on rayon's global thread pool, which uses all cores. To use fewer threads, add rayon to your dependencies and run the call inside rayon::ThreadPoolBuilder::new().num_threads(n).build()?.install(|| ...).