From 3e45fc0f37d027516ac3d112ca7768d698eeac74 Mon Sep 17 00:00:00 2001 From: Hilton Chain Date: Tue, 18 Mar 2025 14:22:55 +0800 Subject: [PATCH] doc: Document lockfile importer based Rust packaging workflow. * doc/guix.texi (Build Systems) [cargo-build-system]: Add cross-reference for the term "Cargo workspaces". * doc/contributing.texi (Packaging Guidelines)[Rust Crates]: Update documentation. * doc/guix-cookbook.texi (Packaging)[Packaging Workflow]: New section. * gnu/packages/rust-crates.scm, * gnu/packages/rust-sources.scm: Stop mentioning guix-rust-registry for now, we may remove the repository if future merges are managed well. Change-Id: Ic0c6378cf5f5df97d6f8bdd040b486be62c7bddc --- doc/contributing.texi | 92 ++++++--- doc/guix-cookbook.texi | 377 ++++++++++++++++++++++++++++++++++ doc/guix.texi | 13 +- gnu/packages/rust-crates.scm | 4 - gnu/packages/rust-sources.scm | 4 - 5 files changed, 452 insertions(+), 38 deletions(-) diff --git a/doc/contributing.texi b/doc/contributing.texi index 6d3f1622f30..c801ba5950a 100644 --- a/doc/contributing.texi +++ b/doc/contributing.texi @@ -1635,34 +1635,78 @@ dashes and prepend the prefix @code{java-}. So the class @subsection Rust Crates @cindex rust -Rust programs standing for themselves are named as any other package, using the -lowercase upstream name. +Rust applications (binary crates) and libraries (library crates) are packaged +separately. We put our main efforts into applications and only package +libraries as sources, utilizing automation with a manual focus on unbundling +vendored dependencies. -To prevent namespace collisions we prefix all other Rust packages with the -@code{rust-} prefix. The name should be changed to lowercase as appropriate and -dashes should remain in place. +Rust applications are treated like any other package and named using the +lowercase upstream name. When using the Cargo build system (@pxref{Build +Systems, @code{cargo-build-system}}), Rust applications should have the +@code{#:install-source?} parameter set to @code{#f}, as this parameter only +makes sense for libraries. When the package source is a Cargo workspace, +@code{#:cargo-install-paths} must be set to enable relevant support. -In the rust ecosystem it is common for multiple incompatible versions of a -package to be used at any given time, so all package definitions should have a -versioned suffix. The versioned suffix is the left-most non-zero digit (and -any leading zeros, of course). This follows the ``caret'' version scheme -intended by Cargo. Examples@: @code{rust-clap-2}, @code{rust-rand-0.6}. +Rust libraries are hidden from the user interface and managed in two modules: -Because of the difficulty in reusing rust packages as pre-compiled inputs for -other packages the Cargo build system (@pxref{Build Systems, -@code{cargo-build-system}}) presents the @code{#:cargo-inputs} and -@code{cargo-development-inputs} keywords as build system arguments. It would be -helpful to think of these as similar to @code{propagated-inputs} and -@code{native-inputs}. Rust @code{dependencies} and @code{build-dependencies} -should go in @code{#:cargo-inputs}, and @code{dev-dependencies} should go in -@code{#:cargo-development-inputs}. If a Rust package links to other libraries -then the standard placement in @code{inputs} and the like should be used. +@table @code +@item (gnu packages rust-crates) +Source definitions imported from Rust packages' @file{Cargo.lock} via the +@code{crate} importer (@pxref{Invoking guix import}). -Care should be taken to ensure the correct version of dependencies are used; to -this end we try to refrain from skipping the tests or using @code{#:skip-build?} -when possible. Of course this is not always possible, as the package may be -developed for a different Operating System, depend on features from the Nightly -Rust compiler, or the test suite may have atrophied since it was released. +Imported definitions must be checked and have vendored dependencies unbundled +before being contributed to Guix. + +@item (gnu packages rust-sources) +More complex definitions that need to be full packages. This includes Rust +libraries requiring external inputs to unbundle and Cargo workspaces. + +These libraries should have the @code{#:skip-build?} parameter set to @code{#t}. +For Cargo workspaces, @code{#:cargo-package-crates} must be set. + +Since they are added manually, they follow the usual naming convention for Guix +packages (@pxref{Package Naming}), with a @code{rust-} prefix. + +In the Rust community it is common for multiple incompatible versions of a +package to be used at any given time, so all libraries should have a versioned +suffix. The versioned suffix is the left-most non-zero digit (and any leading +zeros, of course). This follows the ``caret'' version scheme intended by Cargo. +Examples@: @code{rust-clap-2}, @code{rust-rand-0.6}. + +In practice we often package development snapshots of Rust libraries and can't +simply identify them by version. In this case, the complete version information +can be turned into a version string via @code{git-version}, for example +@code{rust-pipewire-0.8.0.fd3d8f7} and @code{rust-pubgrub-0.3.0.b70cf70}. +@end table + +To avoid merge conflicts with changes from multiple branches, these two modules +are maintained by the Rust team (@pxref{Teams}). + +Rust libraries are not referenced directly. @code{(guix build-sytem cargo)} +provides a @code{cargo-inputs} procedure to lookup input lists generated by the +lockfile importer. + +@cindex cargo inputs +@findex define-cargo-inputs +@findex lookup-cargo-inputs +@deffn {Procedure} cargo-inputs name [#:module '(gnu packages rust-crates)] +Lookup Cargo inputs for @var{name} in @var{module}, return an empty list if +unavailable. + +@var{name} must be consistent with the one used in lockfile importer invocation, +usually matching the variable name of the package: + +@example +$ guix import -i @file{gnu/packages/rust-crates.scm} crate -f @file{/path/to/Cargo.lock} @var{name} +@end example + +@var{module} must export a public interface @code{lookup-cargo-inputs}, a +template is available in the @file{etc/teams/rust} directory of Guix source +tree. +@end deffn + +@xref{Packaging Rust Crates,,, guix-cookbook, GNU Guix Cookbook}, for +packaging workflow. @node Elm Packages diff --git a/doc/guix-cookbook.texi b/doc/guix-cookbook.texi index b0567047033..bd2add42856 100644 --- a/doc/guix-cookbook.texi +++ b/doc/guix-cookbook.texi @@ -106,6 +106,7 @@ Scheme tutorials Packaging * Packaging Tutorial:: A tutorial on how to add packages to Guix. +* Packaging Workflows:: Real life examples on working with specific build systems. Packaging Tutorial @@ -130,6 +131,16 @@ Programmable and automated package definition * Automatic update:: * Inheritance:: +Packaging Workflows + +* Packaging Rust Crates:: + +Packaging Rust Crates + +* Common Workflow for Rust Packaging:: +* Cargo Workspaces and Development Snapshots:: +* Using Rust Libraries in Other Build Systems:: + System Configuration * Auto-Login to a Specific TTY:: Automatically Login a User to a Specific TTY @@ -519,6 +530,7 @@ them. @menu * Packaging Tutorial:: A tutorial on how to add packages to Guix. +* Packaging Workflows:: Real life examples on working with specific build systems. @end menu @node Packaging Tutorial @@ -1599,6 +1611,371 @@ The @uref{https://guix.gnu.org/manual/en/html_node/Defining-Packages.html, packa @uref{https://guix.gnu.org/guix-ghm-andreas-20130823.pdf, ``GNU Guix: Package without a scheme!''}, by Andreas Enge @end itemize +@node Packaging Workflows +@section Packaging Workflows + +The following sections provide real-life examples on working with specific build +systems, serving as extensions to the concise packaging guidelines +(@pxref{Packaging Guidelines,,, guix, GNU Guix Reference Manual}). + +@menu +* Packaging Rust Crates:: +@end menu + +@node Packaging Rust Crates +@subsection Packaging Rust Crates + +In preparation, add the following packages to our environment: + +@example +$ guix shell rust rust:cargo cargo-audit cargo-license +@end example + +@menu +* Common Workflow for Rust Packaging:: +* Cargo Workspaces and Development Snapshots:: +* Using Rust Libraries in Other Build Systems:: +@end menu + +@node Common Workflow for Rust Packaging +@subsubsection Common Workflow for Rust Packaging + +In this example, we'll package @code{cargo-audit}, which is published on the +@uref{https://crates.io, crates.io} Rust package repository. All its +dependencies are on crates.io as well. + +@enumerate +@item +Since @code{cargo-audit} is available on crates.io, we can generate a template +via the crates.io importer (@pxref{Invoking guix import,,, guix, GNU Guix +Reference Manual}): + +@example +$ guix import crate cargo-audit +@end example + +After manual editing, we'll have the following definiton: + +@lisp +(define-public cargo-audit + (package + (name "cargo-audit") + (version "0.21.2") + (source + (origin + (method url-fetch) + (uri (crate-uri "cargo-audit" version)) + (file-name (string-append name "-" version ".tar.gz")) + (sha256 + (base32 "1a00yqpckkw86zh2hg7ra82c5fx0ird5766dyynimbvqiwg2ps0n")))) + (build-system cargo-build-system) + (arguments (list #:install-source? #f)) + (inputs (cargo-inputs 'cargo-audit)) + (home-page "https://rustsec.org/") + (synopsis "Audit Cargo.lock for crates with security vulnerabilities") + (description + "This package provides a Cargo subcommand, @@command@{cargo audit@}, to +audit @@file@{Cargo.lock@} for crates with security vulnerabilities.") + (license (list license:asl2.0 license:expat)))) +@end lisp + +The identifier used to invoke @code{cargo-inputs}, in this case +@code{'cargo-audit}, must be unique, usually matching the variable name of the +package. + +@item +Unpack package source and navigate to the unpacked directory, then run the +following commands: + +@example +$ cargo generate-lockfile +$ cargo audit +$ cargo license +@end example + +@command{cargo generate-lockfile} updates dependencies to compatible versions. +Applying it to all Rust applications helps reduce a great number of Rust +libraries we need to check later. Although sometimes libraries may fail to +follow the @uref{https://semver.org/, semantic versioning} scheme, it's still +acceptable. + +@command{cargo audit} checks known vulnerabilities and @command{cargo license} +checks licenses, for all the dependencies. We must have an acceptable output of +@command{cargo audit} and ensure all dependencies are licensed with our +supported licenses (@pxref{Defining Packages,,, guix, GNU Guix Reference +Manual}). + +@item +Import dependencies from the generated lockfile: + +@example +$ guix import --insert=gnu/packages/rust-crates.scm \ + crate --lockfile=/path/to/Cargo.lock cargo-audit + +# Or use short options, in this case the shell processes file names +# before passing them to Guix, allowing tilde expansion, for example. +$ guix import -i gnu/packages/rust-crates.scm \ + crate -f /path/to/Cargo.lock cargo-audit +@end example + +@code{cargo-audit} here must be consistent with the identifier used for +@code{cargo-inputs} invocation in the package definition. + +At this stage, the package @code{cargo-audit} is buildable. + +@item +Finally we'll unbundle the vendored dependencies. The lockfile importer +inserts @code{TODO:} comments for libraries with high probability of +bundled dependencies. @code{cargo-build-system} also performs +additional check for binary files in its +@code{check-for-pregenerated-files} phase, which usually indicates +bundling: + +@example +$ ./pre-inst-env guix build cargo-audit +@dots{} +starting phase `check-for-pregenerated-files' +Searching for binary files... +./guix-vendor/rust-async-compression-0.4.21.tar.gz/tests/artifacts/dictionary-rust +./guix-vendor/rust-async-compression-0.4.21.tar.gz/tests/artifacts/dictionary-rust-other +./guix-vendor/rust-async-compression-0.4.21.tar.gz/tests/artifacts/lib.rs.zst +./guix-vendor/rust-async-compression-0.4.21.tar.gz/tests/artifacts/long-window-size-lib.rs.zst +./guix-vendor/rust-winapi-i686-pc-windows-gnu-0.4.0.tar.gz/lib/libwinapi_aclui.a +./guix-vendor/rust-winapi-i686-pc-windows-gnu-0.4.0.tar.gz/lib/libwinapi_activeds.a +./guix-vendor/rust-winapi-i686-pc-windows-gnu-0.4.0.tar.gz/lib/libwinapi_asycfilt.a +./guix-vendor/rust-winapi-i686-pc-windows-gnu-0.4.0.tar.gz/lib/libwinapi_amsi.a +@dots{} +@end example + +Although Rust libraries are not publicly exported, we can still select them via +the Guix command-line interface via a Guile expression: + +@example +$ guix build --expression='(@@@@ (gnu packages rust-crates) rust-ring-0.17.14)' +@end example + +To unbundle most dependencies, a snippet is sufficient: + +@lisp +(define rust-curl-sys-0.4.80+curl-8.12.1 + (crate-source "curl-sys" "0.4.80+curl-8.12.1" + "0d7ppx4kq77hc5nyff6jydmfabpgd0i3ppjvn8x0q833mhpdzxsm" + #:snippet '(delete-file-recursively "curl"))) +@end lisp + +@lisp +(define rust-bzip2-sys-0.1.13+1.0.8 + (crate-source "bzip2-sys" "0.1.13+1.0.8" + "056c39pgjh4272bdslv445f5ry64xvb0f7nph3z7860ln8rzynr2" + #:snippet + '(begin + (delete-file-recursively "bzip2-1.0.8") + (delete-file "build.rs") + (with-output-to-file "build.rs" + (lambda _ + (format #t "fn main() @{~@@ + println!(\"cargo:rustc-link-lib=bz2\");~@@ + @}~%")))))) +@end lisp + +In a more complex case, where unbundling one dependency requires a build process +that involves other packages, we should make a full package in @code{(gnu +packages rust-sources)} first and reference it in the imported definition. + +For example, we have defined a @code{rust-ring-0.17} in @code{(gnu packages +rust-sources)}, then the imported definition in @code{(gnu packages +rust-crates)} should be modified to reference it. + +@lisp +(define rust-ring-0.17.14 rust-ring-0.17) +@end lisp + +When one dependency can be safely removed, modify it to @code{#f}. + +@lisp +(define rust-openssl-src-300.4.2+3.4.1 #f) +@end lisp + +To facilitate various tasks in the common workflow, several scripts are provided +in the @file{etc/teams/rust} directory of Guix source tree. +@end enumerate + +@node Cargo Workspaces and Development Snapshots +@subsubsection Cargo Workspaces and Development Snapshots + +In this example, we'll package @code{niri}, which depends on development +snapshots (also Cargo workspaces here). + +As we can't ensure compatibility of a development snapshot, before executing +@command{cargo generate-lockfile}, we should modify @file{Cargo.toml} to pin it +to a known working revision. + +To use our packaged development snapshots, it's also necessary to modify +@file{Cargo.toml} in a build phase, with a package-specific substitution +pattern. + +@lisp +(define-public niri + (package + (name "niri") + (version "25.02") + (source (origin + (method git-fetch) + (uri (git-reference + (url "https://github.com/YaLTeR/niri") + (commit (string-append "v" version)))) + (file-name (git-file-name name version)) + (sha256 + (base32 + "0vzskaalcz6pcml687n54adjddzgf5r07gggc4fhfsa08h1wfd4r")))) + (build-system cargo-build-system) + (arguments + (list #:install-source? #f + #:phases + #~(modify-phases %standard-phases + (add-after 'unpack 'use-guix-vendored-dependencies + (lambda _ + (substitute* "Cargo.toml" + (("# version =.*") + "version = \"*\"") + (("git.*optional") + "version = \"*\", optional") + (("^git = .*") + ""))))))) + (native-inputs + (list pkg-config)) + (inputs + (cons* clang + libdisplay-info + libinput-minimal + libseat + libxkbcommon + mesa + pango + pipewire + wayland + (cargo-inputs 'niri))) + (home-page "https://github.com/YaLTeR/niri") + (synopsis "Scrollable-tiling Wayland compositor") + (description + "Niri is a scrollable-tiling Wayland compositor which arranges windows in a +scrollable format. It is considered stable for daily use and performs most +functions expected of a Wayland compositor.") + (license license:gpl3+))) +@end lisp + +@code{niri} has Cargo workspace dependencies. When packaging a Cargo +workspace dependency, parameter @code{#:cargo-package-crates} is +required. + +@lisp +(define-public rust-pipewire-0.8.0.fd3d8f7 + (let ((commit "fd3d8f7861a29c2eeaa4c393402e013578bb36d9") + (revision "0")) + (package + (name "rust-pipewire") + (version (git-version "0.8.0" revision commit)) + (source + (origin + (method git-fetch) + (uri (git-reference + (url "https://gitlab.freedesktop.org/pipewire/pipewire-rs.git") + (commit commit))) + (file-name (git-file-name name version)) + (sha256 + (base32 "1hzyhz7xg0mz8a5y9j6yil513p1m610q3j9pzf6q55vdh5mcn79v")))) + (build-system cargo-build-system) + (arguments + (list #:skip-build? #t + #:cargo-package-crates + ''("libspa-sys" "libspa" "pipewire-sys" "pipewire"))) + (inputs (cargo-inputs 'rust-pipewire-0.8.0.fd3d8f7)) + (home-page "https://pipewire.org/") + (synopsis "Rust bindings for PipeWire") + (description "This package provides Rust bindings for PipeWire.") + (license license:expat)))) +@end lisp + +Don't forget to modify all workspace members in @code{(gnu packages +rust-crates)}: + +@lisp +(define rust-pipewire-0.8.0.fd3d8f7 rust-pipewire-0.8.0.fd3d8f7) +(define rust-pipewire-sys-0.8.0.fd3d8f7 rust-pipewire-0.8.0.fd3d8f7) +@dots{} +(define rust-libspa-0.8.0.fd3d8f7 rust-pipewire-0.8.0.fd3d8f7) +(define rust-libspa-sys-0.8.0.fd3d8f7 rust-pipewire-0.8.0.fd3d8f7) +@end lisp + +@node Using Rust Libraries in Other Build Systems +@subsubsection Using Rust Libraries in Other Build Systems + +In this example, we'll package @code{libchewing}, which combines two build +systems. + +When building Rust packages in other build systems, we need to add @code{rust}, +and @code{rust:cargo} to @code{native-inputs}, import and use modules from both +build systems, and apply necessary build phases from @code{cargo-build-system}. + +For cross-compilation support, we'll also add @code{rust-sysroot} created with +@code{(make-rust-sysroot (%current-target-system))} to @code{native-inputs}, and +set @code{#:cargo-target} for @code{cargo-build-system}'s build phases. + +@lisp +(define-public libchewing + (package + (name "libchewing") + (version "0.9.1") + (source + (origin + (method git-fetch) + (uri (git-reference + (url "https://github.com/chewing/libchewing") + (commit (string-append "v" version)))) + (file-name (git-file-name name version)) + (sha256 + (base32 "0gh64wvrk5pn0fhmpvj1j99d5g7f7697rk96zbkc8l72yjr819z5")))) + (build-system cmake-build-system) + (arguments + (list #:imported-modules + (append %cmake-build-system-modules + %cargo-build-system-modules) + #:modules + '(((guix build cargo-build-system) #:prefix cargo:) + (guix build cmake-build-system) + (guix build utils)) + #:phases + #~(modify-phases %standard-phases + (add-after 'unpack 'prepare-cargo-build-system + (lambda args + (for-each + (lambda (phase) + (format #t "Running cargo phase: ~a~%" phase) + (apply (assoc-ref cargo:%standard-phases phase) + ;; For cross-compilation. + #:cargo-target #$(cargo-triplet) + args)) + '(unpack-rust-crates + configure + check-for-pregenerated-files + patch-cargo-checksums))))))) + (native-inputs + (append + (list rust `(,rust "cargo") ) + ;; For cross-compilation. + (or (and=> (%current-target-system) + (compose list make-rust-sysroot)) + '()))) + (inputs + (cons* corrosion ncurses sqlite (cargo-inputs 'libchewing))) + (synopsis "Chinese phonetic input method") + (description "Chewing is an intelligent phonetic (Zhuyin/Bopomofo) input +method, one of the most popular choices for Traditional Chinese users.") + (home-page "https://chewing.im/") + (license license:lgpl2.1+))) +@end lisp + + @c ********************************************************************* @node System Configuration @chapter System Configuration diff --git a/doc/guix.texi b/doc/guix.texi index 9147cd61ca6..e2d546be0a0 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -9573,12 +9573,13 @@ crate. Unless @code{install-source? #f} is defined it will also install a source crate repository of itself and unpacked sources, to ease in future hacking on Rust packages. -This build system supports Cargo workspaces. Parameter -@code{#:cargo-package-crates} (default: @code{''()}) allows specifying names of -library crates to package in the @code{package} phase. Specified crates are -packaged from left to right, in case there's dependency among them. For -example, specifying @code{''("pcre2-sys" "pcre2")} will package -@code{"pcre2-sys"} first and then @code{"pcre2"}. Parameter +This build system supports +@url{https://doc.rust-lang.org/cargo/reference/workspaces.html, Cargo +workspaces}. Parameter @code{#:cargo-package-crates} (default: @code{''()}) +allows specifying names of library crates to package in the @code{package} +phase. Specified crates are packaged from left to right, in case there's +dependency among them. For example, specifying @code{''("pcre2-sys" "pcre2")} +will package @code{"pcre2-sys"} first and then @code{"pcre2"}. Parameter @code{#:cargo-install-paths} (default: @code{''()}) allows specifying paths of binary crates to install in the @code{install} phase, @code{''("crates/atuin")}, for example. diff --git a/gnu/packages/rust-crates.scm b/gnu/packages/rust-crates.scm index 8612868871b..73975b79c4f 100644 --- a/gnu/packages/rust-crates.scm +++ b/gnu/packages/rust-crates.scm @@ -27,10 +27,6 @@ ;;; ;;; This file is managed by ‘guix import’. Do NOT add definitions manually. -;;; -;;; Commit changes to: https://codeberg.org/guix/guix-rust-registry -;;; - ;;; ;;; Rust libraries fetched from crates.io and non-workspace development ;;; snapshots. diff --git a/gnu/packages/rust-sources.scm b/gnu/packages/rust-sources.scm index 9fa7689b36a..1950a79a73a 100644 --- a/gnu/packages/rust-sources.scm +++ b/gnu/packages/rust-sources.scm @@ -36,10 +36,6 @@ #:use-module (gnu packages python) #:use-module (gnu packages textutils)) -;;; -;;; Commit changes to: https://codeberg.org/guix/guix-rust-registry -;;; - ;;; ;;; Cargo workspaces and Rust libraries requiring external inputs to unbundle. ;;; These packages are hidden, as they are not interesting to users.