Third-party crates support: `proc-macro2`, `quote`, `syn`, `serde` and `serde_derive` #1007

ojeda · 2023-05-05T23:49:48Z

Like #910, but this time based on top of rust-next (instead of rust), and adds serde and serde_derive on top.

I include a draft commit at the end as a way to test that serde and serde_derive works in both the kernel crate and in a kernel module:

[    0.801425] rust_serde: Rust serde sample (init)
[    0.801634] rust_serde:             original = S { a: (), b: false, c: true, d: () }
[    0.802079] rust_serde:           serialized = [2, 0, 1, 0, 1, 1, 0, 3]
[    0.802506] rust_serde:         deserialized = S { a: (), b: false, c: true, d: () }
[    0.802718] rust_serde:   serialized (local) = [2, 0, 1, 42, 1, 43, 0, 3]
[    0.802895] rust_serde: deserialized (local) = S { a: (), b: false, c: true, d: () }
[    0.808954] rust_serde: Rust serde sample (exit)

Important note: it is not yet decided whether third-party crates will be upstreamed or not, or which ones, but this is useful for others to experiment and prototype solutions assuming these crates are available.

Cc @ariel-miculas who wanted to experiment with serde in the kernel: https://rust-for-linux.zulipchat.com/#narrow/stream/291565-Help/topic/Adding.20crates.20to.20out-of-tree.20modules/near/355792607

bjorn3 · 2023-05-06T12:08:45Z

rust/Makefile

@@ -60,11 +61,73 @@ alloc-cfgs = \
    --cfg no_sync \
    --cfg no_thin

+proc_macro2-skip_flags := \
+    --edition=2021 \
+    -Drust_2018_idioms \


Maybe do --cap-lints allow for all third-party crates? This overrides any -D arguments. It is also what cargo passes for all non-local crates.

bjorn3 · 2023-05-06T12:16:38Z

rust/Makefile

+
+serde-flags := \
+    -Amissing_docs \
+    --cfg no_fp_fmt_parse


If you pass --cfg 'feature="derive"' --extern serde_derive and make the serde target depend on serde_derive, you don't need --extern serde_derive elsewhere, but can instead access the derive macros using use serde::{Serialize, Deserialize};.

nbdd0121 · 2023-05-06T17:11:47Z

I am getting error when rustc-dev component is installed:

error[E0464]: multiple candidates for `dylib` dependency `serde_derive` found
  --> rust/kernel/test_serde.rs:18:5
   |
18 | use serde_derive::{Deserialize, Serialize};
   |     ^^^^^^^^^^^^
   |
   = note: candidate #1: /home/gary/.rustup/toolchains/1.68.2-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libserde_derive-a574e0b043b5a546.so
   = note: candidate #2: /home/gary/Projects/linux/rust/libserde_derive.so

bjorn3 · 2023-05-06T17:23:26Z

That can probably be fixed by passing the path of libserde_derive.so to --extern serde_derive. Might be a good idea to do anyway. Cargo does this too.

ariel-miculas · 2023-05-08T14:45:54Z

I have the following error with rust-analyzer target:

$ make LLVM=1 rust-analyzer
Traceback (most recent call last):
  File "/home/amiculas/work/linux/./scripts/generate_rust_analyzer.py", line 141, in <module>
    main()
  File "/home/amiculas/work/linux/./scripts/generate_rust_analyzer.py", line 134, in main
    "crates": generate_crates(args.srctree, args.objtree, args.sysroot_src),
  File "/home/amiculas/work/linux/./scripts/generate_rust_analyzer.py", line 107, in generate_crates
    if f"{name}.o" not in open(path.parent / "Makefile").read():
FileNotFoundError: [Errno 2] No such file or directory: 'samples/rust/local_data_format/Makefile'
make[1]: *** [rust/Makefile:504: rust-analyzer] Error 1
make: *** [Makefile:1872: rust-analyzer] Error 2

This is a subset of the Rust `proc-macro2` crate, version 1.0.46, licensed under "Apache-2.0 OR MIT", from: https://github.com/dtolnay/proc-macro2/raw/1.0.46/src The files are copied as-is, with no modifications whatsoever (not even adding the SPDX identifiers). For copyright details, please see: https://github.com/dtolnay/proc-macro2/blob/1.0.46/README.md#license https://github.com/dtolnay/proc-macro2/blob/1.0.46/LICENSE-APACHE https://github.com/dtolnay/proc-macro2/blob/1.0.46/LICENSE-MIT The next two patches modify these files as needed for use within the kernel. This patch split allows reviewers to double-check the import and to clearly see the differences introduced. The following script may be used to verify the contents: for path in $(cd rust/proc-macro2/ && find . -type f -name '*.rs'); do curl --silent --show-error --location \ https://github.com/dtolnay/proc-macro2/raw/1.0.46/src/$path \ | diff --unified rust/proc-macro2/$path - && echo $path: OK done Signed-off-by: Miguel Ojeda <[email protected]>

Signed-off-by: Miguel Ojeda <[email protected]>

The `proc-macro2` crate depends on the `unicode-ident` crate to determine whether characters have the XID_Start or XID_Continue properties according to Unicode Standard Annex torvalds#31. However, we only need ASCII identifiers in the kernel, thus we can simplify the check and remove completely that dependency. Signed-off-by: Miguel Ojeda <[email protected]>

This is a subset of the Rust `quote` crate, version 1.0.21, licensed under "Apache-2.0 OR MIT", from: https://github.com/dtolnay/quote/raw/1.0.21/src The files are copied as-is, with no modifications whatsoever (not even adding the SPDX identifiers). For copyright details, please see: https://github.com/dtolnay/quote/blob/1.0.21/README.md#license https://github.com/dtolnay/quote/blob/1.0.21/LICENSE-APACHE https://github.com/dtolnay/quote/blob/1.0.21/LICENSE-MIT The next patch modifies these files as needed for use within the kernel. This patch split allows reviewers to double-check the import and to clearly see the differences introduced. The following script may be used to verify the contents: for path in $(cd rust/quote/ && find . -type f -name '*.rs'); do curl --silent --show-error --location \ https://github.com/dtolnay/quote/raw/1.0.21/src/$path \ | diff --unified rust/quote/$path - && echo $path: OK done Signed-off-by: Miguel Ojeda <[email protected]>

Signed-off-by: Miguel Ojeda <[email protected]>

This is a subset of the Rust `syn` crate, version 1.0.104, licensed under "Apache-2.0 OR MIT", from: https://github.com/dtolnay/syn/raw/1.0.104/src The files are copied as-is, with no modifications whatsoever (not even adding the SPDX identifiers). For copyright details, please see: https://github.com/dtolnay/syn/blob/1.0.104/README.md#license https://github.com/dtolnay/syn/blob/1.0.104/LICENSE-APACHE https://github.com/dtolnay/syn/blob/1.0.104/LICENSE-MIT The next two patches modify these files as needed for use within the kernel. This patch split allows reviewers to double-check the import and to clearly see the differences introduced. The following script may be used to verify the contents: for path in $(cd rust/syn/ && find . -type f -name '*.rs'); do curl --silent --show-error --location \ https://github.com/dtolnay/syn/raw/1.0.104/src/$path \ | diff --unified rust/syn/$path - && echo $path: OK done Signed-off-by: Miguel Ojeda <[email protected]>

Signed-off-by: Miguel Ojeda <[email protected]>

The `syn` crate depends on the `unicode-ident` crate to determine whether characters have the XID_Start or XID_Continue properties according to Unicode Standard Annex torvalds#31. However, we only need ASCII identifiers in the kernel, thus we can simplify the check and remove completely that dependency. Signed-off-by: Miguel Ojeda <[email protected]>

This is a subset of the Rust `serde` crate, version v1.0.156, licensed under "Apache-2.0 OR MIT", from: https://github.com/serde-rs/serde/tree/v1.0.156/serde/src The files are copied as-is, with no modifications whatsoever (not even adding the SPDX identifiers). For copyright details, please see: https://github.com/serde-rs/serde/blob/v1.0.156/README.md#license https://github.com/serde-rs/serde/blob/v1.0.156/LICENSE-APACHE https://github.com/serde-rs/serde/blob/v1.0.156/LICENSE-MIT The next three patches modify these files as needed for use within the kernel. This patch split allows reviewers to double-check the import and to clearly see the differences introduced. The following script may be used to verify the contents: for path in $(cd rust/serde/ && find . -type f -name '*.rs'); do curl --silent --show-error --location \ https://github.com/serde-rs/serde/raw/v1.0.156/serde/src/$path \ | diff --unified rust/serde/$path - && echo $path: OK done Signed-off-by: Miguel Ojeda <[email protected]>

We do not have formatting for floating point in the kernel, thus simple compile out all that. Probably we should name it differently, more similar to `integer128`. Signed-off-by: Miguel Ojeda <[email protected]>

Signed-off-by: Miguel Ojeda <[email protected]>

This is a subset of the Rust `serde_derive` crate, version v1.0.156, licensed under "Apache-2.0 OR MIT", from: https://github.com/serde-rs/serde/tree/v1.0.156/serde_derive/src The files are copied as-is, with no modifications whatsoever (not even adding the SPDX identifiers). For copyright details, please see: https://github.com/serde-rs/serde/blob/v1.0.156/README.md#license https://github.com/serde-rs/serde/blob/v1.0.156/LICENSE-APACHE https://github.com/serde-rs/serde/blob/v1.0.156/LICENSE-MIT The next patch modifies these files as needed for use within the kernel. This patch split allows reviewers to double-check the import and to clearly see the differences introduced. The following script may be used to verify the contents: for path in $(cd rust/serde_derive/ && find . -type f -name '*.rs'); do curl --silent --show-error --location \ https://github.com/serde-rs/serde/raw/v1.0.156/serde_derive/src/$path \ | diff --unified rust/serde_derive/$path - && echo $path: OK done Signed-off-by: Miguel Ojeda <[email protected]>

Signed-off-by: Miguel Ojeda <[email protected]>

…e_derive` With all the new files in place from the new crates, this patch adds support for them in the build system. Note that, if we decide to add third-party crates support to upstream, we would probably want to do it in a different way. But this is useful for playing around with these crates, experimenting, etc. Signed-off-by: Miguel Ojeda <[email protected]>

A trivial example based on `serde`'s `example-format' [1]. It contains both a in-`kernel` data format later used by the kernel module, as well as a local data format in the module. The kernel module gives an output such as: [ 0.801425] rust_serde: Rust serde sample (init) [ 0.801634] rust_serde: original = S { a: (), b: false, c: true, d: () } [ 0.802079] rust_serde: serialized = [2, 0, 1, 0, 1, 1, 0, 3] [ 0.802506] rust_serde: deserialized = S { a: (), b: false, c: true, d: () } [ 0.802718] rust_serde: serialized (local) = [2, 0, 1, 42, 1, 43, 0, 3] [ 0.802895] rust_serde: deserialized (local) = S { a: (), b: false, c: true, d: () } [ 0.808954] rust_serde: Rust serde sample (exit) Note that this is just a quick draft/hack to check the previous commits work. It is not intended to be merged at all. Link: https://github.com/serde-rs/example-format [1] Signed-off-by: Miguel Ojeda <[email protected]>

ojeda · 2023-05-08T14:51:03Z

That should be fixed in commit 5c7548d, but rust-next does not have it yet here.

Let me move rust-next now that v6.4-rc1 is out and rebase this one on top.

XiangruLiu0 · 2024-06-29T05:03:58Z

Could I please know what is preventing this pull request from being merged into the rust-next branch? Incorporating these third-party crates would significantly enhance our ability to develop complex drivers in Rust. 😊

ojeda · 2024-06-29T12:26:15Z

We plan to have support for syn and other selected third-party crates soon in mainline, although in a different form than what is in this PR. What are you planning to use syn for? It could be useful to know. Thanks!

(The PRs here are not used for mainline development; we have some notes in https://rust-for-linux.com/contributing and other pages, in case you have not seen them).

XiangruLiu0 · 2024-06-29T12:50:35Z

Will the lazy_static crate be supported? Static variables are extremely useful, especially in driver development, due to the prevalence of singleton variables.

bjorn3 · 2024-06-29T13:02:27Z

The lazy_static crate has been pretty much subsumed by the once_cell crate. Neither is directly usable in the kernel though as for no_std usages they use a spinlock without any yielding rather than the kernel's version of mutexes. If you want you could write your own version of once_cell which uses the kernel mutex type though. By the way lazy_static is not implemented as proc-macro anyway. It uses a decl macro.

XiangruLiu0 · 2024-07-01T03:21:31Z

Ok I see, thanks 👍

ojeda force-pushed the third-party branch from 9aa35e7 to db6837e Compare May 6, 2023 00:02

bjorn3 reviewed May 6, 2023

View reviewed changes

ojeda added 16 commits May 8, 2023 16:48

rust: proc-macro2: add SPDX License Identifiers

ff8917c

Signed-off-by: Miguel Ojeda <[email protected]>

rust: quote: add SPDX License Identifiers

d7f4b0d

Signed-off-by: Miguel Ojeda <[email protected]>

rust: syn: add SPDX License Identifiers

f9bcfd0

Signed-off-by: Miguel Ojeda <[email protected]>

rust: serde: add no_fp_fmt_parse support

7fcbf51

We do not have formatting for floating point in the kernel, thus simple compile out all that. Probably we should name it differently, more similar to `integer128`. Signed-off-by: Miguel Ojeda <[email protected]>

rust: serde: enable the core_c_str feature

22605b8

Signed-off-by: Miguel Ojeda <[email protected]>

rust: serde: add SPDX License Identifiers

b56a317

Signed-off-by: Miguel Ojeda <[email protected]>

rust: serde_derive: add SPDX License Identifiers

16bebcf

Signed-off-by: Miguel Ojeda <[email protected]>

ojeda force-pushed the third-party branch from db6837e to 7d2ea51 Compare May 8, 2023 14:55

ojeda force-pushed the rust-next branch from bee23a1 to 341faf2 Compare July 19, 2023 18:06

ojeda force-pushed the rust-next branch from bd83e43 to 49a9ef7 Compare August 7, 2023 09:39

ojeda force-pushed the rust-next branch from 71004ce to 4af84c6 Compare August 21, 2023 12:32

ojeda force-pushed the rust-next branch from 2392e2a to e3c3d34 Compare February 18, 2024 20:22

ToolmanP mentioned this pull request Jul 19, 2024

In-tree crate built-in.rlib support? #1091

Open

ojeda force-pushed the rust-next branch from a419846 to 6d1c22d Compare August 21, 2024 22:16

ojeda force-pushed the rust-next branch 2 times, most recently from 278e92e to c9b5ce6 Compare October 7, 2024 21:06

Richardhongyu mentioned this pull request Oct 8, 2024

Tracking issue: refactor RROS kernel code BUPT-OS/RROS#48

Open

3 tasks

ojeda force-pushed the rust-next branch 4 times, most recently from 9ee7197 to 6ce162a Compare October 15, 2024 21:11

ojeda force-pushed the rust-next branch from 0675624 to d072acd Compare November 11, 2024 22:18

ojeda force-pushed the rust-next branch from 92797e9 to 9b98be7 Compare December 16, 2024 20:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Third-party crates support: `proc-macro2`, `quote`, `syn`, `serde` and `serde_derive` #1007

Third-party crates support: `proc-macro2`, `quote`, `syn`, `serde` and `serde_derive` #1007

ojeda commented May 5, 2023 •

edited

Loading

bjorn3 May 6, 2023

bjorn3 May 6, 2023 •

edited

Loading

nbdd0121 commented May 6, 2023

bjorn3 commented May 6, 2023

ariel-miculas commented May 8, 2023

ojeda commented May 8, 2023

XiangruLiu0 commented Jun 29, 2024

ojeda commented Jun 29, 2024

XiangruLiu0 commented Jun 29, 2024

bjorn3 commented Jun 29, 2024 •

edited

Loading

XiangruLiu0 commented Jul 1, 2024

Third-party crates support: proc-macro2, quote, syn, serde and serde_derive #1007

Are you sure you want to change the base?

Third-party crates support: proc-macro2, quote, syn, serde and serde_derive #1007

Conversation

ojeda commented May 5, 2023 • edited Loading

bjorn3 May 6, 2023

Choose a reason for hiding this comment

bjorn3 May 6, 2023 • edited Loading

Choose a reason for hiding this comment

nbdd0121 commented May 6, 2023

bjorn3 commented May 6, 2023

ariel-miculas commented May 8, 2023

ojeda commented May 8, 2023

XiangruLiu0 commented Jun 29, 2024

ojeda commented Jun 29, 2024

XiangruLiu0 commented Jun 29, 2024

bjorn3 commented Jun 29, 2024 • edited Loading

XiangruLiu0 commented Jul 1, 2024

Third-party crates support: `proc-macro2`, `quote`, `syn`, `serde` and `serde_derive` #1007

Third-party crates support: `proc-macro2`, `quote`, `syn`, `serde` and `serde_derive` #1007

ojeda commented May 5, 2023 •

edited

Loading

bjorn3 May 6, 2023 •

edited

Loading

bjorn3 commented Jun 29, 2024 •

edited

Loading