Learning resources

Spring 2021 Global advanced OS courses summary

Advanced OS Course (2021 Spring)

This warehouse is maintained by Professor Chen Yu of Tsinghua University

Github.com/chyyuu/aos_…

Rust -based OS/Hypervisor/VMM/Firmwire:

Github.com/chyyuu/aos_…

Combine Axum, Hyper, Tonic and Tower for hybrid Web/gRPC applications

Series of tutorials written by FPComplete:

  • www.fpcomplete.com/blog/axum-h…
  • www.fpcomplete.com/blog/axum-h…
  • www.fpcomplete.com/blog/axum-h…
  • www.fpcomplete.com/blog/axum-h…

FishFight: A 2D level entry game implemented by the Macroquad game framework

Just open source

  • Github.com/fishfight/F…
  • Fishfight. W.. IO/ff/devlog / 2…

A deep learning researcher learns from Rust’s experience

As a deep learning researcher, at the forefront of deep learning, naturally py and off-the-shelf frameworks are not enough. Rust is an excellent alternative to c++ for implementing the underlying algorithm.

Reddit has related discussion to understand below: www.reddit.com/r/rust/comm…

A few simple tips for writing high-performance Rust code

Keep in mind that the following suggestions are not a substitute for actual analysis and optimization:

  1. Configure release profile:
[profile.release]
lto = "fat"
codegen-units = 1
panic = "abort"

Copy the code
  1. Set up the-C target-cpu=native
  2. Select the Jemalloc or MIMalloc allocator
  3. Read Profile Guided Optimization
  4. Use performance testing tools.

Deterministic. Space/high – perfor…

Use Rust and SIMD acceleration to create the world’s fastest TAC

Tac is a high-performance, simulation-accelerated, cross-platform rewrite of Coreutils’ GNU TAC tool, released under a BSD-compatible (MIT) license. Tac reads the input from a file (or from STDIN) and prints it line by line in reverse.

Tac is cat reversed.

The TAC implementation uses Simd-Acceleration for newline detection (read more about it here) and makes use of memory-mapped files on all supported operating systems. In addition, it is written in Rust for maximum integrity and security.

“It will be much easier to dynamically enable AVX2 support in TAC in 2021!” the authors say. .

If you want to learn about SIMD, check out this article.

  • Readmore
  • tac

Hebi: Snake game powered by Bevy

Hebi is a highly customizable snake clone written in Rust, powered by the Bevy engine and named after the Japanese word for “snake”.

This project helps to learn the Bevy engine.

Github.com/ElnuDev/heb…

Rust in cURL [PodCast

Allen Wyma talks with Daniel, cURL’s original author, about using Rust in cURL.

  • CURL is a command line tool and library for transferring data through urls.
  • CURL and its data transfer core, Libcurl, are written in C, which is notoriously not memory safe.
  • While it’s almost impossible to rewrite it into another language, providing a third-party library written in Rust might go a step further.

Rustacean-station.org/episode/035…

Speed up compilation of Rust

Rust code is notoriously slow to compile. But I have a strong hunch that most Rust code compiles much slower than it could.

For example, Rust’s Rust-Analyzer CI takes eight minutes to operate on GitHub. This is a fairly large and complex project, with 200,000 lines of its own code and 1 million lines of dependencies.

Follow the author as we learn more about how to keep compilation times within reasonable limits!

Matklad. Making. IO / 2021/09/04 /…

Rethinking Rust’s asynchronous syntax

The author of this paper made a conclusion after deep thinking about the problems related to async trait: the current async FN syntax design is wrong.

// this really returns an `impl Future<Output = usize>`, but that's hidden
async fn foo() - >usize { 1 }
Copy the code

This is the current asynchronous syntax, which hides the true return type: IMPl Future.

The authors of this article argue that this return type should not be hidden:

async fn foo() - >impl Future<Output = usize> { 1 }
Copy the code

This article is about the benefits of such explicit returns. The article is very long, you can take a quiet time, follow the author’s thinking to read this article.

Ibraheem. Ca/writings/an…

Rust plug-in development series

This series of articles is from the Issues: Plugin Development Kit (PDK) of Google Code’s Summer Tremor project.

In short, Tremor is an event processing system. It was originally designed as an alternative to software like Logstash or Telegraf. However, tremor moves beyond the scope of this single use case by supporting more complex workflows such as aggregation, summary, ETL language, and query language.

Develop a common interface for the Tremor plug-in to make the library more modular and reduce the core dependency set.

This will greatly reduce the core size of the Tremor, which means that the library will compile faster and the binary size will be smaller. Most importantly, it will transform the Tremor architecture into a fully modular one in which plug-ins can be configured as needed and developed independently in a language-neutral manner.

  • Nullderef.com/series/rust…
  • Github.com/tremor-rs/t…

Why does Rust need Pin and unpin

  • What is a Futures
  • What is a self-referential type
  • Why aren’t they safe
  • How do Pin/Unpin make them secure
  • Write complex futures using Pin/Unpin

Blog.cloudflare.com/pin-and-unp…

Rust’s Logging recommendation

What is the current Recommendation for Logging in Rust? : rust.

Question description: Besides standard logs, there are many options: Env_Logger, tracing, slog, Simplelog, etc. What are the best practices?

From Koxiaet:

There are generally two types of log crate: log interface and log consumer. The interface provides functions to call when you want to log something, and the consumer handles formatting structured log data somewhere (stderr or file). The two main interfaces are log and tracing, which is more powerful because it supports structured logging, but the former is more general. There is also another structured logging interface, Slog, older but less used than Tracing. Each logging interface has its own ecosystem and can be selected according to its own needs. If you’re writing a library, log is a good choice because all logging interfaces are compatible with it. However, if you do need structured logging, tracing can be used instead, depending on your needs, such as whether you want to write to a file or just to a terminal.

Other netizens’ recommendations:

  • File Logging: EMabee/FLEXI_LOGGER: A flexible logger for rust programs that can write to stderr or to log files. (From CFSamson)
  • Tracing interface: Tracing_log-rust, which is particularly convenient when you have multiple interleaved log messages operating at the same time, can group them according to certain attributes and view them individually. (From class_two_perversion)
  • Estk/Log4RS is A highly configurable logging framework for Rust, based on the Java Logback and LOG4J libraries. Through Yaml configuration, to SDout and files, with file size limiting options, you can also configure different levels of logging. (From TMS102)
  • Tracing – appender-crates. IO: Rust Package Registry, the only out-of-thread logging solution known to referrer, not only for asynchronous applications. (From Pand9)
  • Daboross /fern: Simple, efficient logging for Rust, like Python logging and JS Winston. (From RapBeautician)

Rust the whole stack

Full Stack Rust-Blog (Full Stack Rust-Blog)

A year ago, my preferred language was:

  • Python is used for advanced code rapid prototyping, or for code that requires third-party functionality
  • C/C++ is used for long-term low-level projects

Having only heard of Rust and used it briefly at the time, my experience came from writing a widget in Rust that processed transactions for large files (>4GB) and mined some statistics from it. I used a library to map files into memory, and Bin Rui analyzed them in order. There are some cool concepts, such as the compiler statically forcing a memory map to be inaccessible after it has been unmapped — an error that can happen in C++ if you’re not careful.

But it didn’t really appeal to me at the time because it was just a novelty. The trick came when I added new functionality to pdblister to get thousands of PDB files in parallel. Because of the GIL, it’s almost impossible in CPython, and it’s extremely difficult in C/C++ to not face parallel errors. Rust, however, makes it easy. I added asynchrony to the Tokio driver, used tokio::spawn to generate a new task to download the PDB, and fixed the compiler error and it worked fine. The Rust compiler outputs a binary file that can be run anywhere with no runtime dependencies.

Replace the Python

This is the first point. Rust is an excellent alternative to Python as a medium to long term tool language. The nice thing about Python is that it has a huge library and ecosystem, which is available directly through PIP. For quick prototyping of API interactions, you can use Requests, just import Requests. The same is true for Rust’s Reqwest, which you can use in your code simply by typing Cargo Add Reqwest.

However, Python is at a disadvantage when it comes to the longer life cycle, where requests are program dependencies that users need to be able to use later. In addition, Python is even more disadvantaged because of its weak typing and error handling capabilities (compared to Rust). At this point, I can write prototyping tools faster with Rust than with Python, and I can confidently know that my tools are easier to maintain and last longer than their Python equivalents. However, Python may still be better for short-term tools because it doesn’t need to start a project to get intellisense support in VSCode. Rust’s cargo script comes close to pushing Rust into scripting language territory, but unfortunately I haven’t found a plug-in integrated with it in VSCode yet.

Instead of C

Rust is also a direct substitute for C, which is better in every way and can interoperate natively with legacy C code for incremental replacement. Rust’s biggest improvement is the ecosystem: As mentioned above, it’s easy to leverage libraries already in Rust ecology. If you’ve never used C before, you’re lucky. Actually the best way to use advanced features in C is to write them yourself.

C Ecosystems are fragmented and fragile. There is no consistent standard for an ABI or build system:

  • Due to the lack of ABI consistency, you cannot use the same binaries across platforms or operating systems. So you have to build from source.
  • Because of the lack of a consistent build system, you can’t simply build C libraries with your application and must patch or rewrite the build system of the library to make it compatible with your library.
  • C libraries are rarely cross-platform compatible because they lack shared abstractions to rely on.

Then there are Rust’s most distinctive security improvements — I won’t go into them. But in my experience – security is largely a tool that makes it easier for third-party library developers to force me to use their libraries properly, something C libraries can’t do.

The whole stack Rust

All in all, I’ve been using Rust for all parts of the stack for the past year, and I’ve used other languages before. I have used Rust to implement the boot loader: Xenia-project/Xell-RS: Xell Bootloader, rewritten in Rust Because ¯_(ツ)_/¯, which I’ve used to mirror files via advanced HTTP/HTTPS and other techniques in Pdblister and Panamax. I took advantage of and contributed the excellent GDBStub library for controlling VMS running on a custom VMM. These projects are done at different levels of the stack, and Rust fits well at all levels. I have started using Rust exclusively in my personal projects and have pushed for it to be used in my work when appropriate.

Tagged_cell: Fast, initializable, and thread-safe static variable

Implemented with the TaggedCell and Tag types, each instance of the TaggedCell must be unique for safe operation. The TaggedCell must then be initialized with TaggedCell::init(), which initializes the underlying data using a user-supplied function or closure, and then returns a special zero-size init

for accessing the Cell’s data. To ensure that each cell uses a unique tag type, tagged_cell! Provide macros. This macro creates a new tag type based on the name of the variable and applies it to the declaration.

usetagged_cell::tagged_cell; tagged_cell! {static BAR: TaggedCell<Vec<usize>, _> = TaggedCell::new();
}

let tag = BAR.init(|| vec![0.10.20]);
let vec = BAR.get(tag);

assert_eq!(vec[2].20);
Copy the code

To allow cross-threading, only the first call to TaggedCell::init initializes the Cell’s data. All future TaggedCell::init calls will return a new label. It is not determined which thread will initialize the Cell’s data.

use std::thread;
usetagged_cell::tagged_cell; tagged_cell! {static TABLE: TaggedCell<Vec<usize>, _> = TaggedCell::new();
}

thread::spawn(move| | {let tag = TABLE.init(|| vec![0.10.20]);
    let table = TABLE.get(tag);
    assert_eq!(table[2].20);
});

thread::spawn(move| | {let tag = TABLE.init(|| vec![0.10.20]);
    let table = TABLE.get(tag);
    assert_eq!(table[1].10);
});
Copy the code

Github.com/Dasch0/tagg…

Learn Wgpu updated

Wgrpu is the Rust implementation of the WebGPU API Spec. This tutorial has been updated to version 0.10 with extensive principles and code examples.

Sotrh. Making. IO/learn – wgpu /…

Use Rust to write shellcode

Shellcode is an essential part of vulnerability exploitation. Shellcode is used to send the code to the server to exploit specific vulnerabilities. Shellcode can complete some basic and important work in a very small space.

There are basically three ways to write Shellcode:

  1. Writing hexadecimal opcodes directly (impractical);
  2. Programs are written in a high-level language such as C, compiled and disassembled to obtain assembly instructions and hexadecimal opcodes.
  3. Compiles an assembler, assembles the program, and then extracts the hexadecimal opcodes from binary.

The article by the author of Black Hat Rust describes how to write ShellCode using Rust.

Kerkour.com/blog/shellc…

A QR code generation service developed using the Axum framework

An example of axum’s field project is a service that is actually running:

Github.com/sayanarijit…

Rust CI/CD: github action

As with any other language, we tend to have CI/CD requirements in addition to our mastery of syntax:

  • What components are needed to make up the CI pipeline to ensure that my code is healthy?
  • How to deploy it?
  • Do I need to write custom tools or do I have community resources available?

There will be three articles explaining how Rust uses Action to accomplish CI/CD on Github.

www.homeops.dev/continuous-…

Build a static site generator with 100 lines of Rust code

The principle is simple, by parsing markdown files and filling in an HTML template. To complete.

Kerkour.com/blog/rust-s…

Implement RPGS with Rust and Godot

There is a video Series on YouTube called Godot Action RPG Series where you can learn how to make an Action RPG in Godot Engine 3.2.

Godot is an excellent free and open source game engine designed for indie games.

But the YouTube video was not Rust, so someone implemented it in Rust and published it:

Github.com/Nejat/godot…

Build extensible services using Rust and Tokio

With Tokio, web server applications are easy to implement. However, the typical code you often find in guidebooks or such books scales poorly across multiple cpus. I’ll explain why and how to fix this problem by changing a few lines.

medium.com/@fujita.tom…