TOPTRACING

Yazi: blazing fast terminal file manager written in Rust

Dec 25, 2024

0
https://yazi-rs.github.io/https://github.com/sxyazi/yazi

Yazi (means "duck") is a terminal file manager written in Rust, based on non-blocking async I/O. It aims to provide an efficient, user-friendly, and customizable file management experience.

Features

  • Full Asynchronous Support: All I/O operations are asynchronous, CPU tasks are spread across multiple threads, making the most of available resources.
  • Powerful Async Task Scheduling and Management: Provides real-time progress updates, task cancellation, and internal task priority assignment.
  • Built-in Support for Multiple Image Protocols: Also integrated with Überzug++ and Chafa, covering almost all terminals.
  • Built-in Code Highlighting and Image Decoding: Combined with the pre-loading mechanism, greatly accelerates image and normal file loading.
  • Concurrent Plugin System: UI plugins (rewriting most of the UI), functional plugins, custom previewer/preloader/spotter/fetcher; Just some pieces of Lua.
  • Data Distribution Service: Built on a client-server architecture (no additional server process required), integrated with a Lua-based publish-subscribe model, achieving cross-instance communication and state persistence.
  • Package Manager: Install plugins and themes with one command, keeping them up to date, or pin them to a specific version.
  • Integration with ripgrep, fd, fzf, zoxide
  • Vim-like input/pick/confirm/which/notify component, auto-completion for cd paths
  • Multi-Tab Support, Cross-directory selection, Scrollable Preview (for videos, PDFs, archives, code, directories, etc.)
  • Bulk Renaming, Visual Mode, File Chooser
  • Theme System, Mouse Support, Trash Bin, Custom Layouts, CSI u, OSC 52
  • ... and more!

Why is Yazi fast?

Yazi has undergone significant optimizations to enhance user experience. It is designed entirely as an async program, handling all time-consuming tasks (I/O and CPU) as async tasks in a non-blocking, event-driven manner.

Tokio​

Internally, Yazi uses Tokio as its async runtime: hold on! Tokio's async may not be "truly async" as you might perceive it!

Uh, okay. From an application-layer perspective, it indeed is async; however, from a system-level view, there are possibly better solutions.

But! This is not the current performance bottleneck for Yazi. Considering Yazi is a TUI app, unlike CLI programs like ls and eza that need to output all files immediately, Yazi has more optimization opportunities at the application-layer:

  • For large directories (e.g., 100,000 files), Yazi uses chunked loading, which is unmatched by ls and eza since they must load everything at once.
  • Yazi also preloads directory file lists in the background, an optimization that ls and eza do not possess.

I must express my gratitude to Tokio for providing an excellent and convenient way to realize these application-layer optimizations.

I believe that the benefits brought by these application-level optimizations are more noticeable compared to switching to solutions like io_uring. But I'm open to this and welcome any constructive PR.

Here is a relevant discussion on Reddit: https://www.reddit.com/r/rust/comments/16fxr58/comment/k066gmh/

Pre-Loading​

Preloaders are part of Yazi's concurrent plugin system, and the entire pre-loading process is asynchronous and spans multiple threads. This means that preloaders can handle not only expensive IO tasks but also CPU-bound tasks! Here are some built-in preloaders in Yazi:

  • Mimetype: The baseline. Yazi uses the file's mime-type as a reference for tasks such as opening, previewing, and style rendering, and internally utilizes file(1) to obtain the file's mime-type. For better performance, Yazi computes them for files of an entire page, rather than for each file individually, and the entire process is chunked to minimize response latency.
  • Image: To accelerate image previews, Yazi uses a 2-pass process for image files. The first pass is preprocessing, which downscales the image based on user-set max_width/max_height and generates a compressed lossy image as a cache file, significantly reducing file size. The second pass occurs when the user actually switches to the file and downscales it again to fit the terminal size.
  • Video: To speed up video previews, Yazi pre-converts them into images and goes through the first pass of image processing. When the user needs to display the video, it goes the same second pass.
  • PDF: Similar to video.
  • Directory size: Yazi lazily calculates the directory size only when the user sets sorting by file size, as it's a time-consuming operation.

Note: Except for size, all of these are paged, meaning that when you are on the first page, only the first few files will be pre-loaded.

For example, if your directory has 1000 files, your terminal height is 10, and you are on the second page, only files 11 to 20 will be processed. This greatly saves resources.

Discardable Tasks​

Every preview task is discardable. When you navigate between files quickly and the previous file's triggered preview task is still not finished, it will be discarded directly, initiating a new task. This promotes resource utilization:

  • For I/O tasks like loading directory lists, Tokio's abort is used;
  • For CPU tasks like code highlighting, an Atomic is used to store a ticket, and it checks if the value changes on each line code highlight. If it changes, indicates that the current context has changed, and the entire highlighting task is discarded.
  • For I/O and CPU tasks like previewer/preloader plugins, with Lua, Yazi can check whether these tasks are canceled when a specific number of CPU instructions. If canceled, it interrupts the execution of the Lua script immediately, avoiding wasting more I/O and CPU resources.

Code Highlighting​

Yazi has built-in code highlighting and keeps it to a minimum for all text files: if your terminal height is 10, only the first 10 lines of the file are read and highlighted.

Other file managers that rely on external programs like bat need to wait for bat to finish highlighting the entire file before displaying only the first 10 lines.

In cases like JSON that require external program jq, Yazi kills jq directly after reading the first 10 lines to avoid unnecessary resource consumption.

Since code highlighting is a CPU-bound task, it is distributed among multiple blocking threads, managed through Tokio's spawn_blocking, and is also discardable.

Image Preview​

Yazi not only has built-in code highlighting but also includes image decoding and downscaling - there is likely nothing faster than having it directly built-in. It is also distributed among multiple threads and is discardable.

Besides being fast, Yazi's built-in Kitty graphics protocol, Inline images protocol, and Sixel graphics format allow Yazi to finely control when to display or hide images.

This ensures that in Yazi, there won't be issues, like images stacking on top of each other, or image escape code breaking the entire screen, when navigating through images quickly, as stdout is locked while outputting these escape codes. This locking happens after all image data is prepared, so it has no impact on performance.

Yazi even supports partially erasing content in preview images, which is useful for pop-up components (Input, Select). The image won't overlap the input, and when the pop-up disappears, Yazi redraws the image to complete the erased portion automatically.

Async Task Scheduling​

In Yazi, tasks are prioritized based on their severity automatically. Yazi categorizes tasks into two types:

  • Macro tasks: Large and heavy tasks, such as copying large files, typically taking a long time to complete.
  • Micro tasks: Small and urgent tasks, such as fetching file mime-type, pre-loading images, calculating directory size, and so on.

This is similar to having big and small cores in a CPU; when the big cores are idle, they help with the micro tasks. Yazi defaults to starting 5 micro workers and 10 macro workers, and these numbers can be configured by the user!

In addition, Yazi introduces a priority scheduling mechanism. Each task has 3 priority levels: low, normal, and high. High-priority tasks can preempt low-priority tasks, applying to both micro and macro tasks. This increases task concurrency, slowing down HOL blocking caused by queuing execution of sudden requests.

For complex tasks like file copying, a combination of micro and macro approaches is employed. Micro is used to gather a list of all files to be copied recursively, allowing users to see the number of tasks and their sizes in advance. Macro, on the other hand, handles the actual copying process.

The advantage of task scheduling extends beyond providing ample concurrency for I/O and CPU resources; it also indirectly mitigates the depletion of system resources (such as file handles and CPU) due to sudden task surges.

Other optimizations​

The above optimizations are the most noticeable to users, but behind the scenes, Yazi has also done many other optimizations. Include but are not limited to:

  • The re-implemented highly optimized natural sorting algorithm is ~6 times faster than the natord that eza uses in case-insensitive sorting.
  • Yazi caches the directory state that has already been read, avoiding any unnecessary IO operations.
  • When a file in a directory changes, it only updates the changed files rather than re-reading the entire directory list.
  • Merges multiple renders triggered by multiple commands into a single render, avoiding unnecessary CPU consumption.
  • Frequent updates to components, such as progress bars, are rendered independently, which is no cost compared to a complete render.
  • The entire plugin system is designed with an asynchronous-first philosophy to avoid blocking the main thread with time-consuming tasks.

Installation

To use Yazi, you must have the following prerequisites installed:

  • file (for file type detection)

Yazi can be optionally extended with other command line tools to enable additional features.

If the functions are not working properly, please try upgrading them to the latest version.

Arch Linux​

sudo pacman -S yazi ffmpeg p7zip jq poppler fd ripgrep fzf zoxide imagemagick

If you want to use the latest Git version, you can install it from AUR or Arch Linux CN:

paru -S yazi-git ffmpeg p7zip jq poppler fd ripgrep fzf zoxide imagemagick

You can also install the official nightly release binary from AUR, which is built from the latest code within the past 6 hours:

paru -S yazi-nightly-bin ffmpeg p7zip jq poppler fd ripgrep fzf zoxide imagemagick

Nix​

A Nix package for Yazi is available.

# NixOS:
nix-env -iA nixos.yazi

# Not NixOS:
nix-env -iA nixpkgs.yazi

Or add the following to your configuration:

# configuration.nix
environment.systemPackages = with pkgs; [
	yazi
];

Nix flakes​

The upstream repository provides a flake so that Nix users can easily keep up with the bleeding edge. A basic flake.nix setup to get you started:

{
	inputs = {
		nixpkgs.url = "github:NixOS/nixpkgs/nixos-23.11";

		home-manager = {
			url = "github:nix-community/home-manager/release-23.11";
			inputs.nixpkgs.follows = "nixpkgs";
		};

		yazi.url = "github:sxyazi/yazi";
	};

	outputs = { nixpkgs, home-manager, yazi, ... }: {
		# To install Yazi system-wide:
		nixosConfigurations = {
			nixos = nixpkgs.lib.nixosSystem {
				modules = [
					({ pkgs, ... }: {
						environment.systemPackages = [ yazi.packages.${pkgs.system}.default ];
					})
				];
			};
		};

		# To install it for a specific user:
		homeConfigurations = {
			"alice@nixos" = home-manager.lib.homeManagerConfiguration {
				pkgs = nixpkgs.legacyPackages.x86_64-linux;
				modules = [
					({ pkgs, ... }: {
						home.packages = [ yazi.packages.${pkgs.system}.default ];
					})
				];
			};
		};
	};
}

If you want to override the package inside nixpkgs with the one from the flake, you can use overlays:

nixpkgs.overlays = [ yazi.overlays.default ];

A module is also available for both NixOS and home-manager:

programs.yazi = {
	enable = true;
	package = yazi.packages.${pkgs.system}.default; # if you use overlays, you can omit this
};

Cache​

Pre-built artifacts are served at https://yazi.cachix.org, so that Nix users don't have to build Yazi on their machine. You can make use of it by adding the following options to nix.settings, either in your NixOS or home-manager configuration:

extra-substituters = [ "https://yazi.cachix.org" ];
extra-trusted-public-keys = [ "yazi.cachix.org-1:Dcdz63NZKfvUCbDGngQDAZq6kOroIrFoyO064uvLh8k=" ];

Note that the cache will only be applied after you rebuild your Nix config. If you want to ensure that the cache gets applied right away, add the options above to your flake's nixConfig attribute.

If you're having any problems, refer to this entry from Cachix's FAQ.

Homebrew​

First, make sure that Homebrew is fully up-to-date with brew update.

Then you can install Yazi (and the optional dependencies):

brew install yazi ffmpeg sevenzip jq poppler fd ripgrep fzf zoxide imagemagick font-symbols-only-nerd-font

If you prefer to use the most recent code, use the --HEAD flag when installing Yazi.

brew install yazi --HEAD

MacPorts​

sudo port install yazi ffmpeg 7zip jq poppler fd ripgrep fzf zoxide ImageMagick

NetBSD​

pkgin install yazi ffmpeg7 p7zip jq poppler fd ripgrep fzf zoxide ImageMagick

Windows​

Yazi relies on file(1) to detect the mime-type of the file, and the easiest and most reliable way to get it on Windows is to install Git for Windows and use the file.exe that comes with it.

  1. Install Git for Windows by running the official installer, or through your package manager of choice.
  2. To allow Yazi to use file(1), add <Git_Installed_Directory>\usr\bin\file.exe to your YAZI_FILE_ONE environment variable, which differs depending on how you installed Git: If you installed Git with the installer, it would be C:\Program Files\Git\usr\bin\file.exe. If you installed Git with Scoop, it would be C:\Users\<Username>\scoop\apps\git\current\usr\bin\file.exe.
  3. Restart your terminal.

This is the ONLY way we recommend. We do not recommend install file via Scoop or Chocolatey, since they cannot handle Unicode filenames (such as oliver-sjöström.jpg) properly and lack some required parameters.

Most users already have Git installed, and Yazi is also hosted via Git, so this usually isn't an issue. But if you really don't have/want to install it, the mime-ext.yazi plugin can help, which use a extension database instead of relying on the file(1) binary.

Install with Scoop​

scoop install yazi
# Install the optional dependencies (recommended):
scoop install ffmpeg 7zip jq poppler fd ripgrep fzf zoxide imagemagick

Install with WinGet​

winget install sxyazi.yazi
# Install the optional dependencies (recommended):
winget install Gyan.FFmpeg 7zip.7zip jqlang.jq sharkdp.fd BurntSushi.ripgrep.MSVC junegunn.fzf ajeetdsouza.zoxide ImageMagick.ImageMagick

Poppler is not yet on WinGet, install with Scoop or manually download from poppler-windows.

AOSC OS​

sudo oma install yazi

x-cmd​

x env use yazi ffmpeg fzf 7za jq fd rg zoxide

Official binaries​

You can download the latest official binaries from GitHub Releases: https://github.com/sxyazi/yazi/releases

On this page, we offer GNU/Musl builds to meet the needs of different users.

This page also includes a nightly release, which is built from the latest code within the past 6 hours.

Cargo​

Setup the latest stable Rust toolchain via rustup:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
rustup update

Now you can install Yazi from crates.io:

cargo install --locked yazi-fm yazi-cli

Or install the latest Git version:

cargo install --locked --git https://github.com/sxyazi/yazi.git yazi-fm yazi-cli

If it fails to build, please check if make and gcc is installed on your system.

Build from source​

Setup the latest stable Rust toolchain via rustup:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
rustup update

Clone the repository and build Yazi:

git clone https://github.com/sxyazi/yazi.git
cd yazi
cargo build --release --locked

Then, you can run:

./target/release/yazi

If it fails to build, please check if make and gcc is installed on your system.

Get start

Once you've installed Yazi, start the program with:

yazi

Press q to quit and ~ to open the help menu.