fastwalk

Golang 的快速并行目录遍历。

fastwalk 包提供了一个快速并行版本的 filepath.WalkDir， 在 macOS 上速度提升约 2.5 倍，Linux 上约 4 倍，Windows 上约 6 倍， 内存分配减少 50%，且内存分配次数减少 25%。 此外，它在各操作系统上比 godirwalk 快约 4-5 倍。

灵感来源并基于 golang.org/x/tools/internal/fastwalk。

特性

• 快速：多个 goroutine 并发访问文件系统并调用

• 安全的符号链接遍历（Config.Follow）
• 行为和回调签名与 filepath.WalkDir 一致
• 提供包装函数以忽略重复的文件和目录：

• 在 macOS、Linux 和 Windows 上经过广泛测试

用法

用法与 filepath.WalkDir 相同， 但传递给 fastwalk.Walk 的 walkFn 参数必须是并发安全的。

示例可在 examples 目录中找到。

下面的示例是一个非常简单版本的POSIX find 工具：

// fwfind is a an example program that is similar to POSIX find,
// but faster and worse (it's an example).
package main
import (
	"flag"
	"fmt"
	"io/fs"
	"os"
	"path/filepath"
	"github.com/charlievieth/fastwalk"
)
const usageMsg = Usage: %[1]s [-L] [-name] [PATH...]:
%[1]s is a poor replacement for the POSIX find utility
func main() {
	flag.Usage = func() {
		fmt.Fprintf(os.Stdout, usageMsg, filepath.Base(os.Args[0]))
		flag.PrintDefaults()
	}
	pattern := flag.String("name", "", "Pattern to match file names against.")
	followLinks := flag.Bool("L", false, "Follow symbolic links")
	flag.Parse()
	// If no paths are provided default to the current directory: "."
	args := flag.Args()
	if len(args) == 0 {
		args = append(args, ".")
	}
	// Follow links if the "-L" flag is provided
	conf := fastwalk.Config{
		Follow: *followLinks,
	}
	walkFn := func(path string, d fs.DirEntry, err error) error {
		if err != nil {
			fmt.Fprintf(os.Stderr, "%s: %v\n", path, err)
			return nil // returning the error stops iteration
		}
		if *pattern != "" {
			if ok, err := filepath.Match(*pattern, d.Name()); !ok {
				// invalid pattern (err != nil) or name does not match
				return err
			}
		}
		_, err = fmt.Println(path)
		return err
	}
	for _, root := range args {
		if err := fastwalk.Walk(&conf, root, walkFn); err != nil {
			fmt.Fprintf(os.Stderr, "%s: %v\n", root, err)
			os.Exit(1)
		}
	}
}

基准测试

基准测试使用 go1.17.6 创建，可以通过 bench_comp make 目标生成：

$ make bench_comp

达尔文

硬件：

goos: darwin
goarch: arm64
cpu: Apple M1 Max

#### filepath.WalkDir 与 fastwalk.Walk():

              filepath       fastwalk       delta
time/op       27.9ms ± 1%    13.0ms ± 1%    -53.33%
alloc/op      4.33MB ± 0%    2.14MB ± 0%    -50.55%
allocs/op     50.9k ± 0%     37.7k ± 0%     -26.01%

#### godirwalk.Walk() 与 fastwalk.Walk()：

              godirwalk      fastwalk       delta
time/op       58.5ms ± 3%    18.0ms ± 2%    -69.30%
alloc/op      25.3MB ± 0%    2.1MB ± 0%     -91.55%
allocs/op     57.6k ± 0%     37.7k ± 0%     -34.59%

Linux

硬件：

goos: linux
goarch: amd64
cpu: Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz
drive: Samsung SSD 970 PRO 1TB

#### filepath.WalkDir 与 fastwalk.Walk() 的比较：

              filepath       fastwalk       delta
time/op       10.1ms ± 2%    2.8ms ± 2%     -72.83%
alloc/op      2.44MB ± 0%    1.70MB ± 0%    -30.46%
allocs/op     47.2k ± 0%     36.9k ± 0%     -21.80%

#### godirwalk.Walk() 与 fastwalk.Walk():

              filepath       fastwalk       delta
time/op       13.7ms ±16%    2.8ms ± 2%     -79.88%
alloc/op      7.48MB ± 0%    1.70MB ± 0%    -77.34%
allocs/op     53.8k ± 0%     36.9k ± 0%     -31.38%

Windows

硬件：

goos: windows
goarch: amd64
pkg: github.com/charlievieth/fastwalk
cpu: Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz

#### filepath.WalkDir 与 fastwalk.Walk() 比较：

              filepath       fastwalk       delta
time/op       88.0ms ± 1%    14.6ms ± 1%    -83.47%
alloc/op      5.68MB ± 0%    6.76MB ± 0%    +19.01%
allocs/op     69.6k ± 0%     90.4k ± 0%     +29.87%

              filepath       fastwalk       delta
time/op       87.4ms ± 1%    14.6ms ± 1%    -83.34%
alloc/op      6.14MB ± 0%    6.76MB ± 0%    +10.24%
allocs/op     100k ± 0%      90k ± 0%       -9.59%

--- Tranlated By Open Ai Tx | Last indexed: 2026-05-17 ---