fix: stalling on large file sets #18

brianmcgee · 2024-01-04T21:05:32Z

brianmcgee commented

2024-01-04 21:05:32 +00:00

When running against nixpkgs we were stalling. This was due to a long running read tx which was preventing any writes.

This breaks up reading the cache when walking the filesystem into many smaller read txs.

On my laptop I'm now getting the following with the echo sample:

# fresh cache

❯ nix run .# -- -c --config-file ./test/echo.toml --tree-root ../../../github.com/nixos/nixpkgs   
38825 files changed in 320.655826ms

# hot cache

❯ nix run .# -- --config-file ./test/echo.toml --tree-root ../../../github.com/nixos/nixpkgs  
0 files changed in 252.920853ms%

Signed-off-by: Brian McGee brian@bmcgee.ie

When running against nixpkgs we were stalling. This was due to a long running read tx which was preventing any writes. This breaks up reading the cache when walking the filesystem into many smaller read txs. On my laptop I'm now getting the following with the echo sample: ```console # fresh cache ❯ nix run .# -- -c --config-file ./test/echo.toml --tree-root ../../../github.com/nixos/nixpkgs 38825 files changed in 320.655826ms # hot cache ❯ nix run .# -- --config-file ./test/echo.toml --tree-root ../../../github.com/nixos/nixpkgs 0 files changed in 252.920853ms% ``` Signed-off-by: Brian McGee <brian@bmcgee.ie>

brianmcgee added 1 commit 2024-01-04 21:05:32 +00:00

fix: stalling on large file sets 5b91f74f89

When running against nixpkgs we were stalling. This was due to a long running read tx which was preventing any writes.

This breaks up reading the cache when walking the filesystem into many smaller read txs.

Signed-off-by: Brian McGee <brian@bmcgee.ie>

zimbatm was assigned by brianmcgee

2024-01-04 21:05:41 +00:00

zimbatm was unassigned by brianmcgee

2024-01-04 21:45:18 +00:00

brianmcgee requested review from zimbatm 2024-01-04 21:45:22 +00:00

zimbatm approved these changes 2024-01-07 18:53:28 +00:00

brianmcgee merged commit 55ca4468cc into main

2024-01-07 18:57:52 +00:00

brianmcgee referenced this issue from a commit

2024-01-07 18:57:54 +00:00

fix: stalling on large file sets (#18)

This repo is archived. You cannot comment on pull requests.