Workaround Fuse-T behavior on macOS

[LoganDark]

With Fuse-T, readdir seems to also stat every directory entry, causing tup to assume false input dependencies; potentially due to Fuse-T's usage of NFS. Due to this, while bootstrapping succeeds, Fuse-T build/tup will report errors like this on my machine:

* 43) src/luabuiltin: ../lua/lua xxd.lua builtin.lua luabuiltin.h
 *** tup messages ***
tup error: Missing input dependency - a file was read from, and was not specified as an input link for the command. This is an issue because the file was created from another command, and without the input link the commands may execute out of order. You should add this file as an input, since it is possible this could randomly break in the future.
 - [2417] src/lua/.gitignore
 *** Command ran successfully, but failed due to errors processing input dependencies.
 42) [5.027s] src/sqlite3: CC sqlite3.c
 [                             ETA~=5s  Remaining=42 Active=0                               ]  53%
 *** tup: 1 job failed.

(full transcript)

Initializing .tup in /Users/LoganDark/Documents/Projects/tup
.tup repository initialized: .tup/db
[ tup ] [0.000s] Scanning filesystem...
[ tup ] [0.020s] Reading in new environment variables...
[ tup ] [0.021s] Parsing Tupfiles...
 58) [0.006s] src/lua
 57) [0.005s] src/luabuiltin
 56) [0.007s] src/tup
 55) [0.005s] src/tup/flock
 54) [0.005s] src/tup/tup
 53) [0.005s] src/tup/monitor
 52) [0.004s] src/tup/server
 51) [0.004s] src/inih
 50) [0.003s] src/compat
 49) [0.003s] src/sqlite3
 48) [0.006s] .
 47) [0.002s] test
 46) [0.001s] test/make_v_tup
 45) [0.001s] docs
 44) [0.002s] docs/html
 43) [0.003s] docs/html/pub
 42) [0.002s] docs/html/pub/win32
 41) [0.001s] contrib
 40) [0.001s] contrib/debian
 39) [0.001s] contrib/debian/source
 38) [0.001s] contrib/syntax
 37) [0.001s] libfuse
 36) [0.001s] libfuse/example
 35) [0.001s] libfuse/include
 34) [0.001s] libfuse/include/old
 33) [0.001s] libfuse/lib
 32) [0.001s] libfuse/lib/modules
 31) [0.001s] libfuse/doc
 30) [0.001s] .jj
 29) [0.001s] .jj/working_copy
 28) [0.001s] .jj/repo
 27) [0.001s] .jj/repo/op_store
 26) [0.002s] .jj/repo/op_store/operations
 25) [0.002s] .jj/repo/op_store/views
 24) [0.002s] .jj/repo/workspace_store
 23) [0.002s] .jj/repo/op_heads
 22) [0.002s] .jj/repo/op_heads/heads
 21) [0.001s] .jj/repo/index
 20) [0.002s] .jj/repo/index/op_links
 19) [0.002s] .jj/repo/index/changed_paths
 18) [0.002s] .jj/repo/index/segments
 17) [0.001s] .jj/repo/submodule_store
 16) [0.001s] .jj/repo/store
 15) [0.002s] .jj/repo/store/extra
 14) [0.002s] .jj/repo/store/extra/heads
 13) [0.001s] .github
 12) [0.001s] .github/workflows
 11) [0.003s] lib32
 10) [0.001s] local-build-configs
  9) [0.001s] build
  8) [0.001s] build/luabuiltin
  7) [0.001s] src
  6) [0.003s] src/compat/win32
  5) [0.004s] src/compat/win32/detect
  4) [0.002s] src/compat/win32/sys
  3) [0.003s] src/dllinject
  2) [0.001s] src/bsd
  1) [0.003s] src/pcre
  0) [0.003s] src/ldpreload
 [                  ETA~=<1s Remaining=0                     ] 100%
[ tup ] [0.195s] No files to delete.
[ tup ] [0.195s] Generating .gitignore files...
[ tup ] [0.199s] Executing Commands...
 89) [0.124s] src/lua: CC lctype.c
 88) [0.151s] src/lua: CC linit.c
 87) [0.155s] src/lua: CC ldblib.c
 86) [0.163s] src/lua: CC lcorolib.c
 85) [0.180s] src/lua: CC ldump.c
 84) [0.192s] src/lua: CC lfunc.c
 83) [0.202s] src/lua: CC lmathlib.c
 82) [0.211s] src/lua: CC liolib.c
 81) [0.212s] src/lua: CC lauxlib.c
 80) [0.214s] src/lua: CC lbaselib.c
 79) [0.240s] src/lua: CC llex.c
 78) [0.245s] src/lua: CC lapi.c
 77) [0.246s] src/lua: CC ldo.c
 76) [0.261s] src/lua: CC ldebug.c
 75) [0.276s] src/lua: CC lcode.c
 74) [0.120s] src/lua: CC lopcodes.c
 73) [0.295s] src/lua: CC lgc.c
 72) [0.184s] src/lua: CC lmem.c
 71) [0.163s] src/lua: CC loadlib.c
 70) [0.028s] CP src/tup/vardict.h -> tup_client.h
 69) [0.171s] src/lua: CC loslib.c
 68) [0.223s] src/lua: CC lobject.c
 67) [0.144s] src/lua: CC ltablib.c
 66) [0.138s] src/lua: CC lutf8lib.c
 65) [0.194s] src/lua: CC lstring.c
 64) [0.110s] src/tup/flock: CC fcntl.c
 63) [0.225s] src/lua: CC lstate.c
 62) [0.190s] src/lua: CC ltm.c
 61) [0.153s] src/lua: CC lzio.c
 60) [0.149s] src/lua: CC lua.c
 59) [0.233s] src/lua: CC ltable.c
 58) [0.096s] src/tup/monitor: CC null.c
 57) [0.203s] src/lua: CC lundump.c
 56) [0.238s] src/lua: CC lstrlib.c
 55) [0.080s] src/compat: CC dummy.c
 54) [0.102s] src/inih: CC ini.c
 53) [0.089s] src/compat: CC clearenv.c
 52) [0.339s] src/lua: CC lparser.c
 51) [0.162s] src/tup/server: CC symlink.c
 50) [0.201s] src/tup/server: CC master_fork.c
 49) [0.227s] src/tup/server: CC fuse_server.c
 48) [0.222s] src/tup/server: CC fuse_fs.c
 47) [0.271s] src/tup/tup: CC main.c
 46) [0.351s] src/lua: CC lvm.c
 45) [0.072s] src/lua: AR liblua.a
 44) [0.081s] src/lua: LINK lua
* 43) src/luabuiltin: ../lua/lua xxd.lua builtin.lua luabuiltin.h
 *** tup messages ***
tup error: Missing input dependency - a file was read from, and was not specified as an input link for the command. This is an issue because the file was created from another command, and without the input link the commands may execute out of order. You should add this file as an input, since it is possible this could randomly break in the future.
 - [2417] src/lua/.gitignore
 *** Command ran successfully, but failed due to errors processing input dependencies.
 42) [5.027s] src/sqlite3: CC sqlite3.c
 [                             ETA~=5s  Remaining=42 Active=0                               ]  53%
 *** tup: 1 job failed.

With this PR, on macOS when Fuse-T is likely in use, we now test at startup for this behavior by listing a virtual directory and watching for stat of a virtual sentinel file. We always ensure the behavior at runtime before enabling the workaround, because there could be forks of Fuse-T that fix it or other implementations that never had it (such as macFUSE). The virtual directory itself deactivates after being listed once. Then, if we observe a stat on the sentinel, we assume it was performed automatically as a consequence of the readdir, and enable the workaround. We then stat a virtual "done" file to conclude the probe, deactivating both virtual files to prevent the workaround from being enabled when the behavior is not present. Thus the workaround is only enabled when a stat reaches the sentinel following the readdir but before our stat to the done file.

When the workaround is enabled, readdir will record each result into a cache that tells getattr to ignore one next stat. By ignoring the spurious call to getattr, we avoid assuming a corresponding spurious access, because in this case, the first stat for a child following its parent directory's listing is from the filesystem layer itself, not the running task.

This is my first contribution, so let me know if I'll need to sign a CLA.

[LoganDark]

Discovered some test failures still caused by NFS silliness -- debugging now

[gittup]

This looks like a reasonable approach, I'm sure it was not fun to debug.

Any objection to squashing the commits down? It looks like developmental commits rather than feature-complete independent changes. Especially if the intermediate commits aren't fully passing the test cases.

[LoganDark]

Still working on the test cases, since some of them are still failing due to other NFS weirdness. I will keep the PR to one commit once it is ready :)

[LoganDark]

It looks like unfortunately macOS also has a few more NFS behaviors that need to be addressed (like crawling upwards searching for .app bundles), so this will probably take a while to be more complete. Thank you for pointing me at the tests!

[LoganDark]

I failed to make macOS pass tests without a reference, so I'm instrumenting tup and recording its exact behavior on Linux so that I can see where macOS deviates. Might take a couple more days to reach 100% conformance.

[LoganDark]

I found a lot more things that were wrong with Fuse-T mounts on macOS. I instrumented tup to record its exact behavior in a Linux container, then ran the same tests on macOS and studied for deviations. All but five tests now pass on my machine:

 *** t3083-extra-outputs-bang3.sh failed
 *** t4201-ccache2.sh failed
 *** t5074-tup-dies.sh failed
 *** t5103-python-sh.sh failed
 *** t8005-variant2.sh failed

with four of the failures being just because I didn't install ccache/python/etc, and t5074-tup-dies.sh being for another reason but I'll open another PR about that in a bit.

Maybe try to test on a machine that uses macFUSE because probably none of these workarounds should be used in that case.

[LoganDark]

Fixed to also ignore ._ files which were causing errors like this after a reboot:

 91) src/compat: CC dummy.c                                                                                                                                                       
 *** tup messages ***
tup error: Unspecified output files - A command is writing to files that you didn't specify in the Tupfile. You should add them so tup knows what to expect.
 -- Unspecified output: src/compat/._dummy.o
 *** Command failed due to errors processing the output dependencies.
* 90) src/compat: CC clearenv.c                                                                                                                                                    
 *** tup messages ***
tup error: Unspecified output files - A command is writing to files that you didn't specify in the Tupfile. You should add them so tup knows what to expect.
 -- Unspecified output: src/compat/._clearenv.o
 *** Command failed due to errors processing the output dependencies.
* 89) src/inih: CC ini.c                                                                                                                                                           
 *** tup messages ***
tup error: Unspecified output files - A command is writing to files that you didn't specify in the Tupfile. You should add them so tup knows what to expect.
 -- Unspecified output: src/inih/._ini.o
 *** Command failed due to errors processing the output dependencies.
* 88) src/lua: CC lctype.c                                                                                                                                                         
 *** tup messages ***
tup error: Unspecified output files - A command is writing to files that you didn't specify in the Tupfile. You should add them so tup knows what to expect.
 -- Unspecified output: src/lua/._lctype.o
 *** Command failed due to errors processing the output dependencies.
* 87) src/lua: CC linit.c                                                                                                                                                          
 *** tup messages ***
tup error: Unspecified output files - A command is writing to files that you didn't specify in the Tupfile. You should add them so tup knows what to expect.
 -- Unspecified output: src/lua/._linit.o
 *** Command failed due to errors processing the output dependencies.
* 86) src/lua: CC lcorolib.c                                                                                                                                                       
 *** tup messages ***
tup error: Unspecified output files - A command is writing to files that you didn't specify in the Tupfile. You should add them so tup knows what to expect.
 -- Unspecified output: src/lua/._lcorolib.o
 *** Command failed due to errors processing the output dependencies.
* 85) src/lua: CC ldblib.c                                                                                                                                                         
 *** tup messages ***
tup error: Unspecified output files - A command is writing to files that you didn't specify in the Tupfile. You should add them so tup knows what to expect.
 -- Unspecified output: src/lua/._ldblib.o
 *** Command failed due to errors processing the output dependencies.
* 84) src/lua: CC ldump.c                                                                                                                                                          
 *** tup messages ***
tup error: Unspecified output files - A command is writing to files that you didn't specify in the Tupfile. You should add them so tup knows what to expect.
 -- Unspecified output: src/lua/._ldump.o
 *** Command failed due to errors processing the output dependencies.
* 83) src/lua: CC lauxlib.c                                                                                                                                                        
 *** tup messages ***
tup error: Unspecified output files - A command is writing to files that you didn't specify in the Tupfile. You should add them so tup knows what to expect.
 -- Unspecified output: src/lua/._lauxlib.o
 *** Command failed due to errors processing the output dependencies.
* 82) src/lua: CC lfunc.c                                                                                                                                                          
 *** tup messages ***
tup error: Unspecified output files - A command is writing to files that you didn't specify in the Tupfile. You should add them so tup knows what to expect.
 -- Unspecified output: src/lua/._lfunc.o
 *** Command failed due to errors processing the output dependencies.
* 81) src/lua: CC lbaselib.c                                                                                                                                                       
 *** tup messages ***
tup error: Unspecified output files - A command is writing to files that you didn't specify in the Tupfile. You should add them so tup knows what to expect.
 -- Unspecified output: src/lua/._lbaselib.o
 *** Command failed due to errors processing the output dependencies.
* 80) src/lua: CC ldo.c                                                                                                                                                            
 *** tup messages ***
tup error: Unspecified output files - A command is writing to files that you didn't specify in the Tupfile. You should add them so tup knows what to expect.
 -- Unspecified output: src/lua/._ldo.o
 *** Command failed due to errors processing the output dependencies.
* 79) src/lua: CC ldebug.c                                                                                                                                                         
 *** tup messages ***
tup error: Unspecified output files - A command is writing to files that you didn't specify in the Tupfile. You should add them so tup knows what to expect.
 -- Unspecified output: src/lua/._ldebug.o
 *** Command failed due to errors processing the output dependencies.
* 78) src/lua: CC lapi.c                                                                                                                                                           
 *** tup messages ***
tup error: Unspecified output files - A command is writing to files that you didn't specify in the Tupfile. You should add them so tup knows what to expect.
 -- Unspecified output: src/lua/._lapi.o
 *** Command failed due to errors processing the output dependencies.
* 77) src/lua: CC lcode.c                                                                                                                                                          
 *** tup messages ***
tup error: Unspecified output files - A command is writing to files that you didn't specify in the Tupfile. You should add them so tup knows what to expect.
 -- Unspecified output: src/lua/._lcode.o
 *** Command failed due to errors processing the output dependencies.
* 76) src/lua: CC lgc.c                                                                                                                                                            
 *** tup messages ***
tup error: Unspecified output files - A command is writing to files that you didn't specify in the Tupfile. You should add them so tup knows what to expect.
 -- Unspecified output: src/lua/._lgc.o
 *** Command failed due to errors processing the output dependencies.
 [                               ETA=4s  Remaining=76 Active=0                                ]   7%