systemfonts::system_fonts crashes (segfault) when used in future

[stefanoborini]

This code produces a hard segfault:

future::plan(future::multiprocess)

fut <- future::future({
  systemfonts::system_fonts()
})

r <- future::value(fut)
print(r)

[1] "Using environment default"

 *** caught segfault ***
address 0x110, cause 'memory not mapped'

Traceback:
 1: systemfonts::system_fonts()
 2: eval(quote({    systemfonts::system_fonts()}), new.env())
 3: eval(quote({    systemfonts::system_fonts()}), new.env())
 4: eval(expr, p)
 5: eval(expr, p)
 6: eval.parent(substitute(eval(quote(expr), envir)))
 7: local({    systemfonts::system_fonts()})
 8: doTryCatch(return(expr), name, parentenv, handler)
 9: tryCatchOne(expr, names, parentenv, handlers[[1L]])
10: tryCatchList(expr, classes, parentenv, handlers)
11: tryCatch({    ...future.value <- local({        systemfonts::system_fonts()    })    future::FutureResult(value = ...future.value, version = "1.8")}, error = function(cond) {    calls <- sys.calls()    structure(list(value = NULL, value2 = NA, condition = cond,         calls = calls, version = "1.8"), class = "FutureResult")}, finally = {    {        {            {                options(mc.cores = ...future.mc.cores.old)            }            future::plan(list(function (expr, envir = parent.frame(),                 substitute = TRUE, lazy = FALSE, seed = NULL,                 globals = TRUE, workers = availableCores(), gc = FALSE,                 earlySignal = FALSE, label = NULL, ...)             {                if (substitute)                   expr <- substitute(expr)                fun <- if (supportsMulticore())                   multicore                else multisession                fun(expr = expr, envir = envir, substitute = FALSE,                   lazy = lazy, seed = seed, globals = globals,                   workers = workers, gc = gc, earlySignal = earlySignal,                   label = label, ...)            }), .cleanup = FALSE, .init = FALSE)        }        options(...future.oldOptions)    }})
12: eval(expr, env)

R 3.6.0, systemfonts 0.2.3 on macos.

lldb backtrace

* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x110)
  * frame #0: 0x00007fff6f5a2e47 libdispatch.dylib`_dispatch_mgr_queue_push + 41
    frame #1: 0x00007fff6f59ebab libdispatch.dylib`_dispatch_lane_resume_activate + 58
    frame #2: 0x00007fff44919d9a CarbonCore`connectToCoreServicesD() + 245
    frame #3: 0x00007fff44919c74 CarbonCore`getStatus() + 24
    frame #4: 0x00007fff44919bef CarbonCore`scCreateSystemServiceVersion + 49
    frame #5: 0x00007fff44919964 CarbonCore`FileIDTreeGetCachedPort + 213
    frame #6: 0x00007fff449197d0 CarbonCore`FSNodeStorageGetAndLockCurrentUniverse + 83
    frame #7: 0x00007fff4491965a CarbonCore`FileIDTreeGetAndLockVolumeEntryForDeviceID + 38
    frame #8: 0x00007fff449195b7 CarbonCore`FSMount::FSMount(unsigned int, FSMountNumberType, int*, unsigned int const*) + 75
    frame #9: 0x00007fff44919531 CarbonCore`FSMountPrepare + 69
    frame #10: 0x00007fff5784aa30 CoreServicesInternal`MountInfoPrepare(void***, unsigned int, int, void*, unsigned int const*, __CFURL const*, __CFError**) + 43
    frame #11: 0x00007fff5784a428 CoreServicesInternal`parseAttributeBuffer(__CFAllocator const*, unsigned char const*, unsigned char, attrlist const*, void const*, void**, _FileAttributes*, unsigned int*) + 3209
    frame #12: 0x00007fff57849491 CoreServicesInternal`corePropertyProviderPrepareValues(__CFURL const*, __FileCache*, __CFString const* const*, void const**, long, void const*, __CFError**) + 834
    frame #13: 0x00007fff578490ee CoreServicesInternal`prepareValuesForBitmap(__CFURL const*, __FileCache*, _FilePropertyBitmap*, __CFError**) + 360
    frame #14: 0x00007fff5784ca0f CoreServicesInternal`_FSURLCopyResourcePropertyValuesAndFlags + 581
    frame #15: 0x00007fff4368f24b CoreFoundation`_CFURLCopyResourcePropertyValuesAndFlags + 127
    frame #16: 0x00007fff41bb6236 libFontParser.dylib`FPPathGetCatalogValues(char const*, FInfo*, unsigned long long*, unsigned long long*) + 187
    frame #17: 0x00007fff41bb3ec5 libFontParser.dylib`TFont::CreateFontEntities(char const*, bool, bool&, short, char const*, bool) + 85
    frame #18: 0x00007fff41bb664a libFontParser.dylib`TFont::CreateFontEntitiesForFile(char const*, bool, bool, short, char const*) + 264
    frame #19: 0x00007fff41b6649e libFontParser.dylib`FPFontCreateFontsWithPath + 158
    frame #20: 0x00007fff43a983b7 CoreGraphics`create_private_data_array_with_path + 29
    frame #21: 0x00007fff43a980bf CoreGraphics`CGFontCreateFontsWithPath + 26
    frame #22: 0x00007fff43a97ce9 CoreGraphics`CGFontCreateFontsWithURL + 345
    frame #23: 0x00007fff4528642b CoreText`CreateFontsWithURL(__CFURL const*, bool) + 199
    frame #24: 0x00007fff45368a4c CoreText`CTFontManagerCreateFontDescriptorsFromURL + 37
    frame #25: 0x0000000107ab625a systemfonts.so`addFontIndex(FontDescriptor*) + 378
    frame #26: 0x0000000107ab6c80 systemfonts.so`createFontDescriptor(__CTFontDescriptor const*) + 992
    frame #27: 0x0000000107ab72a8 systemfonts.so`getAvailableFonts() + 520
    frame #28: 0x0000000107aa6103 systemfonts.so`system_fonts() + 467

[stefanoborini]

macos. 10.14.6

[thomasp85]

Hmm... it seems to be happening deep inside CoreText so I'm unsure how to approach this. Maybe CoreText is simply not safe to run from forked processes. @HenrikBengtsson would you know anything about this?

[stefanoborini]

Just for the record, it does not seem to happen on linux. Seems to be a mac specific problem indeed.

[stefanoborini]

It also does not crash on mac if instead of multiprocess I use multisession. It does crash with multicore

[thomasp85]

Thanks for confirming that this is indeed something to do it's calling into CoreText from forked processes. Really not sure what to do about this 😕

[HenrikBengtsson]

Does the following also segfault on macOS?

> f <- parallel::mcparallel(systemfonts::system_fonts())
> v <- parallel::mccollect(f)

If so, that's a minimal reproducible example using forked processing. I don't have access to macOS, so I cannot test. It works on Ubuntu.

[stefanoborini]

@HenrikBengtsson Crashes immediately after the first line is executed

> f <- parallel::mcparallel(systemfonts::system_fonts())
> 
 *** caught segfault ***
address 0x110, cause 'memory not mapped'

Traceback:
 1: systemfonts::system_fonts()
 2: eval(expr, env)
 3: doTryCatch(return(expr), name, parentenv, handler)
 4: tryCatchOne(expr, names, parentenv, handlers[[1L]])
 5: tryCatchList(expr, classes, parentenv, handlers)
 6: tryCatch(expr, error = function(e) {    call <- conditionCall(e)    if (!is.null(call)) {        if (identical(call[[1L]], quote(doTryCatch)))             call <- sys.call(-4L)        dcall <- deparse(call)[1L]        prefix <- paste("Error in", dcall, ": ")        LONG <- 75L        sm <- strsplit(conditionMessage(e), "\n")[[1L]]        w <- 14L + nchar(dcall, type = "w") + nchar(sm[1L], type = "w")        if (is.na(w))             w <- 14L + nchar(dcall, type = "b") + nchar(sm[1L],                 type = "b")        if (w > LONG)             prefix <- paste0(prefix, "\n  ")    }    else prefix <- "Error : "    msg <- paste0(prefix, conditionMessage(e), "\n")    .Internal(seterrmessage(msg[1L]))    if (!silent && isTRUE(getOption("show.error.messages"))) {        cat(msg, file = outFile)        .Internal(printDeferredWarnings())    }    invisible(structure(msg, class = "try-error", condition = e))})
 7: try(eval(expr, env), silent = TRUE)
 8: sendMaster(try(eval(expr, env), silent = TRUE))
 9: parallel::mcparallel(systemfonts::system_fonts())
An irrecoverable exception occurred. R is aborting now ...

[HenrikBengtsson]

Interesting. Not that it would be a solution, but it would be interesting to know if preloading {systemfonts} as in:

v0 <- systemfonts::system_fonts()
f <- parallel::mcparallel(systemfonts::system_fonts())
v <- parallel::mccollect(f)

behaves differently, or possibly even work.

[stefanoborini]

@HenrikBengtsson yes, that works. Interesting!

[HenrikBengtsson]

Yes, surely interesting - and one of the obscure properties of forked parallel processing in R (which certainly is not stable).

If so, I guess that:

loadNamespace("systemfonts")
f <- parallel::mcparallel(systemfonts::system_fonts())
v <- parallel::mccollect(f)

might also work, because that also triggers:

> systemfonts:::.onLoad
function (...) 
{
    .Call("sf_init_c", asNamespace("systemfonts"))
    load_emoji_codes()
}

One hypothesis could be that you trigger the necessary initialization in the main R session, which is then inherited by the forked child processes, so it just works there. In contrast, if you initialize in the child process, then that is lost when the fork is terminated leaving you with an object with broken references/pointers. That's just a guess.

I would definitely not rely on this to work. It could just be that it happens to work in this simple example but if you "hit it hard enough" the problem will show up again elsewhere. The problem need to be well understood in order to consider it solved.

[thomasp85]

systemfonts does some caching which means that the forked process inherits the cache and never call into CoreText. My hypothesis is still that it is simply not safe to call into CoreText from a forked process. There is nothing in the systemfonts code that should illicit this behaviour (I think)

[stefanoborini]

For the record, I tried to workaround the issue in my application by adding the systemfonts call earlier in the calling process, and I get a different segfault later

 *** caught segfault ***
address 0x110, cause 'memory not mapped'

Traceback:
 1: m_str_extents_(x, fontname, fontsize, bold, italic, fontfile)
 2: m_str_extents(txt_data$txt, fontname = txt_data$font.family,     fontsize = fontsize, bold = txt_data$bold, italic = txt_data$italic)
 3: text_metric(x)
 4: optimal_sizes(x[[j]])
 5: dim_pretty(x, part = parts)
 6: flextable::autofit(.)
 7: function_list[[i]](value)
 8: freduce(value, `_function_list`)
 9: `_fseq`(`_lhs`)
10: eval(quote(`_fseq`(`_lhs`)), env, env)
11: eval(quote(`_fseq`(`_lhs`)), env, env)
12: withVisible(eval(quote(`_fseq`(`_lhs`)), env, env)

[HenrikBengtsson]

Thanks for these follow-ups - they're helpful for this package but also as a reference/example to others elsewhere.

@thomasp85, I don't know how it can be done, but if we could come up with a way for a package to detect when it runs in a forked process, then we might be able to give a run-time error rather than random crashes. The next level up would be to declare whether a package, or a specific function, can be forked or not, e.g. Forkable: false in DESCRIPTION.

[HenrikBengtsson]

Oh, I forgot that we might be able to detect whether we're in a fork or not via parallel:::isChild(), cf. futureverse/future#224 (comment)

[stefanoborini]

I don't think it's metadata worthy. I suspect it's a bug somewhere, possibly even in the Core libraries. It feels kind of weird that you can't perform the operation in a forked process. After all, all processes are forked.