Skip to content

cmd/go, cmd/link: support end-to-end single-object FullLTO - #347

Open
zhouguangyuan0718 wants to merge 8 commits into
go1.27.masterfrom
codex/single-object-full-lto-backend
Open

zhouguangyuan0718 wants to merge 8 commits into
go1.27.masterfrom
codex/single-object-full-lto-backend

Conversation

@zhouguangyuan0718

@zhouguangyuan0718 zhouguangyuan0718 commented Oct 21, 2026 •

Copy link
Copy Markdown

Compiler prerequisites #355 and #356 are merged. Based on 42dc37cc80160e792a97df75221af57263fc6a2f: package inputs have completed early IR preparation and LLVM pre-link O2 optimization.

Known blocker: the preceding Linux amd64/arm64 FullLTO CI run failed both cgo modes because runtime._cgo_clearenv has an indexed BSS definition in runtime and an initialized linkname definition in runtime/cgo. The current backend still rejects this as multiply defined; a minimal indexed-BSS/initialized-definition probe reproduces it. This PR is not ready to merge. Other integration cases in that run passed, but the subsequent standard-library step did not run.

Enable go build -lto=full end to end. Package bitcode is merged and optimized by LLVM, emitted as one GoObj, and consumed by the ordinary Go linker to produce executable and PIE binaries. The backend supports dynamic and static LLVM/plugin linkage and does not invoke external LLVM tools.

Content-addressed functions retain native hash classification. The compiler includes the compiling package in the existing inline-hash storage discriminator when creating LTO closures, including original closures and imported inline copies. LTO preserves those input names and rejects duplicate content-addressed identities; it no longer strips and regenerates their discriminators. GoObj strips the temporary discriminator before hashing and writing names. Identical machine code and relocations can merge across packages; different bodies stay distinct. No package-qualified output names or post-codegen name patching are needed.

The build driver selects bitcode compilation, includes LLVM/plugin identity in link cache keys, and leaves FIPS packages as ordinary objects. The linker rewrites archives only in a temporary directory, preserving assembly/native members and cached inputs. A package manifest restores imports, fingerprints, cgo directives, original symbol indices, logical compilation units and linkname origins over the single physical object. Mixed native objects, assembly references, initialization order and linkname checks remain supported.

LLVM handles native definition selection, reference remapping, ABI module flags, debug graphs and definition attachments. The Go adapter retains package/index provenance and Go-specific BSS requirements. Declared package names stay in native DWARF instead of being copied into the manifest. Linkname checks use original reference packages, allowing cross-package inlining without extra noinline restrictions. CPU/FMV preparation is performed by the compiler prerequisite; the LTO merger no longer constructs or runs a preparation pipeline, rejects unprepared compiler bitcode, and discards consumed CPU configuration rather than restoring main's copy. Saved input IR is written from the backend's existing parse in optimized/inputs, removing the extra Go-side parse. Compiler flag rejection remains covered by the compiler suite; the LTO suite tests build-driver restrictions.

Validation after rebasing and cleanup:

  • 238/238 plugin CTests pass, including 46 FullLTO cases and amd64/arm64 GoObj hash, stack-map and debug-graph checks.
  • Full integration suite passes locally on macOS arm64 (271.394s): cross-package optimization, cache reuse, content-addressed closures, assembly, mixed native objects, GC/stack growth, reflection, traceback, logical DWARF packages, linkname positive/negative cases, PIE and cgo callbacks with automatic/external linking.
  • New amd64/arm64 compiler regressions verify distinct original/imported closure names and deterministic recompilation; they fail with the preceding compiler. FileCheck verifies unchanged input names, ABI0/FMV suffixes, content hashes and collision rejection in both input orders. A closure probe emits byte-identical ordinary objects before/after with both the native and non-LTO LLVM backends.
  • Compiler slim-archive, pre-link preparation and incompatible-flag tests pass; the FullLTO CPU/FMV runtime fixture passes, including its GODEBUG child processes.
  • Earlier validation: static LLVM/plugin linker builds and runs the runtime fixture with GOGC=1; compiler-bootstrap/non-cgo linker builds; focused archive, linker-version and dist tests pass.
  • CI adds Linux amd64/arm64 FullLTO integration jobs and complete short suites for bytes, strings, strconv, sort, encoding/binary, encoding/json, hash/crc32 and unicode/utf8, without LTO-specific exclusions. Hosted results are the checks on the current PR head.

Scope: FullLTO at O2 for executable/PIE builds. ThinLTO, sanitizers, dynamic Go shared linking, separate compiler linkobj output, -N and custom optimization pipelines remain unsupported. FIPS inputs must retain ordinary object compilation.

Updates #341.

@zhouguangyuan0718 zhouguangyuan0718 changed the title cmd/llvmplugin: add single-module FullLTO backend cmd/llvmplugin: add FullLTO backend using LLVM LTO API Oct 21, 2026
@zhouguangyuan0718 zhouguangyuan0718 changed the title cmd/llvmplugin: add FullLTO backend using LLVM LTO API cmd/llvmplugin: add FullLTO backend with native LLVM merging Oct 28, 2026
zhouguangyuan0718 added a commit that referenced this pull request Oct 28, 2026
Ordinary Go compilation emits module-wide gotype records and repeats
NoSplit and DWARF encoding information. Move gotype references onto
their definitions, derive NoSplit from the existing function attribute,
and use LLVM's standard Dwarf Version flag. These changes let later IR
linking preserve the prevailing definition's type without converting
relationship tables.

Separate the shared object compatibility contract into an Error module
flag (`goobj.abi`) and retain the five package identity fields in
`goobj.config`. Replace the binding's duplicate configuration parser
with LLVM's shared decoder. Imports, package fingerprints, Go symbol
indices and runtime-specific metadata retain their existing meanings.

This PR is based directly on go1.27.master and is independent of #347.
It prepares the ordinary-compilation IR contract; FullLTO adaptation and
BSS/common lowering are separate work.

LLVM dependency: goallc/llvm-project#175
(merged). The workflow pins the immutable release
`goallc-llvm23.1.0-20261028T105443Z` at
`392b51c11256651f7772c5faaa2c689ef2775c72`. Both Linux release archives
were verified against their published SHA-256 files and build manifests.

The frontend and LLVM companion use only the new IR format. The
companion removes the old twelve/thirteen-field configuration decoder,
legacy DWARF configuration fields, and module-level gotype relationship
reader. Existing bitcode must be regenerated.

Validation:
- Complete darwin/arm64 make.bash bootstrap, standard-library/command
build and final cache checks passed with the companion LLVM payload
(dynamic linking) before removal of legacy readers.
- TestLLVMNativeMetadata passes for linux/amd64 and linux/arm64,
checking native and LLVM Go objects for NoSplit and the exact gotype
reference.
- TestLLVMDwarfObject, TestLLVMDwarfLink, TestLLVMDwarfFlag and
TestLLVMSlimArchive pass.
- TestLLVMDeterministicOutput passes all 16 compilations with varied
worker counts.
- After rebuilding and installing the LLVM companion without legacy
readers, all six focused Go test groups listed above passed again.
- LLVM-side tests cover the new configuration, DWARF 4/5, module-flag
conflicts and gotype selection. Broader LLVM GoObj suite results and
reproduced baseline failures are recorded in the companion PR.

CI follow-up: migrated five codegen fixtures to check
definition-attached gotype references and their type targets. The checks
passed for linux/amd64 and linux/arm64, and all 11 pre-pin Go CI checks
passed. A fresh full CI run against the formal release is required
before merging.
@zhouguangyuan0718
zhouguangyuan0718 force-pushed the codex/single-object-full-lto-backend branch from 0148488 to 6b02789 Compare October 30, 2026 12:40
@zhouguangyuan0718
zhouguangyuan0718 changed the base branch from go1.27.master to codex/goobj-native-debug-graph October 30, 2026 12:40
@zhouguangyuan0718
zhouguangyuan0718 force-pushed the codex/single-object-full-lto-backend branch from 6b02789 to 343eae2 Compare October 30, 2026 12:55
@zhouguangyuan0718
zhouguangyuan0718 changed the base branch from codex/goobj-native-debug-graph to go1.27.master October 30, 2026 12:57
@zhouguangyuan0718 zhouguangyuan0718 changed the title cmd/llvmplugin: add FullLTO backend with native LLVM merging cmd/go, cmd/link: support end-to-end single-object FullLTO Oct 31, 2026
@zhouguangyuan0718
zhouguangyuan0718 changed the base branch from go1.27.master to codex/early-ir-before-bitcode October 31, 2026 08:03
@zhouguangyuan0718
zhouguangyuan0718 force-pushed the codex/single-object-full-lto-backend branch 2 times, most recently from 0bb5734 to d279cdf Compare October 31, 2026 08:10
@zhouguangyuan0718
zhouguangyuan0718 changed the base branch from codex/early-ir-before-bitcode to go1.27.master October 31, 2026 09:05
@zhouguangyuan0718
zhouguangyuan0718 force-pushed the codex/single-object-full-lto-backend branch from d279cdf to cb1c893 Compare October 31, 2026 09:05
zhouguangyuan0718 added a commit that referenced this pull request Oct 31, 2026
#356)

LTO package compilation currently serializes prepared but unoptimized
IR. Run LLVM's `lto-pre-link<O2>` pipeline before writing the package
bitcode, leaving whole-program `lto<O2>` and late pre-codegen lowering
to the linker. Ordinary object compilation keeps its existing O0/O2
selection.

The change reuses the compiler's existing optimization path. With
`-llvm-keep-ir`, `.raw.ll` remains the frontend snapshot, `.ll` remains
prepared IR, and `.opt.ll` now records the pre-link optimized LTO input.
Verify the optimized module before serialization. Existing
unsupported-flag checks remain unchanged.

The amd64/arm64 archive regression disables inlining and verifies that
LLVM constant propagation removes a pure package-local call before
bitcode serialization, while both definitions retain their original
package symbol indices and the archive's export data remains usable.
This regression fails against #355 before the change. Focused emission,
early IR, optimization-mode and flag-rejection tests pass; ordinary
CPU/FMV runtime behavior also passes.

End-to-end validation uses a separate checkout combining this patch with
#347; the production PR remains based directly on the merged #355 commit
`8053408d154c`. The FullLTO CPU/FMV runtime and complete integration
suite pass (265.422s), including content hashing, assembly, cache reuse,
GC/stack growth, reflection, traceback, logical DWARF packages, linkname
positive/negative cases, mixed ordinary objects, PIE and cgo
automatic/external linking.

Performance qualification is still open. One macOS arm64 sample of the
45-package runtime fixture, with package caches warmed and the link
forced fresh, gave:

| Measurement | Prepared IR only | Pre-link O2 |
|---|---:|---:|
| Input bitcode | 16,095,200 B | 18,193,840 B |
| Combined optimized bitcode | 16,570,944 B | 17,744,944 B |
| Executable | 2,676,642 B | 2,741,842 B |
| Warm-cache build wall time | 16.39 s | 17.05 s |
| Maximum RSS reported by time | 1,531,494,400 B | 1,567,326,208 B |

This is a single sample, not a statistical benchmark or a measured
general speedup. The size increase is a reason to keep this PR in draft
for review. There are no custom LLVM pass lists or relaxed Go
symbol-identity checks.

Builds on #355; companion to #347. Do not merge automatically.
@zhouguangyuan0718
zhouguangyuan0718 force-pushed the codex/single-object-full-lto-backend branch from cb1c893 to bb112fe Compare October 31, 2026 12:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant