Repository navigation
cmd/go, cmd/link: support end-to-end single-object FullLTO - #347
Open
zhouguangyuan0718 wants to merge 8 commits into
Open
zhouguangyuan0718 wants to merge 8 commits into
zhouguangyuan0718 wants to merge 8 commits into
Conversation
zhouguangyuan0718
added a commit
that referenced
this pull request
Oct 28, 2026
Ordinary Go compilation emits module-wide gotype records and repeats NoSplit and DWARF encoding information. Move gotype references onto their definitions, derive NoSplit from the existing function attribute, and use LLVM's standard Dwarf Version flag. These changes let later IR linking preserve the prevailing definition's type without converting relationship tables. Separate the shared object compatibility contract into an Error module flag (`goobj.abi`) and retain the five package identity fields in `goobj.config`. Replace the binding's duplicate configuration parser with LLVM's shared decoder. Imports, package fingerprints, Go symbol indices and runtime-specific metadata retain their existing meanings. This PR is based directly on go1.27.master and is independent of #347. It prepares the ordinary-compilation IR contract; FullLTO adaptation and BSS/common lowering are separate work. LLVM dependency: goallc/llvm-project#175 (merged). The workflow pins the immutable release `goallc-llvm23.1.0-20261028T105443Z` at `392b51c11256651f7772c5faaa2c689ef2775c72`. Both Linux release archives were verified against their published SHA-256 files and build manifests. The frontend and LLVM companion use only the new IR format. The companion removes the old twelve/thirteen-field configuration decoder, legacy DWARF configuration fields, and module-level gotype relationship reader. Existing bitcode must be regenerated. Validation: - Complete darwin/arm64 make.bash bootstrap, standard-library/command build and final cache checks passed with the companion LLVM payload (dynamic linking) before removal of legacy readers. - TestLLVMNativeMetadata passes for linux/amd64 and linux/arm64, checking native and LLVM Go objects for NoSplit and the exact gotype reference. - TestLLVMDwarfObject, TestLLVMDwarfLink, TestLLVMDwarfFlag and TestLLVMSlimArchive pass. - TestLLVMDeterministicOutput passes all 16 compilations with varied worker counts. - After rebuilding and installing the LLVM companion without legacy readers, all six focused Go test groups listed above passed again. - LLVM-side tests cover the new configuration, DWARF 4/5, module-flag conflicts and gotype selection. Broader LLVM GoObj suite results and reproduced baseline failures are recorded in the companion PR. CI follow-up: migrated five codegen fixtures to check definition-attached gotype references and their type targets. The checks passed for linux/amd64 and linux/arm64, and all 11 pre-pin Go CI checks passed. A fresh full CI run against the formal release is required before merging.
zhouguangyuan0718
force-pushed
the
codex/single-object-full-lto-backend
branch
from
October 30, 2026 12:40
0148488 to
6b02789
Compare
zhouguangyuan0718
changed the base branch from
go1.27.master
to
codex/goobj-native-debug-graph
October 30, 2026 12:40
zhouguangyuan0718
force-pushed
the
codex/single-object-full-lto-backend
branch
from
October 30, 2026 12:55
6b02789 to
343eae2
Compare
zhouguangyuan0718
changed the base branch from
codex/goobj-native-debug-graph
to
go1.27.master
October 30, 2026 12:57
zhouguangyuan0718
changed the base branch from
go1.27.master
to
codex/early-ir-before-bitcode
October 31, 2026 08:03
zhouguangyuan0718
force-pushed
the
codex/single-object-full-lto-backend
branch
2 times, most recently
from
October 31, 2026 08:10
0bb5734 to
d279cdf
Compare
zhouguangyuan0718
changed the base branch from
codex/early-ir-before-bitcode
to
go1.27.master
October 31, 2026 09:05
zhouguangyuan0718
force-pushed
the
codex/single-object-full-lto-backend
branch
from
October 31, 2026 09:05
d279cdf to
cb1c893
Compare
zhouguangyuan0718
added a commit
that referenced
this pull request
Oct 31, 2026
#356) LTO package compilation currently serializes prepared but unoptimized IR. Run LLVM's `lto-pre-link<O2>` pipeline before writing the package bitcode, leaving whole-program `lto<O2>` and late pre-codegen lowering to the linker. Ordinary object compilation keeps its existing O0/O2 selection. The change reuses the compiler's existing optimization path. With `-llvm-keep-ir`, `.raw.ll` remains the frontend snapshot, `.ll` remains prepared IR, and `.opt.ll` now records the pre-link optimized LTO input. Verify the optimized module before serialization. Existing unsupported-flag checks remain unchanged. The amd64/arm64 archive regression disables inlining and verifies that LLVM constant propagation removes a pure package-local call before bitcode serialization, while both definitions retain their original package symbol indices and the archive's export data remains usable. This regression fails against #355 before the change. Focused emission, early IR, optimization-mode and flag-rejection tests pass; ordinary CPU/FMV runtime behavior also passes. End-to-end validation uses a separate checkout combining this patch with #347; the production PR remains based directly on the merged #355 commit `8053408d154c`. The FullLTO CPU/FMV runtime and complete integration suite pass (265.422s), including content hashing, assembly, cache reuse, GC/stack growth, reflection, traceback, logical DWARF packages, linkname positive/negative cases, mixed ordinary objects, PIE and cgo automatic/external linking. Performance qualification is still open. One macOS arm64 sample of the 45-package runtime fixture, with package caches warmed and the link forced fresh, gave: | Measurement | Prepared IR only | Pre-link O2 | |---|---:|---:| | Input bitcode | 16,095,200 B | 18,193,840 B | | Combined optimized bitcode | 16,570,944 B | 17,744,944 B | | Executable | 2,676,642 B | 2,741,842 B | | Warm-cache build wall time | 16.39 s | 17.05 s | | Maximum RSS reported by time | 1,531,494,400 B | 1,567,326,208 B | This is a single sample, not a statistical benchmark or a measured general speedup. The size increase is a reason to keep this PR in draft for review. There are no custom LLVM pass lists or relaxed Go symbol-identity checks. Builds on #355; companion to #347. Do not merge automatically.
zhouguangyuan0718
force-pushed
the
codex/single-object-full-lto-backend
branch
from
October 31, 2026 12:36
cb1c893 to
bb112fe
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Compiler prerequisites #355 and #356 are merged. Based on
42dc37cc80160e792a97df75221af57263fc6a2f: package inputs have completed early IR preparation and LLVM pre-link O2 optimization.Known blocker: the preceding Linux amd64/arm64 FullLTO CI run failed both cgo modes because
runtime._cgo_clearenvhas an indexed BSS definition in runtime and an initialized linkname definition in runtime/cgo. The current backend still rejects this as multiply defined; a minimal indexed-BSS/initialized-definition probe reproduces it. This PR is not ready to merge. Other integration cases in that run passed, but the subsequent standard-library step did not run.Enable
go build -lto=fullend to end. Package bitcode is merged and optimized by LLVM, emitted as one GoObj, and consumed by the ordinary Go linker to produce executable and PIE binaries. The backend supports dynamic and static LLVM/plugin linkage and does not invoke external LLVM tools.Content-addressed functions retain native hash classification. The compiler includes the compiling package in the existing inline-hash storage discriminator when creating LTO closures, including original closures and imported inline copies. LTO preserves those input names and rejects duplicate content-addressed identities; it no longer strips and regenerates their discriminators. GoObj strips the temporary discriminator before hashing and writing names. Identical machine code and relocations can merge across packages; different bodies stay distinct. No package-qualified output names or post-codegen name patching are needed.
The build driver selects bitcode compilation, includes LLVM/plugin identity in link cache keys, and leaves FIPS packages as ordinary objects. The linker rewrites archives only in a temporary directory, preserving assembly/native members and cached inputs. A package manifest restores imports, fingerprints, cgo directives, original symbol indices, logical compilation units and linkname origins over the single physical object. Mixed native objects, assembly references, initialization order and linkname checks remain supported.
LLVM handles native definition selection, reference remapping, ABI module flags, debug graphs and definition attachments. The Go adapter retains package/index provenance and Go-specific BSS requirements. Declared package names stay in native DWARF instead of being copied into the manifest. Linkname checks use original reference packages, allowing cross-package inlining without extra noinline restrictions. CPU/FMV preparation is performed by the compiler prerequisite; the LTO merger no longer constructs or runs a preparation pipeline, rejects unprepared compiler bitcode, and discards consumed CPU configuration rather than restoring main's copy. Saved input IR is written from the backend's existing parse in
optimized/inputs, removing the extra Go-side parse. Compiler flag rejection remains covered by the compiler suite; the LTO suite tests build-driver restrictions.Validation after rebasing and cleanup:
GOGC=1; compiler-bootstrap/non-cgo linker builds; focused archive, linker-version and dist tests pass.Scope: FullLTO at O2 for executable/PIE builds. ThinLTO, sanitizers, dynamic Go shared linking, separate compiler linkobj output, -N and custom optimization pipelines remain unsupported. FIPS inputs must retain ordinary object compilation.
Updates #341.