[design] Lowering hand-built generic MIR (Route A) blocked by GlobalISel/SelectionDAG entry-point tension

[makslevental]

Lowering hand-built generic MIR (Route A) is blocked by the GlobalISel↔SelectionDAG entry-point tension

The MIR stack (#579–#591) ended up supporting two routes from Python-built MIR to native code. Route B is implemented; Route A is not, and this issue records why — the reason is architectural, not an oversight, and it's worth having written down.

The two routes

• Route A — build generic (G_*) MIR, then run instruction selection on it.
  @machine_function / MachineIRBuilder produce target-independent G_ADD/G_ICMP/G_PHI/… MIR. The natural next step would be to hand that MIR to the target's post-IRTranslator pipeline (Legalizer → RegBankSelect → InstructionSelect → regalloc → emission) and get an object out. This does not work today.

• Route B — build already-selected target MIR by hand, run only the back half of codegen. ([eudsl-llvmpy] Build already-selected target MIR by hand (Route B foundation) #589, [eudsl-llvmpy] Emit object from selected MIR + JIT-execute it (Route B capstone) #590)
  Resolve real register classes / physregs, emit target opcodes (ADDWrr, COPY, RET_ReallyLR), set the MachineFunctionProperties, then emit_object() runs addPassesToEmitFile with -start-after=finalize-isel so no instruction selection runs at all — just regalloc + prologue/epilogue + emission. This is what the capstone JIT-executes. It sidesteps the entire problem below.

Why Route A is blocked

1. SelectionDAG is not a MIR→MIR pass. SelectionDAGISel builds its DAG from IR (one basic block at a time) and only then emits MIR. So the SelectionDAG path fundamentally cannot consume hand-built generic MIR — there is nothing to feed it but IR.

2. GlobalISel is MIR→MIR, but its failure path re-enters SelectionDAG from IR. GlobalISel's InstructionSelect (and the earlier legalize/regbankselect steps) operate on MIR, so in principle they could lower hand-built generic MIR. But when GlobalISel can't handle a construct, the abort mode falls back to SelectionDAG — which needs the IR. Hand-built MIR has no faithful IR (Route B's create_machine_function attaches only a trivial stub definition so codegen doesn't skip the function), so the fallback can't reproduce the intended semantics. run_codegen_to_mir(global_isel=True) ([eudsl-llvmpy] Run the GlobalISel pipeline + expose the machine verifier #588) works precisely because it starts from IR: it runs IRTranslator → Legalizer → RegBankSelect → InstructionSelect and never has externally-authored generic MIR to preserve.

3. The pipeline entry point is IR-shaped. TargetPassConfig / addPassesToEmitFile build a pipeline that begins at the IR. To lower pre-existing generic MIR you'd need to start it partway (e.g. -start-before=legalizer / after IRTranslator) while feeding in the already-built MachineFunctions. The only levers today are the fragile process-global start-before/start-after cl::opts (Route B uses start-after=finalize-isel with a set/restore guard — see [eudsl-llvmpy] Emit object from selected MIR + JIT-execute it (Route B capstone) #590 and the note in [TODO] MIR stack: deferred review items (#579–#591) #593 item 1) or a programmatic TargetPassConfig::setStartStopPasses, which isn't cleanly bound.

Current mitigations already in place (context, not fixes for Route A)

• [eudsl-llvmpy] Run the GlobalISel pipeline + expose the machine verifier #588: run_codegen_to_mir(global_isel=True) uses GlobalISelAbortMode::DisableWithDiag (aborting mode would kill the process) and then scans the result for residual G_* ops, raising if selection didn't complete — because the fallback's diagnostic is only a warning. It also sets the GlobalISel flags per-call (AArch64 defaults EnableGlobalISel=true).
• These make the IR→selected-MIR path safe; they do not enable hand-built-generic-MIR→selected-MIR.

What would unblock Route A (for discussion)

• A clean binding for TargetPassConfig::setStartStopPasses (or equivalent) to construct a "start at Legalizer, stop after emission" pipeline over externally-built generic MIR — replacing the process-global start-* cl::opt mechanism entirely.
• Running the GlobalISel legalize→select passes on hand-built MIR with the SelectionDAG fallback disabled and unselected ops surfaced as a catchable error (the residual-G_* scan from [eudsl-llvmpy] Run the GlobalISel pipeline + expose the machine verifier #588 is half of this already).
• Confirming the MachineModuleInfoWrapperPass build path (from create_machine_function) can carry generic MIR through those passes, and deciding what — if any — IR stub the functions need for the passes that still consult it.

Until one of those lands, Route B is the supported way to reach native code from Python-built MIR, and Route A generic-MIR lowering is only reachable indirectly by going back through IR (run_codegen_to_mir).

---

Refs: #588 (GlobalISel gating + residual scan), #589 (build selected MIR), #590 (emit_object / -start-after=finalize-isel), and the deferred-items tracker #593 (item 1 covers the start-after cl::opt fragility).

[makslevental]

Potential upstream (LLVM) patch that would unblock most of this

The single highest-leverage upstream change is making the codegen start/stop insertion points programmatic instead of process-global.

Root cause upstream. TargetPassConfig derives its start/stop points from static command-line options — StartAfterOpt / StartBeforeOpt / StopAfterOpt / StopBeforeOpt (static cl::opt<std::string> in TargetPassConfig.cpp) — which it reads into its member fields at construction time. There is no public per-instance setter, and no parameter on LLVMTargetMachine::addPassesToEmitFile / createPassConfig to pass them in. So an embedder that wants "start after finalize-isel" (Route B) or "start before legalizer" (Route A) has no choice but to mutate the global option and restore it — exactly the fragile, GIL-dependent, non-thread-safe hack in emit_object (see the issue body and #593 item 1).

Proposed patch. Thread the start/stop points through as data rather than globals — e.g. a TargetPassConfig::setStartStopPasses(StartStopInfo) public setter (or a StartStopInfo parameter on addPassesToEmitFile/createPassConfig) that takes precedence over the cl::opts. This is a small, mechanically-contained change — the pipeline plumbing that consumes these points (getStartStopPasses, addMachinePasses) already exists; only the source of the values would change.

What it unblocks here:

• Removes the process-global -start-after mutate/restore entirely in Route B's emit_object — no lock/GIL reliance, no restore-on-throw/abort concern, no cross-thread clobbering.
• Makes Route A tractable: we could build a "start-before = Legalizer, stop after emission" pipeline over externally-built generic MIR without touching any global, feeding in the MachineModuleInfoWrapperPass from create_machine_function.

A second, independent upstream nicety

GlobalISel selection failure currently forces a bad choice: GlobalISelAbortMode::Enable → report_fatal_error (kills the process), or DisableWithDiag → fall back to SelectionDAG-from-IR (which hand-built MIR can't satisfy) with only a warning. A mode that fails recoverably — surface the unselectable instruction as a normal Error/diagnostic, no fatal abort, no IR fallback — would let hand-built generic MIR that can't be selected error cleanly instead of relying on our post-hoc residual-G_* scan (#588). Nice-to-have, but the start/stop patch above is the one that actually gates Route A.

Caveat

Programmatic start/stop is necessary but possibly not sufficient for Route A: some passes in the pre-select region may still consult the IR (or expect state the IRTranslator would have produced). The right next step is a small prototype against a patched (or locally-hacked) TargetPassConfig to see how far a "start at Legalizer over hand-built generic MIR" pipeline actually gets before it wants something only IR can provide.

(Symbol names from memory against a recent LLVM — worth confirming against the pinned third_party/llvm-project before filing anything upstream.)

[makslevental]

The specific patch

Against the pinned third_party/llvm-project (c31c334e1118). Today TargetPassConfig bakes its start/stop points in the constructor from the private no-arg setStartStopPasses(), which reads the global StartBeforeOpt/StartAfterOpt/StopBeforeOpt/StopAfterOpt cl::opts (TargetPassConfig.cpp:466) into the private members StartBefore/StartAfter/StopBefore/StopAfter + instance-num fields (TargetPassConfig.h:87-102). There is no public way to override them per-instance.

This adds a distinct public setter, overrideStartStopPasses, that sets those points programmatically and recomputes Started, so an embedder can run a sub-range of the pipeline without touching the process-global options. (Kept as its own name rather than overloading the private no-arg setStartStopPasses(), so the public API and the ctor-internal cl::opt reader stay clearly separate.)

--- a/llvm/include/llvm/CodeGen/TargetPassConfig.h
+++ b/llvm/include/llvm/CodeGen/TargetPassConfig.h
@@ -192,6 +192,20 @@ class LLVM_ABI TargetPassConfig : public ImmutablePass {
   getStartStopInfo(PassInstrumentationCallbacks &PIC);

   void setDisableVerify(bool Disable) { setOpt(DisableVerify, Disable); }
+
+  /// Programmatically set the start/stop insertion points, overriding the
+  /// -start-before / -start-after / -stop-before / -stop-after command-line
+  /// options. Lets an embedder run a sub-range of the codegen pipeline (e.g.
+  /// start at the legalizer over externally-built MIR) without mutating the
+  /// process-global cl::opts. Must be called before addISelPasses() /
+  /// addMachinePasses(). At most one of StartBefore/StartAfter and one of
+  /// StopBefore/StopAfter may be non-null; a null start point means "begin at
+  /// the normal point", a null stop point means "run to the end".
+  void overrideStartStopPasses(AnalysisID StartBefore, AnalysisID StartAfter,
+                               AnalysisID StopBefore, AnalysisID StopAfter,
+                               unsigned StartInstanceNum = 0,
+                               unsigned StopInstanceNum = 0);

   bool getEnableTailMerge() const { return EnableTailMerge; }
   void setEnableTailMerge(bool Enable) { setOpt(EnableTailMerge, Enable); }

--- a/llvm/lib/CodeGen/TargetPassConfig.cpp
+++ b/llvm/lib/CodeGen/TargetPassConfig.cpp
@@ -493,6 +493,25 @@ void TargetPassConfig::setStartStopPasses() {
   Started = (StartAfter == nullptr) && (StartBefore == nullptr);
 }

+void TargetPassConfig::overrideStartStopPasses(AnalysisID StartBeforeID,
+                                               AnalysisID StartAfterID,
+                                               AnalysisID StopBeforeID,
+                                               AnalysisID StopAfterID,
+                                               unsigned StartInstanceNum,
+                                               unsigned StopInstanceNum) {
+  if (StartBeforeID && StartAfterID)
+    reportFatalUsageError("StartBefore and StartAfter both specified!");
+  if (StopBeforeID && StopAfterID)
+    reportFatalUsageError("StopBefore and StopAfter both specified!");
+  StartBefore = StartBeforeID;
+  StartAfter = StartAfterID;
+  StopBefore = StopBeforeID;
+  StopAfter = StopAfterID;
+  StartBeforeInstanceNum = StartAfterInstanceNum = StartInstanceNum;
+  StopBeforeInstanceNum = StopAfterInstanceNum = StopInstanceNum;
+  Started = (StartAfter == nullptr) && (StartBefore == nullptr);
+  Stopped = false;
+}
+
 CGPassBuilderOption llvm::getCGPassBuilderOption() {
   CGPassBuilderOption Opt;

How eudsl uses it

run_codegen_to_mir already owns the TargetPassConfig directly (tm.createPassConfig(*pm)), so it can call the new setter before adding passes — no global mutation, no restore, no GIL reliance. Route B's emit_object, however, goes through addPassesToEmitFile, which constructs the TPC internally (CodeGenTargetMachineImpl.cpp:123, addPassesToGenerateCode). To reach the setter there, the cleanest follow-on is a TargetPassConfig * handle plumbed out of / a StartStopInfo parameter threaded into addPassesToEmitFile — a slightly larger, signature-touching change — but with the primitive above, emit_object can already be reworked to build its own pipeline via createPassConfig + addPassesToEmitMC-style wiring instead of the -start-after opt hack.

AnalysisID is a pass's &ID (e.g. &FinalizeISel::ID, &Legalizer::ID), so the call reads e.g.:

// Route B today, without the global-opt mutation:
tpc->overrideStartStopPasses(/*StartBefore=*/nullptr, /*StartAfter=*/&FinalizeISel::ID,
                             /*StopBefore=*/nullptr, /*StopAfter=*/nullptr);

// Route A prototype: start at the legalizer over hand-built generic MIR.
tpc->overrideStartStopPasses(/*StartBefore=*/&Legalizer::ID, /*StartAfter=*/nullptr,
                             /*StopBefore=*/nullptr, /*StopAfter=*/nullptr);

Caveats

• As noted in the issue body, this is necessary but possibly not sufficient for Route A: the legalize→select passes may still consult IR/analyses the IRTranslator would have produced. The setter is what lets us build the truncated pipeline and find out; the residual-G_* scan from [eudsl-llvmpy] Run the GlobalISel pipeline + expose the machine verifier #588 is the safety net for what it can't select.
• Line numbers/AnalysisID spellings are against c31c334e1118; confirm before sending upstream.