Skip to content

fix(sp): preserve packed boundaries when padding auxiliary fields - #10180

Open
Excelius-Wang wants to merge 1 commit into
modelscope:mainfrom
Excelius-Wang:fix/ring-extra-field-alignment
Open

Excelius-Wang wants to merge 1 commit into
modelscope:mainfrom
Excelius-Wang:fix/ring-extra-field-alignment

Conversation

@Excelius-Wang

Copy link
Copy Markdown
Contributor

PR type

  • Bug Fix

PR information

Packed ring parallelism pads each sequence independently. Padding an earlier sequence shifts the boundaries of subsequent sequences. extra_split_values currently use the already-padded position IDs to pad tensors that still have their original lengths. With packed lengths [3, 5] and two ring ranks, this produces a four-position visual mask for six-position local embeddings, causing pad_and_split_mm_tokens to raise an IndexError in Qwen deepstack processing.

Pad auxiliary fields using the original sequence boundaries before padding the position IDs. Add CPU regressions for packed and single sequences, single-token samples, pure Ulysses, ring and combined parallel layouts, visual embedding gradients, and the Qwen deepstack caller after the input-splitting hook.

Experiment results

  • The 72 new cases produce 38 failures on the upstream method and all pass after the fix.
  • python -m pytest tests/sequence_parallel/test_input_alignment.py tests/sequence_parallel/test_zigzag_ring_attn.py -q: 114 passed, 4 CUDA tests skipped.
  • 680 additional local cases verify auxiliary integer/boolean/3D float fields and unchanged labels, loss weights, and 2D/mRoPE positions, including RP=3 and RP=2/SP=8.
  • A local reachability probe uses the official Qwen3-VL-2B tokenizer/image processor, Swift template/collator, and a reduced randomly initialized Qwen3-VL model. Two ordinary-format image/text samples encode to lengths [26, 33]. All 4 RP=2/SP=2 and all 16 RP=2/SP=8 simulated ranks fail on the upstream method and pass forward/backward after the fix. The latter model keeps the official 16 attention heads and 8 KV heads. An aligned-prefix [24, 33] control passes before and after.
  • pre-commit run --all-files and git diff --cached --check passed.

Validation uses PyTorch 2.9.1 and Transformers 5.12.1 on CPU. The additional probes are local harnesses outside this diff. Rank selection is simulated, and the reduced-model probe uses local SDPA, not distributed attention; it validates reachability and tensor alignment, not distributed numerical equivalence. No GPU collectives or full-size pretrained training were run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant