Skip to content

Split FreeWithHooksOrPerThread into per-thread and hooked free slow paths. - #1003

Draft
copybara-service[bot] wants to merge 1 commit into
masterfrom
test_983910962
Draft

copybara-service[bot] wants to merge 1 commit into
masterfrom
test_983910962

Conversation

@copybara-service

Copy link
Copy Markdown

Split FreeWithHooksOrPerThread into per-thread and hooked free slow paths.

FreeSmallSlow routed both "delete hooks installed" and "per-CPU caches
inactive" into one NOINLINE callee. That callee unconditionally built a
DeleteInfo (class_to_size load + 4 stack stores), tested delete_hooks_, then
re-tested CpuCacheActive before falling into the inlined ThreadCache::
Deallocate. Per-thread-mode binaries therefore paid the hook plumbing on
every free, and the hooked per-CPU path carried the ThreadCache body in its
frame (3 callee-saved pushes, 0x38 stack).

Dispatch directly from FreeSmallSlow instead:

  • HaveHooks() -> FreeSmallHooked (hook + DeallocateSlow, or the
    per-thread callee if per-CPU is inactive),
  • !UsePerCpuCache() -> FreeSmallPerThread (ThreadCache::Deallocate /
    transfer cache only, no hook plumbing),
  • otherwise -> inlined DeallocateSlowNoHooks (unchanged).

Both are tail calls out of FreeSmallSlow; its hot per-CPU body is unchanged
(register renaming only). Routing still keys on HaveHooks() rather than
delete_hooks_.empty() because new-hook-only installs still need
DeallocateSlow -> MaybeForceSlowPath to uncache the slab. Neither callee is
marked cold: per-thread and hooked binaries respectively take them on every
small free, so they stay NOINLINE only. Sampled-object handling is
untouched. The FreeSmall inline body and fast_path goldens are
byte-identical.

x86-64 -c opt, per-thread free (thread cache present, no overflow), callee
after FreeSmallSlow's dispatch:
before FreeWithHooksOrPerThread: 51 insns, 15 loads, 10 stores, 4 pushes
after FreeSmallPerThread: 32 insns, 9 loads, 5 stores, 1 push
Hooked per-CPU path: same instruction sequence, one fewer ptr spill/reload;
function shrinks 402 -> 249 bytes. FreeSmallSlow +16 bytes (second tail
call).

…aths.

FreeSmallSlow routed both "delete hooks installed" and "per-CPU caches
inactive" into one NOINLINE callee. That callee unconditionally built a
DeleteInfo (class_to_size load + 4 stack stores), tested delete_hooks_, then
re-tested CpuCacheActive before falling into the inlined ThreadCache::
Deallocate. Per-thread-mode binaries therefore paid the hook plumbing on
every free, and the hooked per-CPU path carried the ThreadCache body in its
frame (3 callee-saved pushes, 0x38 stack).

Dispatch directly from FreeSmallSlow instead:
 - HaveHooks()          -> FreeSmallHooked (hook + DeallocateSlow, or the
                           per-thread callee if per-CPU is inactive),
 - !UsePerCpuCache()    -> FreeSmallPerThread (ThreadCache::Deallocate /
                           transfer cache only, no hook plumbing),
 - otherwise            -> inlined DeallocateSlowNoHooks (unchanged).

Both are tail calls out of FreeSmallSlow; its hot per-CPU body is unchanged
(register renaming only). Routing still keys on HaveHooks() rather than
delete_hooks_.empty() because new-hook-only installs still need
DeallocateSlow -> MaybeForceSlowPath to uncache the slab. Neither callee is
marked cold: per-thread and hooked binaries respectively take them on every
small free, so they stay NOINLINE only. Sampled-object handling is
untouched. The FreeSmall inline body and fast_path goldens are
byte-identical.

x86-64 -c opt, per-thread free (thread cache present, no overflow), callee
after FreeSmallSlow's dispatch:
  before FreeWithHooksOrPerThread: 51 insns, 15 loads, 10 stores, 4 pushes
  after  FreeSmallPerThread:       32 insns,  9 loads,  5 stores, 1 push
Hooked per-CPU path: same instruction sequence, one fewer ptr spill/reload;
function shrinks 402 -> 249 bytes. FreeSmallSlow +16 bytes (second tail
call).

PiperOrigin-RevId: 983910962
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant