[BUG] deduplicate_internally misses cross-batch duplicates: np.tril mask uses local row indices

[businessarshgoyal]

Required prerequisites

• I have read the documentation https://docs.camel-ai.org.
• I have searched the Issue Tracker and Discussions that this hasn't already been reported. (+1 or comment there if it has.)

What version of camel are you using?

master (e88b5ee)

Description

deduplicate_internally in camel/utils/deduplication.py processes similarity in batches of batch_size (default 1000), and masks the similarity matrix with np.tril(..., k=-1):

batch_similarities = cosine_similarity(
    embeddings_array[i:batch_end], embeddings_array[:batch_end]
)
tril_mask = np.tril(np.ones_like(batch_similarities), k=-1)
batch_similarities = batch_similarities * tril_mask

batch_similarities has shape (batch_end - i, batch_end): its rows are the texts i .. batch_end, its columns are texts 0 .. batch_end. np.tril(..., k=-1) masks relative to the local row index, so for every batch after the first, row j (global index i + j) only keeps columns < j instead of columns < i + j. All comparisons against texts in [j, i + j) are dropped, so duplicates whose original text lies in an earlier batch are silently missed.

The result is that deduplication output depends on batch_size, which is documented as a memory knob only:

from camel.utils import deduplicate_internally

texts = ["A", "B", "A"]
embeddings = [[1.0, 0.0], [0.0, 1.0], [1.0, 0.0]]

for batch_size in (1, 2, 3):
    print(batch_size, deduplicate_internally(
        texts=texts, threshold=0.9, embeddings=embeddings,
        batch_size=batch_size,
    ).duplicate_to_target_map)

Actual output on master:

1 {}
2 {}
3 {2: 0}

Expected: {2: 0} for every batch size. With the default batch_size=1000 any corpus larger than 1000 texts is affected: for the second batch onwards, roughly the first 1000 columns of each row are never compared, so genuine duplicates come back as unique.

I verified this locally with the snippet above and with a randomized cross-check (60 embeddings plus 20 repeats) where the batched result differs from the single-batch result on master.

Reproducible example code

from camel.utils import deduplicate_internally

texts = ["A", "B", "A"]
embeddings = [[1.0, 0.0], [0.0, 1.0], [1.0, 0.0]]
print(deduplicate_internally(texts=texts, threshold=0.9, embeddings=embeddings, batch_size=2).duplicate_to_target_map)
# {}  -- expected {2: 0}

Proposed fix

Mask using the global row indices instead of the local ones, so a batch row is compared against every text that precedes it:

row_indices = np.arange(i, batch_end)[:, None]
column_indices = np.arange(batch_end)[None, :]
batch_similarities = np.where(column_indices < row_indices, batch_similarities, 0)

With this change the batched result equals the single-batch result for every batch size I tried. Happy to open a PR with regression tests (I have one ready that fails on master and passes with the fix).

Disclosure: I used an AI coding assistant while investigating; I reproduced the behaviour and ran the tests locally myself.