Add shuffling in Nanotron for subsequent epochs when data is repeated

[Lauler]

Nanoset's index builder does not re-shuffle dataset and sample indices within epochs when training secondary, third, etc epochs. It instead concatenates a copy of the same indices for any repeated data. This PR adds unique within-epoch shuffling for each epoch.

See the following issue: #237

I ran the tests in tests/nanotron:

=========================================================================== warnings summary ===========================================================================
tests/helpers/context.py:7: 35 warnings
  /home/faton/projects/text/nanotron_dev/nanotron/tests/helpers/context.py:7: PytestCollectionWarning: cannot collect test class 'TestContext' because it has a __init__ constructor (from: nanoset/test_build_nanoset_dataloader.py)
    class TestContext:

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=================================================================== 12 passed, 35 warnings in 44.55s ===================================================================

@TJ-Solergibert

[NouamaneTazi]

Nice catch! LGTM 🤗
Minor sugg before merging

[NouamaneTazi]

Can we just use self.random_seed + num_epoch for easier reproducibility?

[Lauler]

Good suggestion, makes it a lot simpler and clearer what's going on. I've changed the code to incorporate this in latest commit:

nanotron/src/nanotron/data/nanoset.py

Lines 116 to 123 in f060414

	dataset_indices = []
	dataset_sample_indices = []
	for num_epoch in range(num_epochs):
	# Shuffle the sample and dataset indices in epoch with a given seed
	numpy_random_state = np.random.RandomState(self.random_seed + num_epoch)
	numpy_random_state.shuffle(dataset_index)
	numpy_random_state = np.random.RandomState(self.random_seed + num_epoch)
	numpy_random_state.shuffle(dataset_sample_index)