[lqcd-users] NOTICE: Migration of your data to a new Lustre file-system

[Amitoj Singh]

Dear LQCD JLab compute and storage users,

      As we get ready to deploy a new 4PB Lustre-filesystem based storage hardware we need to address the topic of migrating existing data with small files from /volatile and /cache storage areas to the new filesystem. Small files are a performance killer for the Lustre filesystem and will slow down the data migration process and so we would like to take the opportunity to bring this to your attention.

      Small files are defined as files that have sizes less than 1MB.

      As of now we plan to not migrate any small files to the new storage system. To mitigate this, we encourage everyone to please do the following:


  1.
Check your group/project small file count in /volatile and /cache using this following web pages -> link to volatile<https://lqcd.jlab.org/lqcd/volatileDisk/project>, link to cache<https://lqcd.jlab.org/lqcd/cacheDisk/project>
  2.
Do the following to reduce your small file count:
     *
delete files that are no longer needed -OR-
     *
compress small files into a single file -OR -
     *
move small files to /work.

If you have any questions or concerns, please do not hesitate to email me directly.


Amitoj Singh,

HPC Architect

High Performance Data Facility

Computational Sciences and Technology Division

Thomas Jefferson National Accelerator Facility


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/lqcd-users/attachments/20260114/dc58455e/attachment.htm>