NY_FileStorePlugIn
The plug-in is required to provide files to other services. It is also necessary to download files from other nodes.
Directory
The FileStore directory is located by default in the storage directory. It includes three other directories:
-
The
indexeddirectory contains all files and folders that have been imported into the FileStore. The files in this directory should not be manually altered, as it is not monitored. If files need to be deleted, the node must be restarted afterward so that the FileStore can report the changes to the FileRegistry. -
The
incompletedirectory is used for downloading files. It stores files that have not yet been completely downloaded. -
The
incomingdirectory is monitored by the node. Files and folders copied into this directory are immediately indexed. Subsequently, they are removed from this directory.
Registry
All files are reported to a central file registry with their relative path.
The relative path refers to the location of the file relative to the indexed directory.
Example
If the absolute path of a file on Windows is C:\NodeName\storage\FileStore\indexed\a\b\c\file.txt,
then the relative path is a\b\c\file.txt.
The relative path serves as a search term for the file. The file can also be searched using its hash value. Alongside the path and hash value, the timestamp of the last modification is reported to the registry. This enables the registry to return the most recent file when there are multiple matches and the search is based solely on the path.
File Hashes
The hash value is automatically generated by the system.
We use MD5 hashes as they are fast and meet our requirements.
The generated hash is converted into a string using a
CrockfordBase32
algorithm, suitable for storage in the filesystem.
The files are stored in the FileStore\indexed directory and retain their relative paths.
The file name extension for files in the index is ._32_[Base32-MD5].
Example: annotations-24.1.0.jar._32_71KKNFM7T4JGHK85967FY2WQ64
File Transfer
Files are transported in chunks across the network. This method is used to prevent slowing down other traffic between nodes. No separate TCP channel is opened for the transportation. The messages are assigned a low priority. Files retain their timestamps, both for creation and last modification, during import and transport.
Execution of the Plugin
The plugin should be run on each node that provides files or requires files from other FileStore services.