Transparent Data Movement (TDM) Transparent Data Movement (TDM) is a patent pending technology developed by RackTop to enable the seamless movement of data between tiers of storage within a BrickStor SP to external storage tiers including other BrickStor SP nodes, third party NFS capable storage and S3 compliant object storage. TDM is an advanced hierarchical storage management feature of the BrickStor SP operating system that enables policy-based security and compliance to be applied to data stored in the cloud or on third party storage systems. Policies can be applied to the data set to determine which target the data should tier to when policy dictates it should be moved to a more economical tier of storage. Users continue to access data through the same client protocols using the original file path and do not need to change their workflow. File Chunking BrickStor SP’s TDM feature intelligently chunks large files into smaller objects when tiered to an object store. The benefit of this chunking is that when a large file is updated that has been tiered to an object store, only the chunks with modifications must be updated in the object store. If the file was stored as one large object, then the entire file would have to be retrieved and rewritten as an object. This unique feature of BrickStor SP saves bandwidth and speed of file access and cost when using a cloud-based object store. Demand Cache BrickStor SP’s intelligent demand cache optimizer reduces cost and improves performance by reducing IO for remote files on the economic tier. When a file is tiered with TDM, it is not actually removed from the primary storage tier, the “demand cache”, until the space needs to be reclaimed by the OS for other files. This means if a user opens a file before it is evicted from the demand cache, the file will be opened from the primary tier’s demand cache. This eliminates any latency from the economic tier and costs that may be imposed from a cloud provider for IO and data retrieval. The version of the file on the economic tier will be updated if there are any modifications made to the file. Logical Segmentation – Enclave Elimination Many organizations want to eliminate physical system segmentation and silos to enable centralized monitoring and dynamic resource allocation. Previous security challenges can be overcome with the advanced access control features built into BrickStor SP’s Operating System. BrickStor SP includes granular access control capabilities to restrict access down to the individual file level. BrickStor SP includes discretionary access control across all client platforms, which are the most common access control scenarios. Discretionary access control is sufficient for government security accreditation of multiple enclaves within the same security domain. BrickStor SP supports host-based access control on top of discretionary access control. With SE Linux and NFS 4.2, BrickStor SP can support mandatory access control through the support of context security labels. With this architecture, a single BrickStor SP system can be accredited for access from multiple security domains and enclaves. Configuring TDM To enable TDM on a dataset: Select the dataset in the Connections pane Select the TDM tab in the Detail pane Click the "Enable TDM on Dataset" button Steps to enable TDM on a dataset In the dialog, choose the location: New S3, NFS, or a location that has been set up previously, and choose the local space reclamation policy. Local Space Reclamation There are three ways that TDM will handle the local data once it has been uploaded to the remote location: None - Always keep local copy (mirror) Dynamic - Keep local copy until space is needed. Immediate - Remove local copy after upload. In all cases, the data will appear to a client of a share to be local. When a request for a file is made, the file will be transparently downloaded from the remote location and returned to the client. When configuring TDM to use an NFS location, provide the following details: Server Path When configuring TDM to use an S3 bucket, provide the following details: Region Endpoint Bucket Object (optional) Access Key Secret Key Example S3 configuration TDM Status and Data Distribution In the dataset view, the Data Distribution graph will show the amount of data stored locally and the amount uploaded to the remote location. To view the status of TDM for all datasets or to reconfigure settings, choose the system in the Connections pane, and the TDM tab in the Detail pane. Reconfiguring TDM Once TDM is set up, it is possible to reconfigure local space reclamation settings by clicking the gear icon: Disabling TDM To disable TDM, click the stop icon. The disable operation will download all remote files to the local dataset prior to disabling if local space reclamation was set to Immediate or Dynamic. You will be presented with a choice of whether to delete (remove), or leave (keep) the remote data. To complete the operation, acknowledge warnings and commit the change: