Software Block Size Limitations of a VMFS Datastore

Block Size Limitations of a VMFS Datastore

| Written by some jerk | Print | Email | Hits: 483

VMFS Block Sizes

The block size on a VMFS datastore defines two things:

 

  • The maximum file size
  • The amount of space a file occupies
To determine the block size used by a datastore:

From the VI/vSphere Client:
  1. Select an ESX host that contains the datastore.
  2. Click the Configuration tab.
  3. Click Storage.
  4. Select the datastore.
  5. The block size is identified in the Details window under the Formatting subheading.
From the ESX console:
  1. Log in to the ESX console.
  2. Run the command:

    # vmkfstools -P <path to datastore> 

    The block size is in bold in this line:

    Capacity 429228294144 (409344 file blocks * 1048576), 8896118784 (8484 blocks) available

    Where 1048576 is equivalent to a 1 MB block size on a VMFS datastore.

    This table reports the various block sizes that can be found in this output:

    Block Size Value Actual Block Size
    1048576 1 MB
    2097152 2 MB
    4194304 4 MB
    8388608 8 MB

    Note: vSphere 5.0 and later releases have a block size of 1 MB only.

VMFS-2 and VMFS-3 Size Limitations

There is no noticeable I/O performance difference when using a larger block size on thick provisioned virtual disks (VMDK).
 
Thin provisioned virtual disk (VMDK) performance, especially the performance of first writes, will be reduced as the block size of the VMFS datastore is increased, but subsequent writes to thin VMDK on any block size will be equivalent to eagerzeroedthick.
Therefore VMware recommends using a 1 MB block size when creating VMFS datastores for thin provisioned virtual disks for optimal performance. The benefits of using a smaller block size are:
  • Better performance during first write to thin provisioned virtual disks.
  • Minimize internal fragmentation, as the reason for using thin provisioned virtual disks is to save space. If a bigger block size is used, there will be more space wasted because of internal fragmentation.
The block size of the datastore also has an impact on the maximum file size (including virtual disk size being added to a virtual machine) that can be created on it. Therefore you must also consider the largest file or virtual disk size that you want to use when creating the VMFS datastore.
 
Note: Choose the VMFS block size carefully when creating VMFS datastores, because there is no way to change the block size of a VMFS datastore once it has been created. If you require a larger block size then the datastore will need to be recreated (this procedure is covered later in this article).
 
This table lists the maximum file and virtual disk sizes that are supported depending on the block size of the VMFS datastore:

 

Block Size
Largest virtual disk on VMFS-2
Largest virtual disk on VMFS-3
1 MB
456 GB
256 GB*
2 MB
912 GB
512 GB*
4 MB
1.78 TB
1 TB*
8 MB
2 TB
2 TB minus 512 B*
16 MB
2 TB
Invalid block size
32 MB
2 TB
Invalid block size
64 MB
2 TB
Invalid block size

* Overhead required to take snapshots of a virtual machine.

VMFS-3 uses sub-blocks for directories and small files with a size smaller than 64 KB. When the VMFS uses 1 sub-block of 64 KB, we switch to file blocks. The size of the file block depends on the block size you selected when the datastore was created.

In vSphere 4.0 and later, if snapshots are intended to be used, there are further size limitations on the virtual disk size as snapshots require an additional overhead. For more information, see Creating a snapshot for a virtual machine fails with the error: File is larger than maximum file size supported (1012384).

 

VMFS-5 Size Limitations

With VMFS-5, we use a unified 1 MB block size which is no longer configurable, but we can address larger files than a VMFS-3 1 MB block size can due to enhancements to the VMFS file system. Therefore a 1 MB VMFS-3 block size is not the same as a 1 MB VMFS-5 block size regarding file sizes.

The limits that apply to VMFS-5 datastores are:

 

  • The maximum virtual disk (VMDK) size is 2 TB minus 512 B for ESXi 5.0 and 5.1. In ESXi 5.5, the size is increased to 62TB.
  • The maximum virtual-mode RDM (vRDM) size is 2 TB minus 512 B for ESXi 5.0 and 5.1. In ESXi 5.5, the size is increased to 62TB.
  • Physical-mode RDMs are supported up to 64 TB.
For more information about large VMDK support in vSphere 5.5, see Support for virtual machine disks larger than 2 TB in vSphere 5.5 (2058287).

In VMFS-5, very small files (that is, files smaller than 1 KB) will be stored in the file descriptor location in the metadata rather than using file blocks. Once the file size increases beyond 1 KB, sub-blocks are used. After one 8 KB sub-block is used, 1 MB file blocks are used. As VMFS-5 uses sub-blocks of 8 KB rather than 64 KB (as in VMFS-3), this reduces the amount of disk space being used by small files. For more information on VMFS-5, see vSphere 5 FAQ: VMFS-5 (2003813).

Note on upgrading an existing datastore from VMFS-3 to VMFS-5

When upgrading a VMFS datastore from VMFS-3 to VMFS-5, you can extend a datastore past 2 TB - 512 B. The caveat to upgrading a VMFS-3 datastore to VMFS-5 is that it will inherit the block size properties of the original VMFS-3 datastore.

If you upgrade to VMFS-5 from VMFS-3 then regardless of the block size, VMFS-5 uses double-indirect addressing to cater for large files (up to a size of 2 TB - 512 B) on upgraded VMFS-3 volumes. For example, if the VMDK goes beyond 512 GB it will switch to using double-indirect addressing, which will allow for VMDKs up to 2 TB - 512 B.

Example of upgrading an existing VMFS-3 datastore to VMFS-5

Note: For this example, we will create a VMFS-3 volume and attempt to create a large (513 GB) file on that datastore. Then we will upgrade the datastore to VMFS-5 and test creating a large file on the upgraded datastore.

  1. Create a VMFS-3 datastore:

    vmkfstools -Ph -v10 /vmfs/volumes/cs -ee-d67-local/ 

    VMFS-3.54 file system spanning 1 partitions.
    File system label (if any): cs-ee-d67-local
    Mode: public
    Capacity 837 GB, 836.4 GB available, file block size 2 MB 

  2. Test creation of a large file (a VMDK larger than 512 GB):

    # vmkfstools -c 513G /vmfs/volumes/cs-ee-d67-local/513GB.vmdk
    Failed to create virtual disk: The destination file system does not support large files (12). 

  3. Upgrade the datastore to VMFS-5:

    vmkfstools -T /vmfs/double indirect addressingvolumes/cs-ee-d67-local/

    Please ensure that the VMFS-3 volume /vmfs/volumes/502369f3-0f06dcd7-ee21-0024e84b3c30 is not in active use by any local or remote ESX 3.x/4.x server.

    We recommend the following:

    Back up data on your volume as a safety measure.
    Take precautions to ensure no ESX 3.x/4.x servers are accessing this volume.

    Continue converting VMFS-3 to VMFS-5?

    0) _Yes
    1) _No

    Select a number from 0-1: 

    After entering 0 at the prompt, you see:

    Checking if remote hosts are using this device as a valid file system. This may take a few seconds...
    Upgrading file system /vmfs/volumes/cs-ee-d67-local/...
    done. 

  4. After the upgrade completes, verify that the block size is still 2 MB:

    vmkfstools -Ph -v10 /vmfs/volumes/cs-ee-d67-local/

    VMFS-5.54 file system spanning 1 partitions.
    File system label (if any): cs-ee-d67-local
    Mode: public
    Capacity 837 GB, 836.4 GB available, file block size 2 MB 

  5. Test creation of a large file again:

    vmkfstools -c 513G /vmfs/volumes/cs-ee-d67-local/513GB.vmdk

    Create: 100% done.
Note: For more information on upgrading to VMFS-5 and other criteria, see the VMFS-5 Upgrade Considerations Guide.

To recreate a datastore with a different block size

The block size on a datastore cannot be automatically changed as it is a file system property that can only be specified when the datastore is initially created.

The only way to increase the block size is to move all data off the datastore and recreate it with the larger block size. The preferred method of recreating the datastore is from a console or SSH session, as you can simply recreate the file system without having to make any changes to the disk partition.

Note: All data on a VMFS volume is lost when the datastore is recreated. Migrate or move all virtual machines and other data to another datastore. Back up all data before proceeding.

From the ESX/ESXi console:

Note: This procedure should not be performed on a local datastore on an ESX host where the operating system is located, as it may remove the Service Console privileged virtual machine which is located there.
  1. Storage vMotion, move, or delete the virtual machines located on the datastore you would like to recreate with a different block size.
  2. Log into the Local Tech Support Mode console of the ESX/ESXi host. For more information, see:
  3. Use the esxcfg-scsidevs command to obtain the disk identifier (mpx, naa, or eui) for the datastore you want to recreate:

    # esxcfg-scsidevs -m 

  4. Use vmkfstools to create a new VMFS datastore file system with a different block size over the existing one:

    # vmkfstools -C VMFS-type -b Block-Size -S Datastore-Name /vmfs/devices/disks/Disk-Identifier:Partition-Number 

    Example command to create a datastore called NewDatastore with an 8 MB block size:

    # vmkfstools -C vmfs3 -b 8m -S NewDatastore /vmfs/devices/disks/naa.60901234567890123456789012345678:1 

  5. Rescan from all other ESX hosts with the vmkfstools -V command.

From the VI / vSphere Client:

Note: This procedure should not be performed on a LUN containing the ESX/ESXi operating system, as it may require additional effort to recreate the partition table.
  1. Storage vMotion, move, or delete the virtual machines located on the datastore you would like to recreate with a different block size.
  2. Select the ESX/ESXi host in the inventory and click the Configuration tab.
  3. Select the Storage under hardware, right-click the datastore and choose Delete.

    Note: Do not do this on a datastore located on the same disk/LUN as the ESX/ESXi operating system.

  4. Rescan for VMFS volumes from the other hosts that can see the datastore.
  5. Create the new datastore with the desired block size on one of the hosts using the Add Storage Wizard.
  6. Rescan for VMFS volumes from all other hosts that can see the datastore.
Note: If extending a VMDK over 1 TB (1023 GB) on an ESX 3.5 host using the GUI fails, see Cannot extend a vmdk over 1TB (1008528).

Additional Information

To create a VMFS datastore larger than 2 TB in ESXi 4.x and earlier, you must extend the VMFS datastore over another LUN.
For information on increasing the capacity of the datastore in:
In ESXi 5.x, you can create a VMFS3 volume with a 1MB block size using a LUN of up to 50TB and a native VMFS5 volume using a LUN of up to 64TB.

Note: The actual maximum will depend on the RAID controller or maximum size of LUN supported by storage access driver (FC, iSCSI) that is being used. Contact your vendor to find the maximums.

See Also

Update History

12/07/2010 - Changed the VMDK sizes for VMFS2 - 4 01/24/2011 - Added links to ESX Configuration Guides 01/25/2012 - Added additional symptom 08/13/2012 - Add clarification on VMFS-3 to 5 upgrades 01/21/2013 - Added ESXi 5.1 configuration maximums and VMFS5 considerations