3.9 Selecting Output File Format

Availability: ncap2, ncbo, ncea, ncecat, ncflint, ncpdq, ncra, ncrcat, ncwa Short options: ‘-4’ Long options: ‘--4’, ‘--64bit’, ‘--fl_fmt’, ‘--netcdf4’

All NCO operators support (read and write) all three (or four, depending on how one counts) file formats supported by netCDF4. The operators listed under “Availability” above allow the user to specify the output file format independent of the input file format. These operators allow the user to convert between the various file formats. The operators ncatted and ncrename always write the output netCDF file in the same format as the input netCDF file.

netCDF supports four types of files: CLASSIC, 64BIT, NETCDF4, and NETCDF4_CLASSIC, The CLASSIC format is the traditional 32-bit offset written by netCDF2 and netCDF3. As of 2005, most netCDF datasets are in CLASSIC format. The 64BIT format was added in Fall, 2004.

The NETCDF4 format uses HDF5 as the file storage layer. The files are (usually) created, accessed, and manipulated using the traditional netCDF3 API (with numerous extensions). The NETCDF4_CLASSIC format refers to netCDF4 files created with the NC_CLASSIC_MODEL mask. Such files use HDF5 as the back-end storage format (unlike netCDF3), though they incorporate only netCDF3 features. Hence NETCDF4_CLASSIC files are perfectly readable by applications which use only the netCDF3 API and library. NCO must be built with netCDF4 to write files in the new NETCDF4 and NETCDF4_CLASSIC formats, and to read files in the new NETCDF4 format. Users are advised to use the default CLASSIC format or the NETCDF4_CLASSIC format until netCDF4 is more widespread. Widespread support for NETCDF4 format files is not expected for a few years, 2007–2008, say. If performance or coolness are issues, then use NETCDF4_CLASSIC instead of CLASSIC format files.

All operators write classic CLASSIC (32-bit offset) format files unless told otherwise. Toggling the long option ‘--64bit’ switch (or its key-value equivalent ‘--fl_fmt=64bit’) produces the netCDF3 64-bit offset format named 64BIT. NCO must be built with netCDF 3.6 or higher to produce a 64BIT file. Toggling the ‘-4’ switch (or its long option equivalents ‘--4’ or ‘--netcdf4’), or setting its key-value equivalent ‘--fl_fmt=netcdf4’ produces a NETCDF4 file (i.e., HDF). Casual users are advised to use the default (netCDF3) CLASSIC format until netCDF 3.6 and netCDF 4.0 are more widespread.

These examples show how to convert a file from any netCDF format into any other netCDF format (subject to limits of the format):

     ncks -O --fl_fmt=classic in.nc foo_3c.nc # netCDF3 classic
     ncks -O --fl_fmt=64bit in.nc foo_364.nc # netCDF3 64bit
     ncks -O --fl_fmt=netcdf4 in.nc foo_4.nc # netCDF4
     ncks -O --fl_fmt=netcdf4_classic in.nc foo_4c.nc # netCDF4 classic

To discover whether a netCDF file is a classic (32-bit offset) or newer 64-bit offset netCDF3 format, or is netCDF4 format, examine it with the od (octal dump) command:

     % od -An -c -N4 foo_3c.nc
        C   D   F 001
     % od -An -c -N4 foo_364.nc
        C   D   F 002
     % od -An -c -N4 foo_4.nc
      211   H   D   F
     % od -An -c -N4 foo_4c.nc
      211   H   D   F

Values of ‘C D F 001’ and ‘C D F 002’ indicate 32-bit (classic) and 64-bit netCDF3 formats, respectively, while values of ‘211 H D F’ indicate the newer netCDF4 file format. Note that NETCDF4 and NETCDF4_CLASSIC are the same formats. The latter simply causes an application to fail if it attempts to write a NETCDF4 file that cannot be completely read by the netCDF3 library. As of October, 2005, NCO writes no netCDF4-specific data structures and so always succeeds at writing NETCDF4_CLASSIC files.