Last updated: 2021-04-15 17:07:45

    Overview

    COSFS allows you to mount COS buckets locally and work with the objects in Tencent Cloud COS in the same way as you do with a local file system. COSFS supports the following features:

    • Most features of the POSIX file system, such as reading/writing files, operations on directories/links, permission management, and uid/gid management.
    • Multipart upload of large files.
    • Data verification with MD5.
    • Upload data to COS using COS Migration or COSCMD.

    Limitations

    COSFS is built on S3FS. As the disk is required for COSFS′s read and write operations, COSFS is only suitable for simple management of the mounted files and does not support all features of a local file system. Besides, it cannot outperform Cloud Block Storage (CBS) or Cloud File Storage (CFS). To use COSFS, note that:

    • Randomly writing data or appending data to a file may lead to the re-download/re-upload of the entire file. To avoid this, you can use a CVM in the same region as the bucket to accelerate the upload and download.
    • When a COS bucket is mounted to multiple clients, you need to coordinate the behaviors of these clients, for example, to prevent the clients from simultaneously writing data to the same file.
    • Rename operation on a file/folder is not atomic.
    • For metadata operations such as list directory, COSFS performs unsatisfactorily as it requires remote access to the COS server.
    • COSFS does not support hard link, and is inapplicable to the scenarios involving high-concurrency read/write operations.
    • Mounting and unmounting files cannot be performed on the same mounting point at the same time. You can use the cd command to switch to another directory and then mount and unmount the files at the mounting point.

    Operating Environments

    Ubuntu, CentOS, SUSE, and macOS

    Installation

    You can install COSFS with an installation package or by compiling the source code.

    Method 1: Install with an installation package

    Note:

    This installation method supports only Ubuntu and CentOS.

    Ubuntu

    1. Download the COSFS installation package for your OS.
      wget https://github.com/tencentyun/cosfs/releases/download/v1.0.19/cosfs_1.0.19-ubuntu16.04_amd64.deb
      
    2. Install the package.
      sudo dpkg -i cosfs_1.0.19-ubuntu16.04_amd64.deb
      

    CentOS

    1. Install dependencies.
      sudo yum install  libxml2-devel libcurl-devel -y
      
    2. Download the COSFS installation package for your OS.
      wget https://github.com/tencentyun/cosfs/releases/download/v1.0.19/cosfs-1.0.19-centos7.0.x86_64.rpm
      
    3. Install the package.
      rpm -ivh cosfs-1.0.19-centos7.0.x86_64.rpm
      

    Method 2: Install by compiling the source code

    Note:

    This installation method supports Ubuntu, CentOS, SUSE, and macOS.

    1. Install the dependency software

    The compilation and installation of COSFS depend on the software packages such as automake, git, libcurl-devel, libxml2-devel, fuse-devel, make, and openssl-devel. The following describes how to install dependency software on Ubuntu, CentOS, SUSE, and macOS:

    • Install dependency software on the Ubuntu system:

      sudo apt-get install automake autotools-dev g++ git libcurl4-gnutls-dev libfuse-dev libssl-dev libxml2-dev make pkg-config fuse
      
    • Install dependency software on the CentOS system:

      sudo yum install automake gcc-c++ git libcurl-devel libxml2-devel fuse-devel make openssl-devel fuse
      
    • Install dependency software on the SUSE system:

      sudo zypper install gcc-c++ automake make libcurl-devel libxml2-devel openssl-devel pkg-config
      
    • Install dependency software on the macOS system:

      brew install automake git curl libxml2 make pkg-config openssl 
      brew cask install osxfuse
      

    2. Obtain the source code

    Download the COSFS Source Code from GitHub to a specified directory. The following uses /usr/cosfs as an example. You can use another directory as needed.

    git clone https://github.com/tencentyun/cosfs /usr/cosfs
    

    3. Compile and install COSFS

    Open the installation directory, and execute the following command to compile and install COSFS:

    cd /usr/cosfs
    ./autogen.sh
    ./configure
    make
    sudo make install
    cosfs --version  #View the COSFS version number
    

    4. Troubleshooting configure issues

    Messages displayed during the configure operation vary depending on the OS. If your FUSE version is earlier than 2.8.4, the following error message will be displayed:

    checking for common_lib_checking... configure: error: Package requirements (fuse >= 2.8.4 libcurl >= 7.0 libxml-2.0 >= 2.6) were not met:
      Requested 'fuse >= 2.8.4' but version of fuse is 2.8.3 
    

    In this case, you need to manually install fuse 2.8.4 or above as shown below:

    yum -y remove fuse-devel
    wget https://github.com/libfuse/libfuse/releases/download/fuse_2_9_4/fuse-2.9.4.tar.gz
    tar -zxvf fuse-2.9.4.tar.gz
    cd fuse-2.9.4
    ./configure
    make
    make install
    export PKG_CONFIG_PATH=/usr/lib/pkgconfig:/usr/lib64/pkgconfig/:/usr/local/lib/pkgconfig
    modprobe fuse   #Mount the fuse's kernel module
    echo "/usr/local/lib" >> /etc/ld.so.conf
    ldconfig   #Update the dynamic link library
    pkg-config --modversion fuse  #View the fuse version number. If "2.9.4" is displayed, fuse 2.9.4 is installed successfully. 
    
    • Install FUSE 2.8.4 or later on the SUSE system manually, as shown below:

      Note:

      During installation, you need to comment out the content of line 222 in example/fusexmp.c by using /*content*/. Otherwise, an error will be reported when you use Make.

      zypper remove fuse libfuse2
      wget https://github.com/libfuse/libfuse/releases/download/fuse_2_9_4/fuse-2.9.4.tar.gz
      tar -zxvf fuse-2.9.4.tar.gz
      cd fuse-2.9.4
      ./configure
      make 
      make install
      export PKG_CONFIG_PATH=/usr/lib/pkgconfig:/usr/lib64/pkgconfig/:/usr/local/lib/pkgconfig
      modprobe fuse   #Mount the fuse's kernel module
      echo "/usr/local/lib" >> /etc/ld.so.conf
      ldconfig   #Update the dynamic link library
      pkg-config --modversion fuse   #View the fuse version number. If "2.9.4" is displayed, fuse 2.9.4 is installed successfully. 
      
    • When the "configure" operation is performed on macOS, the following may be displayed:

      configure: error: Package requirements (fuse >= 2.7.3 libcurl >= 7.0 libxml-2.0 >2.6 libcrypto >= 0.9) were not met
      No package 'libcrypto' found
      

      In this case, you need to set the variable PKG_CONFIG_PATH, so that the pkg-config tool can find openssl. The command is as follows:

      brew info openssl 
      export PKG_CONFIG_PATH=/usr/local/opt/openssl/lib/pkgconfig #You may need to modify this command based on the message displayed for the previous command.
      

    Usage

    1. Configure the key file

    Write the bucket information in the /etc/passwd-cosfs file, including the bucket name (in BucketName-APPID format) <SecretId>, as well as <SecretKey>, and use colons (:) to separate them. To avoid compromising your key, you need to set permissions for the key file to 640. You can run the following command to configure the /etc/passwd-cosfs key file:

    sudo su  # Switch to the root account to modify the /etc/passwd-cosfs file. Skip this step if you already logged in with the root account
    echo <BucketName-APPID>:<SecretId>:<SecretKey> > /etc/passwd-cosfs
    chmod 640 /etc/passwd-cosfs
    

    Note:

    You need to replace the content enclosed in <> with the actual information.

    • <BucketName-APPID> indicates the name of the bucket. For more information, please see Bucket Naming Conventions.
    • <SecretId> and <SecretKey> are information about the key, which can be obtained and created at Manage API Key in the CAM console.
    • You can configure the key in $HOME/.passwd-cosfs. Alternatively, you can run -opasswd_file=[path] to specify the directory of the key file and then set permissions of the key file to 600.

    Sample:

    echo examplebucket-1250000000:AKIDHTVVaVR6e3****:PdkhT9e2rZCfy6**** > /etc/passwd-cosfs
    chmod 640 /etc/passwd-cosfs
    

    Note:

    If your COSFS version is v1.0.5 or earlier, the configuration file format is as follows:

    <BucketName>:<SecretId>:<SecretKey>
    
    
    ### 2. Run the tool
    You can run the following command to mount the bucket configured in the key file to a specified directory:
    
    ```shell
    cosfs <BucketName-APPID> <MountPoint> -ourl=<CosDomainName> -odbglevel=info -oallow_other
    

    Where,

    • <MountPoint> is the mount point, for example, /mnt.
    • <CosDomainName> is the bucket′s access endpoint formatted as http://cos.<Region>.myqcloud.com (applicable to XMLs API and should not carry the bucket name). Note that <Region> is the abbreviation for the region, such as ap-guangzhou and eu-frankfurt. For more information about the region abbreviations, please see Regions and Access Endpoints.
    • -odbglevel specifies the log level.
    • -oallow_other allows other users to access the mount point.

    Sample:

    mkdir -p /mnt/cosfs
    cosfs examplebucket-1250000000 /mnt/cosfs -ourl=http://cos.ap-guangzhou.myqcloud.com -odbglevel=info -onoxattr -oallow_other
    

    Note:

    If your COSFS is v1.0.5 or earlier, use the following command for the mount:

    cosfs <APPID>:<BucketName> <MountPoint> -ourl=<CosDomainName> -oallow_other
    
    
    
    #### 3. Unmount a bucket
    
    Unmount a bucket using the following commands:
    
    ```shell
    Method 1: Use `fusermount -u /mnt, fusermount` to unmount a FUSE file system 
    Method 2: Use `umount -l /mnt`. The unmount operation will be performed when no program is using any file in the file system.
    Method 3: Use `umount /mnt`. If any program is using a file in the file system during the unmount, an error will be reported.
    

    Common Mounting Options

    -omultipart_size=[size]

    Specifies the size (in MB) of each part for the multipart upload. It is 10 MB by default. Up to 10,000 parts are allowed for a file in a multipart upload. If the file is larger than 100 GB (10 MB * 10000), you need to adjust this parameter accordingly.

    -oallow_other

    Allows other users to access the folder to which the bucket is mounted.

    -odel_cache

    By default, to ensure optimal performance, the COSFS does not clear local cached data after a bucket is unmounted. To enable the COSFS to automatically clear cached data upon its exit, you can add this option during mounting.

    -onoxattr

    Disables getattr/setxattr. For the COSFS earlier than 1.0.9, you cannot set or obtain extended attributes. If the use_xattr option is used during mounting, the files may fail to be copied to the bucket.

    -opasswd_file=[path]

    Specifies the path for the COSFS key file. You need to set the permission for the key file to 600.

    -odbglevel=[dbg|info|warn|err|crit]

    Sets the log level for COSFS. Valid values are info, dbg, warn, err, and crit. You are advised to set it to info in the production environment, and dbg for debugging. If you do not clear system logs regularly, or numerous logs will be generated due to a huge access volume, you can set it to err or crit.

    -oumask=[perm]

    Removes the permission of a specified type of users to operate files in the mounting destination directory. For example, when -oumask=755, the permission for the mounting destination directory is changed to 022.

    -ouid=[uid]

    Allows the user whose id is [uid] to access all the files in the mounting destination directory without being restricted by the file permission bits.
    You can obtain the uid of a user using the id command id -u username. For example, you can execute id -u user_00 to obtain the uid of user_00.

    FAQs

    If you have any questions about COSFS, please see COSFS FAQs.