发布时间：2025-12-10 19:28:31 浏览次数：2
tfs安装配置教程_ubuntu文件系统文章浏览阅读2.6k次。TFS（TaobaoFileSystem）是一个高可扩展、高可用、高性能、面向互联网服务的分布式文件系统，其设计目标是支持海量的非结构化数据的存储；TFS使用C++语言开发，需要运行在64bitLinuxOS上，TF
TFS（Taobao FileSystem）是一个高可扩展、高可用、高性能、面向互联网服务的分布式文件系统，其设计目标是支持海量的非结构化数据的存储；TFS使用C++语言开发，需要运行在64bit Linux OS上，TFS为淘宝提供海量小文件存储，通常文件大小不超过1M，满足了淘宝对小文件存储的需求，被广泛地应用 在淘宝各项应用中。它采用了HA架构和平滑扩容，保证了整个文件系统的可用性和扩展性。同时扁平化的数据组织结构，可将文件名映射到文件的物理地址，简化了文件的访问流程，一定程度上为TFS提供了良好的读写性能。本文介绍如何在Linux环境编译安装TFS及相关使用说明：
一、安装依赖的软件包:
1、automake TFS基于automake工具构建:
1 | yum install automake.noarch |
2、libtool automake需要使用libtool:
1 | yum install libtool |
3、realine 用于命令行编辑的库：
1 | yum install readline-devel |
4、libz-devel 用于数据压缩/解压缩：
1 | yum install zlib-devel |
5、uuid-devel 用于生成全局唯一ID：
1 2 | yum install e2fsprogs-devel yum install libuuid-devel |
6、tcmalloc google的内存管理库(由玩googl被封，那就暂且跳过吧，可选)
二、安装tb-common-utils
TFS使用tb-common-utils软件包，tb-common-utils包含淘宝使用的基础系统库tbsys和网络库tbnet两个组件；安装tb-common-utils前需要设置环境变量TBLIB_ROOT，tbsys和tbnet将会被安装TBLIB_ROOT对应的路径（必须是绝对路径）下，TFS会在这个路径下查找tbsys、tbnet头文件和库。
设置TBLIB_ROOT环境变量：
1、在~/.bash_profile文件中加入，export TBLIB_ROOT=path_to_tbutil , 然后执行source：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | ~/.bash_profile vi ~/.bash_profile # .bash_profile # Get the aliases and functions if [ -f ~/.bashrc ]; then . ~/.bashrc fi # User specific environment and startup programs PATH=$PATH:$HOME/bin export PATH export TBLIB_ROOT=/usr/local/tb-common-utils # source ~/.bash_profile |
2、下载源码：
1 | # svn co -r 18 http://code.taobao.org/svn/tb-common-utils/trunk tb-common-utils |
注意： 这里不要checkout最新版本，version18以后的修改导致部分接口不能前向兼容
1 2 3 4 5 6 7 8 9 | # cd /usr/local/tb-common-utils/ # sh build.sh checking for C++ compiler default output file name… configure: error: in `/usr/local/tb-common-utils/tbnet’: configure: error: C++ compiler cannot create executables See `config.log’ for more details. make: *** No targets specified and no makefile found.Stop. make: *** No rule to make target `install’.Stop. |
这个错误是因为缺少gcc-c++：
1 | yum install gcc-c++ |
如果一切顺利，tb-common-utils已经安装成功到$TBLIB_ROOT路径下；
三、安装TFS
TFS开源用户大都只使用TFS的基本功能，所以这个版本我们默认只编译TFS的nameserver，dataserver，client和tool，以去除对mysql的依赖，需要使用到rcserver（全局资源管理服务），metaserver(TFS自定义文件名服务）的用户请自行编译安装这两个服务。
下载源码并安装：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 | # svn co http://code.taobao.org/svn/tfs/branches/dev_for_outer_users tfs # cd tfs/ # sh build.sh init + aclocal + libtoolize –force –copy libtoolize: putting auxiliary files in `.’. libtoolize: copying file `./ltmain.sh’ libtoolize: Consider adding `AC_CONFIG_MACRO_DIR([m4])’ to configure.ac and libtoolize: rerunning libtoolize, to keep the correct libtool macros in-tree. libtoolize: Consider adding `-I m4′ to ACLOCAL_AMFLAGS in Makefile.am. libtoolize: `AC_PROG_RANLIB’ is rendered obsolete by `LT_INIT’ + autoconf –force + automake –foreign –copy –add-missing configure.ac:16: installing `./config.guess’ configure.ac:16: installing `./config.sub’ configure.ac:8: installing `./install-sh’ configure.ac:8: installing `./missing’ src/adminserver/Makefile.am: installing `./depcomp’ # ./configure –prefix=/usr/local/taobaoFS –with-release checking for tc_cfree in -ltcmalloc… no configure: error: in `/usr/local/tfs/tfs’: configure: error: tcmalloc link failed (–without-tcmalloc to disable) See `config.log’ for more details. |
注意，这里报错了，是因为之前我们没有安装tcmalloc，所以这里要改一下编译条件：
1 2 3 | # ./configure –prefix=/usr/local/taobaoFS –with-release –without-tcmalloc # make # make install |
–prefix 指定tfs安装路径，默认会被安装到~/tfs_bin目录
–with-release 指定按release版本的参数进行编译，如果不指定这个参数，则会按开发版本比较严格的参数编译，包含-Werror参数，所有的警告都会被当错误，在高版本gcc下会导致项目编译不过，很多开源用户反馈的编译问题都跟这个有关，因为gcc高版本对代码的检查越来越严格，淘宝内部使用的gcc版本是gcc-4.1.2。
四、配置文件：
ns.conf:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 | # cd /usr/local/taobaoFS/conf/ # cat ns.conf | grep -v ^# | grep -v ^$ [public] log_size=1073741824 log_num = 8 log_level=info task_max_queue_size = 10240 port = 8108 work_dir=/usr/local/taobaoFS dev_name= eth0 thread_count =32 ip_addr = 192.168.1.63 [nameserver] safe_mode_time = 360 ip_addr_list = 192.168.1.63|192.168.0.2 group_mask = 255.255.255.255 max_write_timeout = 3 cluster_id = 1 block_max_use_ratio = 98 block_max_size = 75497472 max_replication = 2 min_replication = 2 replicate_ratio = 50 max_write_filecount = 64 use_capacity_ratio = 96 heart_interval = 4 object_dead_max_time = 300 object_clear_max_time = 180 heart_thread_count = 4 heart_max_queue_size = 2048 report_block_thread_count = 6 report_block_max_queue_size = 32 report_block_hour_range = 2~4 report_block_time_interval = 1 repl_wait_time = 180 compact_delete_ratio =10 compact_max_load = 200 compact_hour_range = 1~10 dump_stat_info_interval = 60000000 balance_percent = 0.05 add_primary_block_count = 3 task_percent_sec_size = 200 oplog_sync_max_slots_num = 1024 oplog_sync_thread_num = 1 group_count = 1 group_seq= 0 discard_newblk_safe_mode_time = 360 choose_target_server_random_max_num = 128 |
ds.conf:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 | # cat ds.conf | grep -v ^# | grep -v ^$ [public] log_size=1073741824 log_num = 8 log_level=info task_max_queue_size = 10240 port = 8200 work_dir=/usr/local/taobaoFS dev_name= eth0 thread_count = 32 ip_addr = 192.168.1.64 [dataserver] ip_addr = 192.168.1.63 ip_addr_list = 192.168.1.63|192.168.0.2 port = 8108 heart_interval = 2 check_interval = 2 replicate_threadcount = 2 block_max_size = 75497472 dump_visit_stat_interval = 60 backup_type = 1 mount_name = /data/disk mount_maxsize = 768959044 base_filesystem_type = 1 superblock_reserve = 0 avg_file_size = 15360 mainblock_size = 75497472 extblock_size = 4194304 block_ratio = 0.5 hash_slot_ratio = 0.5 |
第三台服务器的ds配置如下：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 | # cat ds.conf | grep -v ^# | grep -v ^$ [public] log_size=1073741824 log_num = 8 log_level=info task_max_queue_size = 10240 port = 8200 work_dir=/usr/local/taobaoFS dev_name= eth0 thread_count = 32 ip_addr = 192.168.1.66 [dataserver] ip_addr = 192.168.1.63 ip_addr_list = 192.168.1.63|192.168.0.2 port = 8108 heart_interval = 2 check_interval = 2 replicate_threadcount = 2 block_max_size = 75497472 dump_visit_stat_interval = 60 backup_type = 1 mount_name = /data/disk mount_maxsize = 768959044 base_filesystem_type = 1 superblock_reserve = 0 avg_file_size = 15360 mainblock_size = 75497472 extblock_size = 4194304 block_ratio = 0.5 hash_slot_ratio = 0.5 |
五、启动说胆：
1、运行TFS
启动nameserver
执行scripts目录下的tfs：
1 2 | # ./tfs start_ns nameserver is up SUCCESSFULLY pid: 24744 |
2、启动DS
现有TFS可以在一台服务器上启动多个DataServer进程。一般每个DataServer进程负责一个磁盘。
将数据盘格式化成EXT4文件系统，并挂载到/data/tfs1至/data/tfs(i)，其中i为磁盘号。
启动步骤：
A、存储区预分配。执行scripts下的stfs format n (n为挂载点的序号，具体用法见stfs的Usage)。例如stfs format 2,4-6 则会对/data/tfs2,
/data/tfs4,/data/tfs5,/data/tfs6，进行预分配。运行完后会在生成/data/tfs2, /data/tfs4,/data/tfs5,/data/tfs6下预先创建主块，扩展块及相应的统计信息。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 | # ./stfs clear 1 clear ds 1 SUCCESSFULLY [2014-08-07 10:59:11] INFOblockfile_manager.cpp:111 [140398366021408] clear block file system end. mount_point: /data/disk1, ret: 1 # ./stfs format 1 format ds 1 SUCCESSFULLY mount name: /data/disk1 max mount size: 768959044 base fs type: 1 superblock reserve offset: 0 main block size: 75497472 extend block size: 4194304 block ratio: 0.5 file system version: 1 avg inner file size: 15360 hash slot ratio: 0.5 [2014-08-07 12:16:15] INFOblockfile_manager.cpp:1091 [140579583227680] super block mount point: /data/disk1. [2014-08-07 12:16:15] INFOblockfile_manager.cpp:1171 [140579583227680] cal block count. avail data space: 783334178816, main block count: 9338, ext block count: 18676 tag TAOBAO mount time 1407384975 mount desc /data/disk1 max use space 787414061056 base filesystem 1 superblock reserve 0 bitmap start offset 324 avg inner file size 15360 block type ratio 0.5 main block count 9338 main block size 75497472 extend block count 18676 extend block size 4194304 used main block count 0 used extend block count 0 hash slot ratio 0.5 hash slot size 2730 first mmap size 122880 mmap size step 4096 max mmap size 3686400 version 1 [2014-08-07 12:16:15] INFOblockfile_manager.cpp:1213 [140579583227680] cal bitmap count. item count: 28015, slot count: 3502 |
B、运行data server。有两种方法：
通过adminserver来启动dataserver（推荐）: 执行scripts下的./tfs admin_ds
直接启动dataserver，执行scripts下的./tfs start_ds 2,4-6， 则会启动dataserver2，dataserver4，dataserver5，dataserver6
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | # ./tfs start_ds 1 dataserver 1 is up SUCCESSFULLY pid: 2212 # ./tfs start_ds 2 dataserver 2 is up SUCCESSFULLY pid: 28715 # netstat -lantp | grep dataserver tcp00 0.0.0.0:82000.0.0.0:* LISTEN2960/dataserver tcp00 0.0.0.0:82010.0.0.0:* LISTEN2960/dataserver tcp00 192.168.1.64:36866192.168.1.63:8108 ESTABLISHED 2960/dataserver tcp01 192.168.1.64:38846192.168.0.2:8108SYN_SENT2960/dataserver # netstat -lantp | grep dataserver tcp00 0.0.0.0:82030.0.0.0:* LISTEN29636/dataserver tcp00 0.0.0.0:82020.0.0.0:* LISTEN29636/dataserver tcp00 192.168.1.66:56412192.168.1.63:8108 ESTABLISHED 29636/dataserver tcp01 192.168.1.66:49068192.168.0.2:8108SYN_SENT29636/dataserver |
程序基本上启动起来了
第六、相关操作
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 | # /usr/local/taobaoFS/bin/ssm -s 192.168.1.63:8108 show > server -b SERVER_ADDR CNT BLOCK 192.168.1.64:8200194 2846284728482849285028512852285328542855 2856285728582859286028612862286328642865 2866286728682869287028712872287328742875 2876287728782879288028812882288328842885 2886288728882889289028912892289328942895 2896289728982899290029012902290329042905 2906290729082909291029112912291329142915 2916291729182919292029212922292329242925 2926292729282929293029312932293329342935 2936293729382939294029412942294329442945 2946294729482949295029512952295329542955 2956295729582959296029612962296329642965 2966296729682969297029712972297329742975 2976297729782979298029812982298329842985 2986298729882989299029912992299329942995 2996299729982999300030013002300330043005 3006300730083009301030113012301330143015 3016301730183019302030213022302330243025 3026302730283029303030313032303330343035 3036303730383139 192.168.1.66:8202194 2846284728482849285028512852285328542855 2856285728582859286028612862286328642865 2866286728682869287028712872287328742875 2876287728782879288028812882288328842885 2886288728882889289028912892289328942895 2896289728982899290029012902290329042905 2906290729082909291029112912291329142915 2916291729182919292029212922292329242925 2926292729282929293029312932293329342935 2936293729382939294029412942294329442945 2946294729482949295029512952295329542955 2956295729582959296029612962296329642965 2966296729682969297029712972297329742975 2976297729782979298029812982298329842985 2986298729882989299029912992299329942995 2996299729982999300030013002300330043005 3006300730083009301030113012301330143015 3016301730183019302030213022302330243025 3026302730283029303030313032303330343035 3036303730383139 show > server -w SERVER_ADDR CNT WRITABLE BLOCK 192.168.1.64:8200 90 2849285028522856285728592860286228652866 2871287828792884288628872891289528992900 2901290529062907291129122913291429152917 2920292129252927292929302932293429352936 2937293829422944294529462948295029512955 2957295829592960296129642967296829782979 2980298229842985298729882990299329943000 3004300530063007301030123013301430153016 3018302030223025302730293030303530383139 192.168.1.66:8202103 2846284728482851285328542855285828612863 2864286728682869287028722874287528762877 2880288128822883288528882889289028922893 2894289628972898290229032904290829092910 2916291829192922292329242926292829312933 2939294029412943294729492952295329542956 2962296329652966296929702971297229732974 2975297629772981298329862989299129922995 2996299729982999300130023003300830093011 3017301930213023302430263028303130323033 303430363037 show > server -m SERVER_ADDR CNT MASTER BLOCK 192.168.1.64:8200 64 2849285028522856285728592860286228652866 2871287828792884288628872891289528992900 2901290529062907291129122913291429152917 2920292129252927292929302932293429352936 2937299029932994300030043005300630073010 3012301330143015301630183020302230253027 3029303030353038 192.168.1.66:8202 64 2846284728482851285328542855285828612863 2864286728682869287028722874287528762877 2880288128822883288529692970297129722973 2974297529762977298129832986298929912992 2995299629972998299930013002300330083009 3011301730193021302330243026302830313032 3033303430363037 show > machine -a SERVER_IP NUMS UCAP/ TCAP =URBLKCNTLOAD TOTAL_WRITETOTAL_READLAST_WRITELAST_READMAX_WRITE MAX_READ ————— —- —————— ——– —- ———–———-———-—————–——— 192.168.1.64 1 15.16G 729.53G 2% 19410 4.1K 0 0 0 0 00 0 0 0 0 0 192.168.1.66 1 15.16G 729.53G 2% 19410 4.1K 0 3 0 0 00 0 0 0 0 0 Total : 22 30.31G 1.42T 2% 38810 8.3K 0 3 0 0 00 0 show > machine -p SERVER_IP NUMS UCAP/ TCAP =URBLKCNTLOAD LAST_WRITELAST_READMAX_WRITEMAX_READ STARTUP_TIME ————— —- —————— ——– —- ———-————————– ———— 192.168.1.64 1 15.16G 729.53G 2% 19410 00 0 0 0 00 0 2014-08-07 18:57:11 192.168.1.66 1 15.16G 729.53G 2% 19410 00 0 0 0 00 0 2014-08-07 19:00:06 Total : 22 30.31G 1.42T 2% 388100 0 0 0 |
存数据至tfs：
1 2 3 4 5 6 7 8 | # /usr/local/taobaoFS/bin/tfstool -s 192.168.1.63:8108 -i “put /root/haproxy.cfg” 中间省略N行… [2014-08-07 14:49:38] DEBUG tfs_file.cpp:803 [139796598765408] do response success. index: 0, phase: 3, ret: 0, blockid: 2866, fileid: 1, offset: 0, size: 0, crc: -1605709817, inneroffset: 0, filenumber: 4612152245717303297, status: 4, rserver: 192.168.1.64:8200, wserver: 192.168.1.64:8200. put /root/haproxy.cfg => T1YybTByJT1RCvBVdK success. [2014-08-07 14:49:39] INFOtransport.cpp:460 [139796598765408] DELIOC, IOCount:1, IOC:0x26076b0 [2014-08-07 14:49:39] DEBUG socket.cpp:122 [139796598765408] 1?? fd=4, addr=192.168.1.63:8108 [2014-08-07 14:49:39] INFOtransport.cpp:460 [139796598765408] DELIOC, IOCount:0, IOC:0x2609460 [2014-08-07 14:49:39] DEBUG socket.cpp:122 [139796598765408] 1?? fd=6, addr=192.168.1.64:8200 |
从tfs取数据：
1 2 3 4 5 6 7 8 | # /usr/local/taobaoFS/bin/tfstool -s 192.168.1.63:8108 -i “get T1YybTByJT1RCvBVdK /root/test/haproxy.cfg” 中间省略N行… [2014-08-07 15:27:27] DEBUG tfs_file.cpp:225 [139699370776416] file read reach end, offset: 0, size: 1048576 fetch T1YybTByJT1RCvBVdK => /root/test/haproxy.cfg success. [2014-08-07 15:27:28] INFOtransport.cpp:460 [139699370776416] DELIOC, IOCount:1, IOC:0xb6e6b0 [2014-08-07 15:27:28] DEBUG socket.cpp:122 [139699370776416] 1?? fd=4, addr=192.168.1.63:8108 [2014-08-07 15:27:28] INFOtransport.cpp:460 [139699370776416] DELIOC, IOCount:0, IOC:0xb70680 [2014-08-07 15:27:28] DEBUG socket.cpp:122 [139699370776416] 1?? fd=6, addr=192.168.1.66:8202 |
存放大文件至tfs:
1 2 3 4 5 6 7 8 9 | # /usr/local/taobaoFS/bin/tfstool -s 192.168.1.63:8108 -i “putL /root/CentOS-6.5-x86_64-minimal-freewaf-1.2.1-release_21152.iso” 中间省略N行… put /root/CentOS-6.5-x86_64-minimal-freewaf-1.2.1-release_21152.iso => L1YybTByZT1RCvBVdK success. [2014-08-07 17:03:50] INFOtransport.cpp:460 [140184226105184] DELIOC, IOCount:2, IOC:0x194c6b0 [2014-08-07 17:03:50] DEBUG socket.cpp:122 [140184226105184] 1?? fd=4, addr=192.168.1.63:8108 [2014-08-07 17:03:50] INFOtransport.cpp:460 [140184226105184] DELIOC, IOCount:1, IOC:0x194e970 [2014-08-07 17:03:50] DEBUG socket.cpp:122 [140184226105184] 1?? fd=8, addr=192.168.1.66:8202 [2014-08-07 17:03:50] INFOtransport.cpp:460 [140184226105184] DELIOC, IOCount:0, IOC:0x194f930 [2014-08-07 17:03:50] DEBUG socket.cpp:122 [140184226105184] 1?? fd=9, addr=192.168.1.64:8200 |
取大文件：
1 2 3 4 5 6 7 8 9 | # /usr/local/taobaoFS/bin/tfstool -s 192.168.1.63:8108 -i “get L1YybTByZT1RCvBVdK /data/CentOS-6.5-x86_64-minimal-freewaf-1.2.1-release_21152.iso” [2014-08-07 17:08:14] DEBUG tfs_file.cpp:201 [139878759466848] file read reach end, offset: 645128192, size: 0 fetch L1YybTByZT1RCvBVdK => /data/CentOS-6.5-x86_64-minimal-freewaf-1.2.1-release_21152.iso success. [2014-08-07 17:08:15] INFOtransport.cpp:460 [139878759466848] DELIOC, IOCount:2, IOC:0xc5a6b0 [2014-08-07 17:08:15] DEBUG socket.cpp:122 [139878759466848] 1?? fd=4, addr=192.168.1.63:8108 [2014-08-07 17:08:15] INFOtransport.cpp:460 [139878759466848] DELIOC, IOCount:1, IOC:0xc5c7b0 [2014-08-07 17:08:15] DEBUG socket.cpp:122 [139878759466848] 1?? fd=6, addr=192.168.1.64:8200 [2014-08-07 17:08:15] INFOtransport.cpp:460 [139878759466848] DELIOC, IOCount:0, IOC:0xc63790 [2014-08-07 17:08:15] DEBUG socket.cpp:122 [139878759466848] 1?? fd=7, addr=192.168.1.66:8202 |
对比结果：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | [root@IP-1-63 ~]# ll -h /root/ total 828M -rw——-. 1 root root 1.1K Jul 31 10:48 anaconda-ks.cfg -rw-r–r–. 1 root root 3.5M Jul 31 14:16 Atlas-2.1.el6.x86_64.rpm -rw-r–r–1 root root 616M Aug3 13:27 CentOS-6.5-x86_64-minimal-freewaf-1.2.1-release_21152.iso -rw-r–r–. 1 root root 3.1K Jul 31 13:53 haproxy.cfg -rw-r–r–. 1 root root 8.4K Jul 31 10:48 install.log -rw-r–r–. 1 root root 3.4K Jul 31 10:47 install.log.syslog -rw-r–r–1 root root51M Jun 19 06:05 Ipart.apk -rw-r–r–1 root root 137M Mar 22 14:47 lmnp.tar.gz -rw-r–r–1 root root22M Nov 132013 Percona-Server-5.5.31-rel30.3.tar.gz drwxr-xr-x2 root root 4.0K Aug7 15:27 test [root@IP-1-63 ~]# ll -h test/ total 3.5M -rw-r–r– 1 root root 3.5M Aug7 15:26 Atlas-2.1.el6.x86_64.rpm -rw-r–r– 1 root root 3.1K Aug7 15:27 haproxy.cfg [root@IP-1-63 ~]# ll-h /data/ total 616M -rw-r–r–1 root root 616M Aug7 17:08 CentOS-6.5-x86_64-minimal-freewaf-1.2.1-release_21152.iso drwx——. 2 root root16K Jul 31 10:45 lost+found |
对比结果大小相同。
免责声明：本站内容（文字信息+图片素材）来源于互联网公开数据整理或转载，仅用于学习参考，如有侵权问题，请及时联系本站删除，我们将在5个工作日内处理。联系邮箱：303555158#qq.com（把#换成@）
微信便捷交流