FreeNAS有mirror的存储池,想用它作为统一存储各服务数据的位置;既提供了高可用性,又方便备份和管理。
三种方案:
切换到Neon,cloud-native的存算分离,后端使用远端object storage
直接通过iSCSI挂载远端块设备
(新增)通过NFSv4.2直接挂载
配置
Neon + MinIO
MinIO作为自带的插件运行在FreeNAS上,向外提供兼容S3协议的API;Neon运行在应用服务器上,配置page server使用MinIO作为后端。pageserver_config/pageserver.toml:
broker_endpoint='http://storage_broker:50051'
pg_distrib_dir='/usr/local/'
# Initial configuration file created by 'pageserver --init'
listen_pg_addr='0.0.0.0:6400'
listen_http_addr='0.0.0.0:9898'
control_plane_api='http://0.0.0.0:6666' # No storage controller in docker compose, specify a junk address
control_plane_emergency_mode=true
virtual_file_io_mode="buffered" # the CI runners where we run the docker compose tests have slow disks
#checkpoint_distance = 268435456 # in bytes
#checkpoint_timeout = '10m'
#gc_period = '1 hour'
#gc_horizon = 67108864
#max_file_descriptors = 100
remote_storage={ endpoint='http://nas.dappur.net:9001', bucket_name='neon', bucket_region='eu-north-1', prefix_in_bucket='/pageserver' }iSCSI挂载远端块设备
确认发现IQN
sudo iscsiadm -m discovery -t st -p nas
[nas]:3260,-1 iqn.2005-10.org.freenas.ctl:testvol-share
[nas]:3260,-1 iqn.2005-10.org.freenas.ctl:iserver-db
连接target
dappur@iserver:~/openproject/neon$ sudo iscsiadm -m node -T iqn.2005-10.org.freenas.ctl:iserver-db -p nas -l
Logging in to [iface: default, target: iqn.2005-10.org.freenas.ctl:iserver-db, portal: nas,3260]
Login to [iface: default, target: iqn.2005-10.org.freenas.ctl:iserver-db, portal: nas,3260] successful.
dappur@iserver:~/openproject/neon$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
loop0 7:0 0 105.1M 1 loop /snap/core/17284
loop1 7:1 0 105.2M 1 loop /snap/core/17292
loop2 7:2 0 63.8M 1 loop /snap/core20/2769
loop3 7:3 0 63.8M 1 loop /snap/core20/2866
loop4 7:4 0 55.5M 1 loop /snap/core18/2979
loop5 7:5 0 8.3M 1 loop /snap/gost/539
loop6 7:6 0 55.5M 1 loop /snap/core18/2999
loop7 7:7 0 92.2M 1 loop /snap/lxd/40111
loop9 7:9 0 50.1M 1 loop /snap/snapd/27710
loop10 7:10 0 9M 1 loop /snap/gost/559
loop11 7:11 0 92.2M 1 loop /snap/lxd/40364
loop12 7:12 0 50.1M 1 loop /snap/snapd/27591
sda 8:0 0 64G 0 disk
├─sda1 8:1 0 1M 0 part
└─sda2 8:2 0 64G 0 part /
sdb 8:16 0 16G 0 disk
sr0 11:0 1 1024M 0 rom格式化块设备为简单的ext4。因为postgres自带日志和一致性事务,为避免重复开销,关闭ext4的journal。
dappur@iserver:/mnt$ mkfs.ext4 -O ^has_journal /dev/sdb
mke2fs 1.47.0 (5-Feb-2023)
mkfs.ext4: Permission denied while trying to determine filesystem size
dappur@iserver:/mnt$ sudo mkfs.ext4 -O ^has_journal /dev/sdb
mke2fs 1.47.0 (5-Feb-2023)
Discarding device blocks: done
Creating filesystem with 4194304 4k blocks and 1048576 inodes
Filesystem UUID: da10fa1c-e5d0-4570-b1a6-a375c2bd796f
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000
Allocating group tables: done
Writing inode tables: done
Writing superblocks and filesystem accounting information: done最后更改postgres配置文件以使用挂载点作为新的数据库位置 /etc/postgresql/18/main/postgresql.conf:
data_directory = '/mnt/db/pgdata' # iSCSI disk for benchmark通过NFSv4.2挂载
新建一个db-data dataset使用16KB+lz4来匹配数据库的固定定长写入。Postgres固定使用8K页大小,但是FreeNAS的最小推荐配置是16KB,加之可能有后续扩展使用MySQL的需求,而其使用的是16K固定页大小;于是折中使用了16KB,并配合lz4来缓和postgres的写放大。
另新建一个db-waldataset来存储WAL,其使用128KB的record size来优化WAL的连续写入。
使用initdb --waldir来指定WAL目录。
sudo -u postgres /usr/lib/postgresql/18/bin/initdb -D /mnt/app-storage/db-data --waldir=/mnt/app-storage/db-walNAS挂载方面,使用v4.2协议支持DEALLOCATE回收空间,使用hard+tcp选项确保数据落盘,wsize=131072减小网络开销。
nas.dappur.net:/mnt/Workspace/Applications/db-data on /mnt/app-storage/db-data type nfs4 (rw,relatime,vers=4.2,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.16.18,local_lock=none,addr=192.168.16.8)
nas.dappur.net:/mnt/Workspace/Applications/db-wal on /mnt/app-storage/db-wal type nfs4 (rw,relatime,vers=4.2,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.16.18,local_lock=none,addr=192.168.16.8)数据库配置方面,由于NFS直接导出ZFS文件(而不像iSCSI有ext4的额外层次),得益于ZFS提供的原子写入,我们可以在postgresql.conf中设置full_page_writes = off(以及将来在my.cnf中设置innodb_doublewrite = 0)来避免重复开销。
性能测试
Setup
本地硬盘使用NVMe SSD,远端硬盘是mirror的HDD。
FreeNAS和应用服务器都是虚拟机,运行在同一台物理机上(ESXi),通过本地vSwitch相连。
测试使用postgres自带的pgbench,运行两项测试
并发事务测试
pgbench -c 4 -j 2 -t 1000 bench30秒持续测试
pgbench -c 4 -j 2 -T 30 bench
测试运行三轮并取最好成绩
benchmark初始化使用pgbench -i -s 50 bench
原版postgresql使用18,Neon使用17
本地硬盘 (baseline)
postgres@iserver:~$ pgbench -i -s 50 bench
dropping old tables...
NOTICE: table "pgbench_accounts" does not exist, skipping
NOTICE: table "pgbench_branches" does not exist, skipping
NOTICE: table "pgbench_history" does not exist, skipping
NOTICE: table "pgbench_tellers" does not exist, skipping
creating tables...
generating data (client-side)...
vacuuming...
creating primary keys...
done in 10.48 s (drop tables 0.00 s, create tables 0.00 s, client-side generate 7.73 s, vacuum 0.28 s, primary keys 2.46 s).
postgres@iserver:~$ pgbench -c 4 -j 2 -t 1000 bench
pgbench (18.4 (Ubuntu 18.4-1.pgdg24.04+1))
starting vacuum...end.
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 50
query mode: simple
number of clients: 4
number of threads: 2
maximum number of tries: 1
number of transactions per client: 1000
number of transactions actually processed: 4000/4000
number of failed transactions: 0 (0.000%)
latency average = 1.374 ms
initial connection time = 13.475 ms
tps = 2910.555712 (without initial connection time)
postgres@iserver:~$ pgbench -c 4 -j 2 -T 30 bench
pgbench (18.4 (Ubuntu 18.4-1.pgdg24.04+1))
starting vacuum...end.
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 50
query mode: simple
number of clients: 4
number of threads: 2
maximum number of tries: 1
duration: 30 s
number of transactions actually processed: 88648
number of failed transactions: 0 (0.000%)
latency average = 1.353 ms
initial connection time = 15.273 ms
tps = 2956.132833 (without initial connection time)Neon + MinIO
postgres@iserver:~$ pgbench -h localhost -p 55433 -U cloud_admin -i -s 50 bench
Password:
dropping old tables...
NOTICE: table "pgbench_accounts" does not exist, skipping
NOTICE: table "pgbench_branches" does not exist, skipping
NOTICE: table "pgbench_history" does not exist, skipping
NOTICE: table "pgbench_tellers" does not exist, skipping
creating tables...
generating data (client-side)...
vacuuming...
creating primary keys...
done in 36.38 s (drop tables 0.00 s, create tables 0.02 s, client-side generate 7.13 s, vacuum 11.30 s, primary keys 17.93 s).
postgres@iserver:~$ pgbench -h localhost -p 55433 -U cloud_admin -c 4 -j 2 -t 1000 bench
Password:
pgbench (18.4 (Ubuntu 18.4-1.pgdg24.04+1), server 17.5)
starting vacuum...end.
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 50
query mode: simple
number of clients: 4
number of threads: 2
maximum number of tries: 1
number of transactions per client: 1000
number of transactions actually processed: 4000/4000
number of failed transactions: 0 (0.000%)
latency average = 5.468 ms
initial connection time = 28.326 ms
tps = 731.569702 (without initial connection time)
postgres@iserver:~$ pgbench -h localhost -p 55433 -U cloud_admin -c 4 -j 2 -T 30 bench
Password:
pgbench (18.4 (Ubuntu 18.4-1.pgdg24.04+1), server 17.5)
starting vacuum...end.
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 50
query mode: simple
number of clients: 4
number of threads: 2
maximum number of tries: 1
duration: 30 s
number of transactions actually processed: 19473
number of failed transactions: 0 (0.000%)
latency average = 6.159 ms
initial connection time = 24.827 ms
tps = 649.478169 (without initial connection time)
postgres@iserver:~$虽然latency和tps都不占优,但是得益于本地WAL的acknowledgement,数据库初始化时client-side generation非常快,7.73s的时间和baseline无异。然而,后端传输的巨大开销导致vacuum和primary keys极度缓慢。
iSCSI
postgres@iserver:~$ pgbench -i -s 50 bench
dropping old tables...
NOTICE: table "pgbench_accounts" does not exist, skipping
NOTICE: table "pgbench_branches" does not exist, skipping
NOTICE: table "pgbench_history" does not exist, skipping
NOTICE: table "pgbench_tellers" does not exist, skipping
creating tables...
generating data (client-side)...
vacuuming...
creating primary keys...
done in 35.75 s (drop tables 0.00 s, create tables 0.01 s, client-side generate 29.19 s, vacuum 0.70 s, primary keys 5.84 s).
postgres@iserver:~$ pgbench -c 4 -j 2 -t 1000 bench
pgbench (18.4 (Ubuntu 18.4-1.pgdg24.04+1))
starting vacuum...end.
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 50
query mode: simple
number of clients: 4
number of threads: 2
maximum number of tries: 1
number of transactions per client: 1000
number of transactions actually processed: 4000/4000
number of failed transactions: 0 (0.000%)
latency average = 4.541 ms
initial connection time = 15.015 ms
tps = 880.926488 (without initial connection time)
postgres@iserver:~$ pgbench -c 4 -j 2 -T 30 bench
pgbench (18.4 (Ubuntu 18.4-1.pgdg24.04+1))
starting vacuum...end.
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 50
query mode: simple
number of clients: 4
number of threads: 2
maximum number of tries: 1
duration: 30 s
number of transactions actually processed: 24804
number of failed transactions: 0 (0.000%)
latency average = 4.838 ms
initial connection time = 14.096 ms
tps = 826.835802 (without initial connection time)注意到average latency和initial connection time都更低,但是初始化数据库时生成数据非常慢(29.19秒),可能是由于频繁的sync导致的;但是vacuum和primary key没有落后baseline太多。
NFS v4.2
postgres@iserver:~$ pgbench -i -s 50 bench
dropping old tables...
NOTICE: table "pgbench_accounts" does not exist, skipping
NOTICE: table "pgbench_branches" does not exist, skipping
NOTICE: table "pgbench_history" does not exist, skipping
NOTICE: table "pgbench_tellers" does not exist, skipping
creating tables...
generating data (client-side)...
vacuuming...
creating primary keys...
done in 37.15 s (drop tables 0.00 s, create tables 0.02 s, client-side generate 25.58 s, vacuum 3.09 s, primary keys 8.47 s).
postgres@iserver:~$ pgbench -c 4 -j 2 -t 1000 bench
pgbench (18.6 (Ubuntu 18.6-1.pgdg24.04+2))
starting vacuum...end.
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 50
query mode: simple
number of clients: 4
number of threads: 2
maximum number of tries: 1
number of transactions per client: 1000
number of transactions actually processed: 4000/4000
number of failed transactions: 0 (0.000%)
latency average = 2.811 ms
initial connection time = 23.342 ms
tps = 1423.166446 (without initial connection time)
postgres@iserver:~$ pgbench -c 4 -j 2 -T 30 bench
pgbench (18.6 (Ubuntu 18.6-1.pgdg24.04+2))
starting vacuum...end.
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 50
query mode: simple
number of clients: 4
number of threads: 2
maximum number of tries: 1
duration: 30 s
number of transactions actually processed: 34457
number of failed transactions: 0 (0.000%)
latency average = 3.480 ms
initial connection time = 25.919 ms
tps = 1149.370766 (without initial connection time)初始化的用时和其他的网络方案接近。average latency是三者中最好的,tps也相应来到了1423,约为baseline的一半。initial connection的用时劣于iSCSI两倍,但略优于Neon,毕竟NFS的I/O延迟在二者之中。总体上来讲,该方案兼具部署的简易性和较好的连续事务性能。
其他尝试
md write-behind
store WAL locally as a fastpath for immediate acknoledgement and transfer the actual update in the background with a small lag
However, it breaks sequential write semantics and will corrupt the database at crash (no crash consistency). write-behind doesn't guarantee the same sync order as the arrival of writes.
DRBD (Distributed Replicated Block Device)
Linux kernel module, need to run two individual kernels. To connect to iSCSI, it needs another VM dedicated for write-behind, which exposes a DRBD port and forward requests to FreeNAS through iSCSI. This introduces extra complexity, overhead, risk of unavailiability, bond to kernel compatibility.