openstack-裸金属ironic创建及调度-3

云盘支持

When user starts bare metal instance with Cinder volume, Nova orchestrates the communication with Cinder and Ironic. The work flow of the boot process is as follows:

  1. (Preparation) Administrator registers a node with initiator information.
  2. User asks Cinder to create a boot volume.
  3. User asks Nova to boot a node from the Cinder volume.
  4. Nova calls Ironic to collect iSCSI/FC initiator information. Ironic collects initiator information and returns it to Nova.
  5. Nova calls Cinder to attach the volume to the node. Cinder attaches the volume to the node and returns connection information which includes target information.
  6. Nova passes the target information for the node to Ironic
  7. Nova calls Ironic to spawn the instance. Ironic prepares the bare metal node to boot from the remote volume which is identified by target information and powers on the bare metal node.

In the work flow above, Nova calls Ironic to get/set initiator/target information (4 and 6) and also administrator calls Ironic to set initiator information (1) but currently Ironic has neither those information nor APIs for them.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
#pip install rbd-iscsi-client==0.1.8 -i https://pypi.tuna.tsinghua.edu.cn/simple
#cinder type-create rbd_iscsi
#cinder type-key rbd_iscsi set volume_backend_name=rbd_iscsi
#cinder create 100 --display-name iscsi_boot1 --volume-type rbd_iscsi --image-id 3ec7c300-78bc-4d4b-bb1d-aaab4b4c35d7
#cinder create 100 --display-name iscsi_boot2 --volume-type rbd_iscsi
#openstack baremetal node set --storage-interface cinder 5e10c067-c0c6-48be-ac6e-749a70045b97
#openstack baremetal node set --property capabilities=iscsi_boot:True 5e10c067-c0c6-48be-ac6e-749a70045b97

这一步可以理解为注册主机，target中host信息，id名称随便起，最大64bytes
#openstack baremetal volume connector create --node 5e10c067-c0c6-48be-ac6e-749a70045b97 --type iqn --connector-id iqn.2021-04.net.*****.sh.cluster.5e10c067-c0c6-48be-ac6e-749a70045b97
为该裸机分配磁盘 volume
#openstack baremetal volume target create --node 5e10c067-c0c6-48be-ac6e-749a70045b97 --type iscsi --boot-index 0 --volume cfa8c986-7a79-48fd-88a0-aa997d579fbf --property auth_method="CHAP" --property auth_username="admin" --property auth_password="admin"
#openstack baremetal volume target create --node 5e10c067-c0c6-48be-ac6e-749a70045b97 --type iscsi --boot-index 1 --volume 4e4e41e3-fbfb-4737-bd93-863f48470244 --property auth_method="CHAP" --property auth_username="admin" --property auth_password="admin"

#openstack server create --image 3ec7c300-78bc-4d4b-bb1d-aaab4b4c35d7 --flavor bms.large --nic net-id=cb1356ca-e79b-4d7b-ad29-3267e4c2bc41 bm01

1
2
3
4
5
6
7
#cinder create 100 --display-name centos7-grub2-iscsi-pwd  --image-id ea4266e4-456e-4fe0-91c9-a8c15dd2d2d6 --volume-type rbd_iscsi
#gwcli
>cd /disks/
>create iscsi-data image=volume-98ebbe5c-a32f-45ee-be04-61c295bcabe4 size=100g
>goto hosts
>create iqn.2021-01.net.*****.sh***:server01
>disk add iscsi-data/volume-98ebbe5c-a32f-45ee-be04-61c295bcabe4 size=100g

常见问题：

  1. volume状态是否是available
1
#cat /data/docker/volumes/ironic_ipxe/_data/78\:ac\:44\:04\:f1\:a9.conf 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
#!ipxe

set attempts:int32 10
set i:int32 0

goto deploy

:deploy
imgfree
kernel  selinux=0 troubleshoot=0 text nofb nomodeset vga=normal console=tty0 console=ttyS0,115200n8 ipa-debug=1 BOOTIF=${mac} ipa-api-url=http://10.218.105.100:6385 initrd=deploy_ramdisk coreos.configdrive=0 || goto retry

initrd  || goto retry
boot

:retry
iseq ${i} ${attempts} && goto fail ||
inc i
echo No response, retrying in {i} seconds.
sleep ${i}
goto deploy

:fail
echo Failed to get a response after ${attempts} attempts
echo Powering off in 30 seconds.
sleep 30
poweroff

:boot_partition
imgfree
kernel no_kernel root={{ ROOT }} ro text nofb nomodeset vga=normal console=tty0 console=ttyS0,115200n8 ipa-debug=1 initrd=ramdisk || goto boot_partition
initrd no_ramdisk || goto boot_partition
boot

:boot_ramdisk
imgfree
kernel no_kernel root=/dev/ram0 text nofb nomodeset vga=normal console=tty0 console=ttyS0,115200n8 ipa-debug=1  initrd=ramdisk || goto boot_ramdisk
initrd no_ramdisk || goto boot_ramdisk
boot

:boot_iscsi
imgfree
set username iqn.2021-04.net.*****.5e10c067-c0c6-48be-ac6e-749a70045b97
set password Cc6BK5uTbCsH6hCF
set initiator-iqn iqn.2021-04.net.*****.5e10c067-c0c6-48be-ac6e-749a70045b97
sanhook --drive 0x80 iscsi:10.206.79.34::3260:1:iqn.2021-04.net.*****.iscsi-gw:ceph-igw || goto fail_iscsi_retry
set username admin
set password admin
sanhook --drive 0x81 iscsi::::: || goto fail_iscsi_retry
set username iqn.2021-04.net.*****.5e10c067-c0c6-48be-ac6e-749a70045b97
set password Cc6BK5uTbCsH6hCF
sanboot --no-describe || goto fail_iscsi_retry

:fail_iscsi_retry
echo Failed to attach iSCSI volume(s), retrying in 10 seconds.
sleep 10
goto boot_iscsi

:boot_whole_disk

bs=16k, iodepth=1

吞吐

顺序读 顺序写 随机读 随机写 混合随机读写
数据云盘sata 246MB 41.8MB 291MB 10.1MB 7799kB/7894kB
物理盘(sas*2 raid1) 584MB 1418MB/s 127MB 65.4MB 40.2MB/40.1MB

IOPS

顺序读 顺序写 随机读 随机写 混合随机读写
数据云盘sata 15.0k 2550 17.8k 618 476/481
物理盘(sas*2 raid1) 35.7k 86.6k 7747 3990 2451/2450

bs=4k,iodepth=1

吞吐

顺序读 顺序写 随机读 随机写 混合随机读写
数据云盘sata 112MB 14.1MB 93.1MB 2490kB 2212kB/2238kB
物理盘(sas*2 raid1) 367MB 372MB 21.4MB 15.0MB 7252kB/7263kB

IOPS

顺序读 顺序写 随机读 随机写 混合随机读写
数据云盘sata 27.3k 3431 22.7k 607 559/565
物理盘(sas*2 raid1) 89.5k 90.8k 5223 3904 1770/1773

吞吐 bs=1024k,iodepth=128,size=50G,ioengine=psync

顺序读 顺序写 随机读 随机写 混合随机读写
iSCSI云盘sata 600G 840MB 85MB 830MB 140MB 135MB/140MB
iSCSI云盘sata 4T 840MB 97MB 870MB 140MB 135MB/140MB
物理盘(sas*2 raid1) 600G 520MB 2650MB 290MB 150MB 95MB/95MB
物理盘(sata*1 raid0) 4T 340MB 140MB 100MB 120MB 55MB/60MB
物理盘(ssd*1 raid0) 800G 280MB 230MB 420MB 350MB 175MB/180MB

IOPS bs=4k,iodepth=128,size=50G,ioengine=psync

顺序读 顺序写 随机读 随机写 混合随机读写
iSCSI云盘sata 600G 25.9k 2663 20k 799 602/609
iSCSI云盘sata 4T 26k 3167 25k 828 600/600
物理盘(sas*2 raid1) 600G 82.0k 92.3k 901 1232 685/694
物理盘(sata*1 raid0) 4T 5.8k 7k 327 276 142/148
物理盘(ssd*1 raid0) 800G 47.9k 6k 46.4k 39.6k 20.2k/20.2k