فهرست عناوین
چگونه Multipath را در PVM راه‌اندازی کنیم؟
Multipath چیست؟
افزونگی در مسیرهای اتصال داده سرورها به SAN Storage و تشخیص این افزونگی و مدیریت fail شدن مسیرها به نام multipath شناخته می‌شود. انواع روش‌ها برای این موضوع وجود دارد همچون، مسیرهای مختلف به یک کنترلر، مسیرهای مختلف به دو کنترلر. در این مستند نحوه راه‌اندازی multipath در PVM مورد بررسی قرار می‌گیرد.
نصب Multipath در PVM
مراحل ساخت multipath در PVM بصورت زیر است.
ساخت فایل کانفیگ multipath
Copy to Clipboard
یک نمونه ساده که به صورت پیش‌فرض جوابگو است.
Copy to Clipboard
تنظیم شروع خودکار سرویس multipathd و شروع آن به صورت زیر است.
Copy to Clipboard
برای اطمینان از وجود کار کردن درست multipath، دستور زیر را وارد می‌کنیم.
Copy to Clipboard
این دستور توپولوژی وضعیت multipath را نشان خواهد داد
يک نکته در راه‌اندازی اولیه
در راه‌اندازی‌هایی که انجام شد مشخص شد که anaconda در ابتدای راه افتادن سیستم یک فایل کانفیگ می‌سازد که براساس این فایل کانفیگ دستور multipath -l هیچگونه خروجی ندارد.
در ابتدا نصب توصیه شده است که ابتدا فایل /etc/multipath.conf/ حذف گردد.
دیوایس‌های multipath
دیوایس‌ها با WWID یا World Wide Identiier شناخته می‌شوند که در تمام کلاستر این عدد برای هر دیوایس یکتا می‌باشد.
این دیوایس‌ها در مسیر /dev/mapper/ قرار دارند.
دستور multipath -l به شما نشان خواهد داد که هر دیوایس شامل چه دیسک‌هایی است.
چنانچه user_friendly_names فعال باشد، دیوایس‌هایی با نام mpath* ایجاد می‌شوند که به جای WWID می‌توانند مورد استفاده قرار گیرند و در مسیر /dev/mapper/ قرار دارند.
لیست WWIDها همچنین در فایل /etc/multipath/wwids وجود دارد که توسط multipathd و multipath ایجاد و مدیریت می‌شود.
| Component | Description |
|---|---|
| .Reroutes I/O and supports failover for paths and path groups | dm_multipath kernel module |
| .Configures and enables device mapper multipathing
این دستور فایل کانفیگ را چنانچه وجود دارد تغییر می‌دهد و اگر نیست می‌سازد پارامترهای این دستور: mpathconf -h چنانچه این دستور به صورت خالی در خط فرمان وارد شود وضعیت کانفیگ‌های را نشان می‌دهد: mpathconf طبق گفته مستند منبع، تنظیمات پیش‌فرض به صورت کامپایل شده در multipath وجود دارد و چنانچه نیاز به انجام تغییرات در مورد تنظیمات پیش‌فرض بود باید فایل multipath.conf را تغییر داد. ساده ترین محتوای فایل multipath.conf به این شرح است: defaults { user_friendly_names no } blacklist { } |
mpathconf utility |
| Lists and configures multipath devices. Normally started up with /etc/rc.sysinit, it can also be started up by a udev program whenever a .block device is added or it can be run by the initramfs file system |
multipath command |
| Monitors paths; as paths fail and come back, it may initiate path group switches Provides for interactive changes to multipath devices. This must be restarted for any changes to the /etc/multipath.conf file |
multipathd daemon |
| Creates device mapper devices for the partitions on a device It is necessary to use this command for DOS-based partitions with DM-MP. The kpartx is provided in its own package, but the device-mapper-multipath package depends on it. |
kpartx command |
دستور multipath و انجام تغییرات در حین کار بدون restart
تجربه کار نشان داد که در صورت user_friendly_names ابتدا yes باشد و بعد no شود و multipathd هم reload شود نامها user_friendly حذف نمی‌شوند.
همچنین stop کردن multipathd موجب حذف شدن دیوایس‌هایی که ساخته شده است نمی‌شود.
برای حذف دیوایس‌ها و ساخت مجدد آن از روش زیر استفاده کنید:
Copy to Clipboard
از این شیوه برای تغییراتی که بعد از انجام در فایل کانفیگ و reload اعمال نمی‌شوند استفاده کنید.
دقت داشته باشید که در هنگام استفاده از دستور multipath -F سیستم در وضعیت عملیاتی نباشد.
نکات مهم
- هرگونه تغییر در فایل کانفیگ /etc/multipath.conf/ می‌بایست سرویس multipathd راه‌اندازی مجدد یا reload گردد.
- دستور multipathd -k کنسول تعاملی با سرویس multipathd را در اختیار قرار می‌دهد.
فایل تنظیمات /etc/multipath.conf/
این فایل از ۵ قسمت اصل تشکیل می‌شود:
blacklist
.Listing of specific devices that will not be considered for multipath
blacklist_exceptions
.Listing of multipath candidates that would otherwise be blacklisted according to the parameters of the blacklist section
defaults
.General default settings for DM-Multipath
multipaths
Settings for the characteristics of individual multipath devices. These values overwrite what is specified in the defaults and devices sections of the configuration file
devices
Settings for the individual storage controllers. These values overwrite what is specified in the defaults section of the configuration file. If you are using a storage array that is not supported by default, you may need to create a devices subsection for your array.
blacklist_exceptions و Blacklist
نمونه‌هایی از نحوه بلاک کردن به شرح زیر آمده است:
Copy to Clipboard
قابل ذکر است که Exepstion هم به همین طریق قابل انجام است.
قسمت defaults
این قسمت مقادیر پیش‌فرض را برای تمامی multipathها در خود دارد. نمونه‌ای از مقادیر که می‌توانند ذکر شوند و توضیح مقدار آنها به شرح زیر است:
Copy to Clipboard
| Attribute | Description |
|---|---|
| polling_interval | Specifies the interval between two path checks in seconds. For properly functioning paths, the interval between checks will gradually increase to (4 * .polling_interval). The default value is 5 |
| udev_dir | .The directory where udev device nodes are created .The default value is /dev |
| multipath_dir | The directory where the dynamic shared objects are stored. The default value is system dependent, .commonly /lib/multipath |
| find_multipaths | Defines the mode for setting up multipath devices. If this parameter is set to yes, then multipath will not try to create a device for every non-blacklisted path. Instead multipath will create a device only if one of :three conditions are met There are at least two non-blacklisted paths with the – .same WWID The user manually forces the creation of the device – by specifying a device with the multipath .command A path has the same WWID as a multipath device – that was previously created. Whenever a multipath device is created with find_multipaths set, multipath remembers the WWID of the device so that it will automatically create the device again as soon as it sees a path with that WWID. This allows you to have multipath automatically choose the correct paths to make into multipath devices, without having to edit the multipath blacklist. For instructions on the procedure to follow if you have previously created multipath devices when the find_multipaths parameter was not set, see Section 4.2, .” “Configuration File Blacklist .The default value is no |
| verbosity | The default verbosity. Higher values increase the verbosity level. Valid levels are between 0 and 6. The .default value is 2 |
| path_selector | Specifies the default algorithm to use in determining what path to use for the next I/O operation.:Possible values include round-robin 0: Loop through every path in the |
| path_grouping_policy | Specifies the default path grouping policy to apply to :unspecified multipaths. Possible values include .failover: 1 path per priority group .multibus: all valid paths in 1 priority group group_by_serial: 1 priority group per detected .serial number group_by_prio: 1 priority group per path priority value. Priorities are determined by callout programs specified as global, per-controller, or per-multipath .options group_by_node_name: 1 priority group per target node name. Target node names are fetched in /sys/class/fc_transport/target*/node_ .name .The default value is failover |
| getuid_callout | Specifies the default program and arguments to call out to obtain a unique path identifier. An absolute path is .required The default value is /lib/udev/scsi_id .–whitelisted –device=/dev/%n |
| prio | Specifies the default function to call to obtain a path priority value. For example, the ALUA bits in SPC-3 provide an exploitable prio value. Possible values :include .const: Set a priority of 1 to all paths .emc: Generate the path priority for EMC arrays alua: Generate the path priority based on the SCSI-3 .ALUA settings tpg_pref: Generate the path priority based on the .SCSI-3 ALUA settings, using the preferred port bit .ontap: Generate the path priority for NetApp arrays rdac: Generate the path priority for LSI/Engenio .RDAC controller hp_sw: Generate the path priority for Compaq/HP controller in active/standby mode. hds: Generate the path priority for Hitachi HDS Modular storage arrays. The default value is const |
| features | The default extra features of multipath devices. The only existing feature is queue_if_no_path, which is the same as setting no_path_retry to queue. For information on issues that may arise when using this feature, see Section 5.6, “Issues with .” queue_if_no_path feature |
| path_checker | Specifies the default method used to determine the state of the paths. Possible values include: readsector0: Read the first sector of the device. tur: Issue a TEST UNIT READY to the device. emc_clariion: Query the EMC Clariion specific EVPD page 0xC0 to determine the path. hp_sw: Check the path state for HP storage arrays with Active/Standby firmware. rdac: Check the path stat for LSI/Engenio RDAC storage controller.directio: Read the first sector with direct I/O. The default value is directio. |
| failback | .Manages path group failback A value of immediate specifies immediate failback to the highest priority path group that contains active .paths A value of manual specifies that there should not be immediate failback but that failback can happen only .with operator intervention A value of followover specifies that automatic failback should be performed when the first path of a path group becomes active. This keeps a node from automatically failing back when another node .requested the failover A numeric value greater than zero specifies deferred .failback, expressed in seconds .The default value is manual |
| rr_min_io | Specifies the number of I/O requests to route to a path before switching to the next path in the current path group. This setting is only for systems running kernels older than 2.6.31. Newer systems should use .rr_min_io_rq. The default value is 1000 |
| rr_min_io_rq | Specifies the number of I/O requests to route to a path before switching to the next path in the current path group, using request-based device-mapper-multipath. This setting should be used on systems running current kernels. On systems running kernels older than 2.6.31, .use rr_min_io. The default value is 1 |
| rr_weight | If set to priorities, then instead of sending rr_min_io requests to a path before calling path_selector to choose the next path, the number of requests to send is determined by rr_min_io times the path’s priority, as determined by the prio function. If set to uniform, all path weights are equal. The .default value is uniform |
| no_path_retry | A numeric value for this attribute specifies the number of times the system should attempt to use a .failed path before disabling queueing A value of fail indicates immediate failure, without .queueing A value of queue indicates that queueing should not .stop until the path is fixed .The default value is 0 |
| user_friendly_names | If set to yes, specifies that the system should use the /etc/multipath/bindings file to assign a persistent and unique alias to the multipath, in the form of mpathn. If set to no, specifies that the system should use the WWID as the alias for the multipath. In either case, what is specified here will be overridden by any device-specific aliases you specify in the multipaths section of the configuration file. The .default value is no |
| queue_without_daemon | If set to no, the multipathd daemon will disable queueing for all devices when it is shut down. The .default value is no |
| flush_on_last_del | If set to yes, the multipathd daemon will disable queueing when the last path to a device has been .deleted. The default value is no |
| max_fds | Sets the maximum number of open file descriptors that can be opened by multipath and the multipathd daemon. This is equivalent to the ulimit -n command. As of the Red Hat Enterprise Linux 6.3 release, the default value is max, which sets this to the system limit from /proc/sys/fs/nr_open. For earlier releases, if this is not set the maximum number of open file descriptors is taken from the calling process; it is usually 1024. To be safe, this should be set to the maximum number of paths plus 32, if that .number is greater than 1024 |
| checker_timeout | The timeout to use for path checkers that issue SCSI commands with an explicit timeout, in seconds. The default value is taken from .sys/block/sdx/device/timeout |
| fast_io_fail_tmo | The number of seconds the SCSI layer will wait after a problem has been detected on an FC remote port before failing I/O to devices on that remote port. This value should be smaller than the value of dev_loss_tmo. Setting this to off will disable the .timeout. The default value is determined by the OS |
| dev_loss_tmo | The number of seconds the SCSI layer will wait after a problem has been detected on an FC remote port before removing it from the system. Setting this to infinity will set this to 2147483647 seconds, or 68 .years. The default value is determined by the OS |
| hwtable_regex_match | Red Hat Enterprise Linux Release 6.3 and later)) Controls how multipath integrates the device configurations from the devices section of the configuration file with the built-in device .configurations Each device configuration in the devices section of the multipath.conf file will either create its own device configuration or it will modify one of the builtin device configurations. Prior to Red Hat Enterprise Linux 6.3, if the vendor, product, and revision strings in a user’s device configuration exactly matched those strings in a built-in device configuration, the built-in configuration was modified by the options in the user’s configuration. Otherwise. the user’s device .configuration was treated as a new configuration If hwtable_regex_match is set to yes, a regular expression match is used instead. The vendor, product, and revision strings are all regular expressions. The user device configuration values for these options are matched against the built-in device configuration values. This match works the same way that an actual device’s vendor product and revision strings are matched against a device configuration’s strings to see which configuration should be used for the device. If the user’s device configuration matches, then the built-in configuration is modified by the options in the user’s configuration. Otherwise the user’s device configuration is treated as a new .configuration |
| retain_attached_hw_handler | Red Hat Enterprise Linux Release 6.4 and later) If this)
parameter is set to yes and the scsi layer has already |
| detect_prio | Red Hat Enterprise Linux Release 6.4 and later) If this) is set to yes, multipath will first check if the device supports ALUA, and if so it will automatically assign the device the alua prioritizer. If the device doesn’t support ALUA, it will determine the prioritizer as it .always does. The default value is no |
قسمت multipath
مشخصات ویژه هر مسیر multipath در اینجا ذکر می‌شود همچون alias. یک نمونه از این مورد به شرح زیر می‌باشد.
Copy to Clipboard
| Attribute | Description |
|---|---|
| wwid | Specifies the WWID of the multipath device to which the multipath attributes apply. This parameter is mandatory for this .section of the multipath.conf file |
| alias | Specifies the symbolic name for the multipath device to which the multipath attributes apply. If you are using user_friendly_names, do not set this value to mpathn; this may conflict with an automatically assigned user friendly name and .give you incorrect device node names |
| path_grouping_policy | Specifies the default path grouping policy to apply to :unspecified multipaths. Possible values include failover = 1 path per priority group multibus = all valid paths in 1 priority group group_by_serial = 1 priority group per detected serial number group_by_prio = 1 priority group per path priority value group_by_node_name = 1 priority group per target node nam |
| path_selector | Specifies the default algorithm to use in determining what path :to use for the next I/O operation. Possible values include round-robin 0: Loop through every path in the path .group, sending the same amount of I/O to each queue-length 0: Send the next bunch of I/O down the .path with the least number of outstanding I/O requests service-time 0: Send the next bunch of I/O down the path with the shortest estimated service time, which is determined by dividing the total size of the outstanding I/O to .each path by its relative throughput |
| failback | Manages path group failback A value of immediate specifies immediate failback to the .highest priority path group that contains active pathsA value of manual specifies that there should not be immediate failback but that failback can happen only with .operator intervention A value of followover specifies that automatic failback should be performed when the first path of a path group becomes active. This keeps a node from automatically failing .back when another node requested the failover A numeric value greater than zero specifies deferred failback, .expressed in seconds |
| prio | Specifies the default function to call to obtain a path priority value. For example, the ALUA bits in SPC-3 provide an :exploitable prio value. Possible values include .const: Set a priority of 1 to all paths .emc: Generate the path priority for EMC arrays alua: Generate the path priority based on the SCSI-3 ALUA .settings tpg_pref: Generate the path priority based on the SCSI-3 .ALUA settings, using the preferred port bit .ontap: Generate the path priority for NetApp arrays rdac: Generate the path priority for LSI/Engenio RDAC .controller hp_sw: Generate the path priority for Compaq/HP controller .in active/standby mode hds: Generate the path priority for Hitachi HDS Modular .storage arrays |
| no_path_retry | A numeric value for this attribute specifies the number of times the system should attempt to use a failed path before disabling .queueing A value of fail indicates immediate failure, without .queueing A value of queue indicates that queueing should not stop.until the path is fixed |
| rr_min_io | Specifies the number of I/O requests to route to a path before switching to the next path in the current path group. This setting is only for systems running kernels older that 2.6.31. Newer systems .should use rr_min_io_rq. The default value is 1000 |
| rr_min_io_rq | Specifies the number of I/O requests to route to a path before switching to the next path in the current path group, using requestbased device-mapper-multipath. This setting should be used on systems running current kernels. On systems running kernels older .than 2.6.31, use rr_min_io. The default value is 1 |
| rr_weight | If set to priorities, then instead of sending rr_min_io requests to a path before calling path_selector to choose the next path, the number of requests to send is determined by rr_min_io times the path’s priority, as determined by the prio .function. If set to uniform, all path weights are equal |
| flush_on_last_del | If set to yes, then multipath will disable queueing when the last .path to a device has been deleted |
| user_friendly_names | If set to yes, specifies that the system should use the /etc/multipath/bindings file to assign a persistent and unique alias to the multipath, in the form of mpathn. If set to no, specifies that the system should use use the WWID as the alias for the multipath. In either case, what is specified here will be overridden by any device-specific aliases you specify in the .multipaths section of the configuration file |
قسمت devices
این قسمت بر اساس Product ID و Vendor ID تنظیمات مورد نظر را اعمال می‌کند.
این تنظیمات، قسمت defaults را overwrite می‌کند ولی توسط multipaths برای مسیری که این دیوایس عضو آن است overwrite می‌شود.
Copy to Clipboard
| Attribute | Description |
|---|---|
| vendor | Specifies the vendor name of the storage device to which the .device attributes apply, for example COMPAQ |
| product | Specifies the product name of the storage device to which the .device attributes apply, for example HSV110 (C)COMPAQ |
| revision | Specifies the product revision identifier of the storage device |
| product_blacklist | .Specifies a regular expression used to blacklist devices by product |
| hardware_handler | Specifies a module that will be used to perform hardware specific actions when switching path groups or handling I/O errors. Possible values include: 1 emc: hardware handler for EMC storage arrays. 1 alua: hardware handler for SCSI-3 ALUA arrays. 1 hp_sw: hardware handler for Compaq/HP controllers. 1 rdac: hardware handler for the LSI/Engenio RDAC controllers. |
| path_grouping_policy | Specifies the default path grouping policy to apply to :unspecified multipaths. Possible values include failover = 1 path per priority group multibus = all valid paths in 1 priority group group_by_serial = 1 priority group per detected serial number group_by_prio = 1 priority group per path priority value group_by_node_name = 1 priority group per target node name |
| getuid_callout | Specifies the default program and arguments to call out to obtain a .unique path identifier. An absolute path is required |
| path_selector | Specifies the default algorithm to use in determining what path :to use for the next I/O operation. Possible values include round-robin 0: Loop through every path in the path .group, sending the same amount of I/O to each queue-length 0: Send the next bunch of I/O down the .path with the least number of outstanding I/O requests service-time 0: Send the next bunch of I/O down the path with the shortest estimated service time, which is determined by dividing the total size of the outstanding I/O to .each path by its relative throughput |
| path_checker | Specifies the default method used to determine the state of the :paths. Possible values include .readsector0: Read the first sector of the device .tur: Issue a TEST UNIT READY to the device emc_clariion: Query the EMC Clariion specific EVPD .page 0xC0 to determine the path hp_sw: Check the path state for HP storage arrays with .Active/Standby firmware rdac: Check the path stat for LSI/Engenio RDAC storage .controller .directio: Read the first sector with direct I/O |
| features | The extra features of multipath devices. The only existing feature is queue_if_no_path, which is the same as setting no_path_retry to queue. For information on issues that may arise when using this feature, see Section 5.6, “Issues with .” queue_if_no_path feature |
| prio | Specifies the default function to call to obtain a path priority value. For example, the ALUA bits in SPC-3 provide an :exploitable prio value. Possible values include .const: Set a priority of 1 to all paths .emc: Generate the path priority for EMC arraysalua: Generate the path priority based on the SCSI-3 ALUA .settings tpg_pref: Generate the path priority based on the SCSI-3 .ALUA settings, using the preferred port bit .ontap: Generate the path priority for NetApp arrays rdac: Generate the path priority for LSI/Engenio RDAC .controller hp_sw: Generate the path priority for Compaq/HP controller .in active/standby mode hds: Generate the path priority for Hitachi HDS Modular .storage arrays |
| failback | .Manages path group failback A value of immediate specifies immediate failback to the .highest priority path group that contains active paths A value of manual specifies that there should not be immediate failback but that failback can happen only with .operator intervention A value of followover specifies that automatic failback should be performed when the first path of a path group becomes active. This keeps a node from automatically failing .back when another node requested the failover A numeric value greater than zero specifies deferred failback, .expressed in seconds |
| rr_weight | If set to priorities, then instead of sending rr_min_io requests to a path before calling path_selector to choose the next path, the number of requests to send is determined by rr_min_io times the path’s priority, as determined by the prio .function. If set to uniform, all path weights are equa |
| no_path_retry | A numeric value for this attribute specifies the number of times the system should attempt to use a failed path before disabling .queueing A value of fail indicates immediate failure, without.queueing A value of queue indicates that queueing should not stop until .the path is fixed |
| rr_min_io | Specifies the number of I/O requests to route to a path before switching to the next path in the current path group. This setting is only for systems running kernels older that 2.6.31. Newer systems .should use rr_min_io_rq. The default value is 1000 |
| rr_min_io_rq | Specifies the number of I/O requests to route to a path before switching to the next path in the current path group, using requestbased device-mapper-multipath. This setting should be used on systems running current kernels. On systems running kernels older .than 2.6.31, use rr_min_io. The default value is 1 |
| fast_io_fail_tmo | The number of seconds the SCSI layer will wait after a problem has been detected on an FC remote port before failing I/O to devices on that remote port. This value should be smaller than the value of .dev_loss_tmo. Setting this to off will disable the timeout |
| dev_loss_tmo | The number of seconds the SCSI layer will wait after a problem has been detected on an FC remote port before removing it from the system. Setting this to infinity will set this to 2147483647 seconds,or .68 years |
| flush_on_last_del | If set to yes, the multipathd daemon will disable queueing when .the last path to a device has been deleted |
| user_friendly_names | If set to yes, specifies that the system should use the /etc/multipath/bindings file to assign a persistent and unique alias to the multipath, in the form of mpathn. If set to no, specifies that the system should use use the WWID as the alias for the multipath. In either case, what is specified here will be overridden by any device-specific aliases you specify in the multipaths section of the configuration file. The default value is .no |
| retain_attached_hw_handler | Red Hat Enterprise Linux Release 6.4 and later) If this parameter is) set to yes and the scsi layer has already attached a hardware handler to the path device, multipath will not force the device to use the hardware_handler specified by the mutipath.conf file. If the scsi layer has not attached a hardware handler, multipath will .continue to use its configured hardware handler as usual |
| detect_prio | Red Hat Enterprise Linux Release 6.4 and later) If this is set to )
yes, multipath will first check if the device supports ALUA, and if |
