This action might not be possible to undo. Are you sure you want to continue?
• Issue on page 1 • Environment on page 1 Resolution on page 1 • 1. Introduction on page 1 • 2. Disable the irqbalance service on page 3 • 3. Make a list of numeric interrupts on page 3 • 4. Determine the correct IRQ SMP affinity mask on page 4 • 5. Set the IRQ SMP affinity mask on page 5 • 6. Make IRQ SMP Affinity Mask Settings Persistent on page 5 • 7. Assign Desired Application(s) to the Isolated Processors on page 6 Examples on page 6 • Application already running on page 6 • Start application with specified CPU affinity on page 7 • More information on page 7
• How do I dedicate one or more CPUs on a multi-processor system for exclusive use by one or more specific applications?
• Red Hat Enterprise Linux 4 • Red Hat Enterprise Linux 5 • Red Hat Enterprise Linux 6 • Note: Older releases do not have all of the capabilities discussed in this article.
1. Introduction In some situations it may be desirable to configure a server equipped with multiple processors so that a subset of the total available processors are reserved for use only by the applications specifically assigned to them, and have all other server processes -- including the servicing of hardware interrupts and kernel maintenance tasks such as periodic flushing of the kernel's routing cache -- handled by the remaining processors. This can be useful for
Copyright (c) 2011 by Red Hat, Inc. All rights reserved. 1
2 .7 initrd /initrd-2. For more information on Red Hat Enterprise Linux MRG.0) kernel /vmlinuz-2.el5 ro root=LABEL=/ isolcpus=4.6. For applications requiring realtime operating system support. to isolate CPU cores 5-8 (processor ID numbers 4-7).18-194. it is possible to effectively dedicate a subset of processors to servicing one or more applications. Inc.6.conf file for the desired kernel.6.el5) root (hd0.conf reads: title Red Hat Enterprise Linux Server (2.com/mrg/.0) kernel /vmlinuz-2.el5 ro root=LABEL=/ initrd /initrd-2.6. or when it is desired otherwise that an application always have one or more processors available to it.5.5.18-194. which may help improve performance. Red Hat Enterprise Linux MRG-Realtime is recommended.img Copyright (c) 2011 by Red Hat. and deterministic latency.18-194.18-194.7 to the kernel command line options.6.18-194. While Red Hat Enterprise Linux is not designed to serve as a real-time operating system. add isolcpus=4.img the modified stanza would read: title Red Hat Enterprise Linux Server (2.6.18-194. If the original kernel stanza in /boot/grub/grub.el5. For example. listing each processor to be isolated separated by commas.el5.6.6.How can I dedicate one or more CPUs on a multi-processor system for exclusive use by one or more specific applications? some applications which are highly timing-sensitive.el5) root (hd0. consult your Red Hat sales representative or visit the Red Hat Enterprise Linux MRG product overview page at http://www. Edit the kernel command line in the /boot/grub/grub. All rights reserved.redhat. 1. Add the appropriate isolcpus kernel boot command line parameter 2.
The Symmetric MultiProcessing (SMP) affinity will need to be set for each of these to prevent the isolated CPUs from being assigned to them by the kernel. The interrupts listed are those which have been active since booting the system. The simplest way to prevent the irqbalance daemon from distributing IRQ service tasks to the isolated processors is to disable the irqbalance service. All rights reserved. 3 . nvidia 17: 32 36 39 42 36 29 36 35 IO-APIC-fasteoi HDA Intel 18: 11423 11423 11531 11517 11551 11495 11379 11440 IO-APIC-fasteoi eth0 20: 87758 87742 87837 87780 87764 87794 87861 88025 IO-APIC-fasteoi ehci_hcd:usb1. service irqbalance stop chkconfig --level 12345 irqbalance off 3. ahci NMI: 0 0 0 0 0 0 0 0 Non-maskable interrupts LOC: 3832781 3410057 3135853 1628674 1521292 1545106 1187711 1247285 Local timer interrupts RES: 96836 80994 40640 31803 50957 35632 23327 18266 Rescheduling interrupts CAL: 17378 17232 17098 19941 22691 22167 22265 22218 function call interrupts TLB: 36208 36502 26155 23681 29545 23042 21444 17505 TLB shootdowns TRM: 0 0 0 0 0 0 0 0 Thermal event interrupts THR: 0 0 0 0 0 0 0 0 Threshold APIC interrupts SPU: 0 0 0 0 0 0 0 0 Spurious interrupts ERR: 0 Copyright (c) 2011 by Red Hat.How can I dedicate one or more CPUs on a multi-processor system for exclusive use by one or more specific applications? 2. Disable the irqbalance service The irqbalance service periodically redistributes the servicing of hardware interrupt request (IRQ) signals among available processors to equally balance the load. noting each numeric interrupt listed. uhci_hcd:us 21: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb3 22: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4 23: 6528 6656 6574 6590 6583 6688 6595 6612 IO-APIC-fasteoi uhci_hcd:usb5. Inc. Make a list of numeric interrupts Examine the output of /proc/interrupts. Example output: [root@localhost ~]$ cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 0: 90 0 0 0 1 0 0 0 IO-APIC-edge timer 1: 3 2 1 3 1 1 1 2 IO-APIC-edge i8042 4: 0 0 0 1 0 1 0 1 IO-APIC-edge 6: 0 0 1 0 1 0 0 0 IO-APIC-edge floppy 7: 0 0 0 0 0 0 0 0 IO-APIC-edge parport0 8: 0 0 1 0 0 0 0 0 IO-APIC-edge rtc 9: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi acpi 12: 0 1 0 1 0 1 0 1 IO-APIC-edge i8042 14: 0 0 0 0 0 0 0 0 IO-APIC-edge ata_piix 15: 0 0 0 0 0 0 0 0 IO-APIC-edge ata_piix 16: 168930 168814 168690 168740 168737 168665 168801 168558 IO-APIC-fasteoi firewire_ohci.
10. Copyright (c) 2011 by Red Hat. By listing the contents of the / proc/irq virtual directory. inactive interrupts should also be considered. Finally. note that IRQ numbers 0 and 2 are special. and 20-23. but is used occasionally. Furthermore. we will also assume that IRQ numbers 3. 7. 14. 6-9. and should be omitted from the list of numeric interrupts to set an SMP affinity mask for. etc. 15-18. The following table can be used: Zero-based CPU ID: 7 Decimal Value: 128 6 64 5 32 4 16 3 8 2 4 1 2 0 1 For systems with a larger number of processors. Inc. additional interrupt numbers which do not display in the output of / proc/interrupts can also be identified. we will need to set an SMP affinity mask for active IRQs 1. the decimal values represented by the bit number representing each processor we wish to allow a hardware interrupt to execute on are added together. 4. and have decided for simplicity to have processors 1-4 (ID's 0-3) handle all hardware interrupt service. Determine the correct IRQ SMP affinity mask For simplicity we will work with decimal masks. 11.How can I dedicate one or more CPUs on a multi-processor system for exclusive use by one or more specific applications? Note that the IRQ numbers seen will likely vary from server to server. 12. For example CPU ID 8 = 256. and 19 are also seen when listing the contents of the /proc/irq virtual directory. and can also change on a given server if hardware is added or removed. For the sake of example. To determine the correct mask. So in our example. the decimal value doubles with each subsequently higher processor number. 4. interrupts previously unused become active. 13. The kernel will not allow these two to be changed anyway. 5. Therefore all numeric interrupt numbers between 1-23 (omitting IRQ 0 and 2) will need to have their SMP affinity masks set. 4 . or if options affecting IRQ routing are changed in the server BIOS or on the kernel command line. All rights reserved. This would be useful in the case of a device which is normally inactive. CPU ID 9 = 512.
/etc/sysconfig/irqbalance provides a variable IRQBALANCE_BANNED_CPUS that provides a similar function. but this should be recognized as a hardware or BIOS/firmware implementation problem. For example: # echo "15" > /proc/irq/7/smp_affinity This should be repeated for each numeric interrupt number. they exist outside of /proc/sys and therefore cannot be added to /etc/sysctl. 6. the settings will need to be placed in a file which executes at each server boot. we wish to allow all hardware interrupts be serviced by CPU's 1-4 (CPU ID's 0-3). Note that with some hardware platforms. quad-core machine (8 total cores). where NUMBER is the IRQ number to be set. Set the IRQ SMP affinity mask To set the affinity mask.How can I dedicate one or more CPUs on a multi-processor system for exclusive use by one or more specific applications? In our example. the same exclusion would be: IRQBALANCE_BANNED_CPUS=fc0 5. 5 . For example on a 2-socket. to exclude CPU cores 0 and 1 from irqbalancing. the calculated mask needs to be echoed into /proc/irq/NUMBER/ smp_affinity. but takes arguments in hexadecimal notation. It should be noticed with repeated viewing of /proc/interrupts output that IRQ service counts do not increment on processors excluded by the SMP affinity mask. Inc. Make IRQ SMP Affinity Mask Settings Persistent To make the customized IRQ SMP affinity settings persistent across server reboots. Correct behavior can be verified by monitoring the output of /proc/interrupts.local. While these settings are kernel tunable parameters. such as /etc/rc. the variable would be: IRQBALANCE_BANNED_CPUS=fc On a 4-socket. so we add 1 + 2 + 4 + 8 to arrive at a decimal mask of 15.d/ rc. All rights reserved.conf. quad-core machine (16 total cores). interrupts can still be serviced by CPUs excluded by the defined affinity mask. Copyright (c) 2011 by Red Hat.
as well as in the form of hexadecimal notation bitmasks. Examples Application already running To set an application which is already running. taskset will accept CPU masks in the form of numeric CPU IDs.7 12345 Copyright (c) 2011 by Red Hat. using the taskset utility. Inc. 6 . since the kernel may direct some interrupt work to the isolated processors until the script file containing the custom IRQ SMP affinity settings is executed. All rights reserved. having pid number 12345. which is packaged for Red Hat Enterprise Linux in the schedutils RPM.How can I dedicate one or more CPUs on a multi-processor system for exclusive use by one or more specific applications? It is normal to see the isolated processors initially handle some hardware interrupts after a server boot. Assign Desired Application(s) to the Isolated Processors Application developers can use the sched_setaffinity and sched_getaffinity system function calls directly within an application to set and retrieve the application's CPU affinity. to execute on CPU cores 5-8 (CPU ID's 4-7): # taskset -p -c 4-7 12345 or this would be equally valid syntax: # taskset -p -c 4. Applications can also be set to execute only on a specific processor.6. taskset can be used to modify the CPU affinity of running processes if the program's process id (pid) is known. respectively. or group of processors.5. 7. and can also be used to launch a command or program using the CPU affinity specified on the command line. For simplicity the examples below will utilize numeric CPU IDs.
More information For more information. Inc.3/my-special-program Note: The CPU affinity of a child process created via the fork system function call is inherited from its parent process. All rights reserved. refer to the taskset and sched_setaffinity man pages.How can I dedicate one or more CPUs on a multi-processor system for exclusive use by one or more specific applications? Start application with specified CPU affinity # taskset -c 4-7 /opt/foobar1. 7 . so if the application launches several related processes via fork. they will also run on the isolated CPUs. Copyright (c) 2011 by Red Hat.
This action might not be possible to undo. Are you sure you want to continue?
We've moved you to where you read on your other device.
Get the full title to continue reading from where you left off, or restart the preview.