Cray Linux Environment™ (CLE) 3.1 Software Release Overview Supplement S–2497–3102

TM Cray Linux Environment™ (CLE) 3.1 Software Release Overview Supplement S–2497–3102 © 2010 Cray Inc. All Rights Reserved. This document or parts thereof may not be reproduced in any form unless permitted by contract or by written permission of Cray Inc. U.S. GOVERNMENT RESTRICTED RIGHTS NOTICE The Computer Software is delivered as "Commercial Computer Software" as defined in DFARS 48 CFR 252.227-7014. All Computer Software and Computer Software Documentation acquired by or for the U.S. Government is provided with Restricted Rights. Use, duplication or disclosure by the U.S. Government is subject to the restrictions described in FAR 48 CFR 52.227-14 or DFARS 48 CFR 252.227-7014, as applicable. Technical Data acquired by or for the U.S. Government, if any, is provided with Limited Rights. Use, duplication or disclosure by the U.S. Government is subject to the restrictions described in FAR 48 CFR 52.227-14 or DFARS 48 CFR 252.227-7013, as applicable. Cray, LibSci, and PathScale are federally registered trademarks and Active Manager, Baker, Cascade, Cray Apprentice2, Cray Apprentice2 Desktop, Cray C++ Compiling System, Cray CX, Cray CX1, Cray CX1-iWS, Cray CX1-LC, Cray CX1000, Cray CX1000-C, Cray CX1000-G, Cray CX1000-S, Cray CX1000-SC, Cray CX1000-SM, Cray CX1000-HN, Cray Fortran Compiler, Cray Linux Environment, Cray SHMEM, Cray X1, Cray X1E, Cray X2, Cray XD1, Cray XE, Cray XE6, Cray XMT, Cray XR1, Cray XT, Cray XTm, Cray XT3, Cray XT4, Cray XT5, Cray XT5h , Cray XT5m, Cray XT6, Cray XT6m, CrayDoc, CrayPort, CRInform, ECOphlex, Gemini, Libsci, NodeKARE, RapidArray, SeaStar, SeaStar2, SeaStar2+, Threadstorm, and UNICOS/lc are trademarks of Cray Inc. Linux is a trademark of Linus Torvalds. LSI is a trademark of LSI Logic Corporation. Lustre is a trademark of Oracle and/or its affiliates. Other names may be trademarks of their respective owners. Moab and TORQUE are trademarks of Adaptive Computing Enterprises, Inc. PBS Professional is a trademark of Altair Grid Technologies. UNIX, the “X device,” X Window System, and X/Open are trademarks of The Open Group in the United States and other countries. All other trademarks are the property of their respective owners. RECORD OF REVISION S–2497–3102 Published December 2010 Supports the 3.1.UP02 update release of the Cray Linux Environment (CLE) operating system running on Cray XT and Cray XE systems. S–2497–3101a Published October 2010 Supports the 3.1.UP01 update release of the Cray Linux Environment (CLE) operating system running on Cray XT and Cray XE systems. S–2497–3101 Published September 2010 Supports the 3.1.UP01 update release of the Cray Linux Environment (CLE) operating system running on Cray XT and Cray XE systems. Contents Page Software Enhancements [1] 5 1.1 Software Enhancements in CLE 3.1.UP02 . . . . . . . . . . . . . . . . . 5 1.1.1 DVS POSIX Atomicity . . . . . . . . . . . . . . . . . . . . . 5 1.1.2 Changes to GPCD (Gemini Performance Counters Device) to Improve Performance when Accessing Memory Mapped Registers Using CrayPat . . . . . . . . . . . . . . . . . 6 1.1.3 Repurposed Compute Node Support for PBS Professional . . . . . . . . . . . 7 1.1.4 New xtverifydefaults Command . . . . . . . . . . . . . . . . 7 1.1.5 Core Specialization is NUMA-aware . . . . . . . . . . . . . . . . . 8 1.1.6 Cray XT5 and Cray XT4 Systems Supported . . . . . . . . . . . . . . . 9 1.2 Software Enhancements in CLE 3.1.UP01 . . . . . . . . . . . . . . . . . 10 1.2.1 Gemini Network Resiliency and Congestion Management . . . . . . . . . . . 10 1.2.2 Repurposed Compute Nodes . . . . . . . . . . . . . . . . . . . 11 1.2.3 Lustre Upgraded to Version 1.8.2 . . . . . . . . . . . . . . . . . . 13 1.2.4 Application Completion Reporting . . . . . . . . . . . . . . . . . . 13 1.2.5 Topology and NID Ordering on Cray XE Systems . . . . . . . . . . . . . 14 1.3 Software Enhancements in CLE 3.1.UP00 . . . . . . . . . . . . . . . . . 15 1.4 Bugs Addressed Since the Last Release . . . . . . . . . . . . . . . . . . 16 1.4.1 Resolved Limitations . . . . . . . . . . . . . . . . . . . . . 16 1.4.1.1 CLE 3.1.UP02 . . . . . . . . . . . . . . . . . . . . . . 16 1.4.1.2 CLE 3.1.UP01 . . . . . . . . . . . . . . . . . . . . . . 16 1.5 Compatibilities and Differences . . . . . . . . . . . . . . . . . . . . 17 Support Requirements [2] 19 2.1 Supported Cray System Hardware Platforms . . . . . . . . . . . . . . . . 19 2.2 Supported Software Upgrade Path . . . . . . . . . . . . . . . . . . . 19 2.2.1 System Management Workstation (SMW) Requirements . . . . . . . . . . . . 19 2.2.2 Supported CLE Software Installations . . . . . . . . . . . . . . . . . 19 2.3 Binary Compatibility . . . . . . . . . . . . . . . . . . . . . . . 21 2.4 Additional Software Requirements . . . . . . . . . . . . . . . . . . . 21 2.4.1 Release Level Requirements for Other Cray Software Products . . . . . . . . . . 21 S–2497–3102 3 Cray Linux Environment™ (CLE) 3.1 Software Release Overview Supplement Page 2.4.1.1 Programming Environments Requirements for Migrations . . . . . . . . . . 22 2.4.2 Third-party Software Requirements . . . . . . . . . . . . . . . . . 22 Documentation [3] 23 3.1 Cray-developed Books Provided with This Release . . . . . . . . . . . . . . . 23 3.1.1 CLE Migration Publications . . . . . . . . . . . . . . . . . . . 24 3.2 Changes to Man Pages . . . . . . . . . . . . . . . . . . . . . . 25 3.2.1 New Cray Man Pages CLE 3.1.UP02 . . . . . . . . . . . . . . . . . 25 3.2.2 New Cray Man Pages CLE 3.1.UP01 . . . . . . . . . . . . . . . . . 25 3.2.3 Changed Cray Man Pages . . . . . . . . . . . . . . . . . . . . 25 3.2.3.1 CLE3.1.UP02 . . . . . . . . . . . . . . . . . . . . . . 25 3.2.3.2 CLE 3.1.UP01 . . . . . . . . . . . . . . . . . . . . . . 26 Tables Table 1. Books Provided with This Release . . . . . . . . . . . . . . . . . 23 Figures Figure 1. CLE Migration Roadmap . . . . . . . . . . . . . . . . . . . 20 4 S–2497–3102 Software Enhancements [1] CLE 3.1 software update releases provide bug fixes and a limited set of software enhancements or features. This chapter provides an overview of software enhancements that are introduced in each update release. Software enhancements and features that were introduced with the initial or base CLE 3.1 release are described in Cray Linux Environment (CLE) 3.1 Software Release Overview (S–2425–31), which is also provided with the release package. 1.1 Software Enhancements in CLE 3.1.UP02 1.1.1 DVS POSIX Atomicity Who will use DVS atomic stripe parallel mode? System administrators. How can atomic stripe parallel mode help me? These enhancements can prevent interleaving of data, which could occur when an application performs I/O on a shared file without using file locking on a DVS stripe parallel mount point. What does atomic stripe parallel mode do? Atomic stripe parallel, a Cray DVS mode, adheres to POSIX read/write atomicity rules while still allowing for possible parallelism within a file. It is similar to stripe parallel mode in that the server used to perform the read, write, or metadata operation is selected using an internal hash involving the underlying file or directory inode number and the offset of data into the file relative to the DVS block size. S–2497–3102 5 Cray Linux Environment™ (CLE) 3.1 Software Release Overview Supplement Where can I find more information? For information on Cray DVS, see Introduction to Cray Data Virtualization Service. For information on how to set up Cray DVS, see Installing and Configuring Cray Linux Environment (CLE) Software and Managing System Software for Cray XE and Cray XT Systems. 1.1.2 Changes to GPCD (Gemini Performance Counters Device) to Improve Performance when Accessing Memory Mapped Registers Using CrayPat Who will use this enhancement? End-users. How can this enhancement help me? CLE has been modified to decrease MMR (Memory Mapped Register) access time, thus giving a significant performance improvement when using CrayPat. Users should be able to make traces that involve MMR data throughout their program with less overhead than previous releases. Another direct end-user benefit of this improvement is that Gemini network counter event collection can be done more frequently during application execution. The user can now monitor a half-dozen counters without drastically degrading performance. As a reference, when tracing function enter/return, the overhead to count one Gemini event is approximately equivalent to four processor counter events. What is modified to increase performance? The Gemini application specific integrated circuit contains MMRs, also known as network performance counters, that are used within CrayPat to capture network performance data during trace experiments. The performance improvement introduced in this update release decreases the access time experienced when using CrayPat to view Gemini MMR data. This is done via a modification to a kernel-level interface, gpcd (Gemini Performance Counters Device); thus, end-users do not need to take any further action to take advantage of this benefit. Where can I find more information? For more information on Gemini MMRs used as performance counters, see Using the Cray Gemini Hardware Counters. For more information on CrayPat, see Cray Performance Analysis Tools Release Overview and Installation Guide and Using Cray Performance Analysis Tools. 6 S–2497–3102 Software Enhancements [1] 1.1.3 Repurposed Compute Node Support for PBS Professional Who will use repurposed compute node support for PBS Professional? System administrators. How can this help me? You can potentially improve performance or maintainability of PBS Professional batch system services by moving the MOM node to a dedicated node instead of sharing resources with other services on an existing service node. What is repurposed compute node support for PBS? Beginning with CLE 3.1.UP02, Cray tests and supports repurposing a compute node as a PBS Professional MOM node. CLE 3.1.UP01 introduced new functionality to repurpose compute nodes as service nodes. For the initial release of this feature, support for repurposing compute nodes as a PBS Professional MOM nodes was not available. For more information on repurposed compute nodes, see Repurposed Compute Nodes on page 11. Where can I find more information? For information on how to setup and configure compute nodes as PBS Professional MOM nodes, see Repurposing Compute Nodes as Service Nodes on Cray XE and Cray XT Systems and PBS Professional documentation available from Altair Engineering, Inc. at http://www.altair.com. 1.1.4 New xtverifydefaults Command Who will use this command? System administrators, automatically via the xtopview utility. How can this command help me? The xtopview utility automatically invokes a new command, xtverifydefaults, to verify that the default software version for Cray packages is the same on all service nodes. In rare cases, the shared root links to Cray packages can become inconsistent so that not all service nodes point to the same default software version. This situation can be difficult to detect. By default, the xtopview utility invokes the xtverifydefaults command during exit processing to correct any inconsistent links that may have been generated during the xtopview session. System administrators can also run the command manually from within xtopview. S–2497–3102 7 Cray Linux Environment™ (CLE) 3.1 Software Release Overview Supplement What does this command do? The xtverifydefaults command verifies and optionally fixes the default links in the shared root for Cray system software packages. If the version of software a link points to is different from the version that the default view link of the same name points to, the link is considered inconsistent. The xtverifydefaults command flags and optionally fixes inconsistent links. All of the node and class view default links for Cray packages (e.g. those that start with "cray-") are forced to match the default view default links. Where can I find more information? See the xtverifydefaults(8) man page. 1.1.5 Core Specialization is NUMA-aware Who will use NUMA-aware core specialization? End-users. How can NUMA-aware core specialization help me? This enhancement may help to "even out" the distribution of the application-generated service processes when the aprun -r core specialization option is greater than 1. What is NUMA-aware core specialization? Core specialization may improve application performance by binding system processes and daemons to a set of cores in each processor. Initially, only one specialized core per node was allowed. In CLE 3.1.UP01, this limitation was removed. This feature provides more support for core specialization for specialized cores greater than 1. ALPS allocates specialized cores round-robin across the NUMA nodes on a node (starting with the highest-numbered core on the highest-numbered NUMA node), unless the user specifies the aprun -cc cpu_list option. In that case, specialized cores are allocated from the highest-numbered cores, avoiding the cores in cpu_list. Where can I find more information? For more information on how to use core specialization, see the aprun(1) man page and Workload Management and Application Placement for the Cray Linux Environment. 8 S–2497–3102 Software Enhancements [1] 1.1.6 Cray XT5 and Cray XT4 Systems Supported Who will use this functionality? All users and administrators on Cray XT5 and Cray XT4 systems. What is required to install CLE 3.1 on these systems? Moving from CLE 2.2 to CLE 3.1 requires a migration from SLES 10 SP2 to SLES 11. The migration process involves preserving some of your local configuration data from the CLE 2.2 system set, performing an initial install of CLE 3.1 on a different system set, and finally restoring the preserved configuration data. Lustre file system data is preserved. What prerequisites do I need to know about? The System Management Workstation (SMW) must be running the SMW 5.1 release. To install the SMW 5.1 release on your SMW, you must run release version CLE 2.2.UP03 or later on your Cray XT system. Note: To run SMW 5.1 with CLE 2.2.UP02, you must contact your Cray service representative and request a special CLE 2.2.UP02 release package that includes modifications to the CLE installation software. Where can I find more information? To move from CLE 2.2 to CLE 3.1 on a Cray XT5 or Cray XT4 system, follow Migrating to Cray Linux Environment (CLE) 3.1 Software for Cray XT Systems (S–2489). If you have a Cray XT5 that is being hardware upgraded to a Cray XE system, select either a single phase or a dual phase migration path as follows: For a single phase migration, start with a Cray XT5 system running CLE 2.2 and directly migrate to a Cray XE6 or Cray XE5 system running CLE 3.1 (follow Migrating a Cray XT System with Cray Linux Environment (CLE) 2.2 to a Cray XE System with CLE 3.1 (S–0035) and specific sections in Installing and Configuring Cray Linux Environment (CLE) Software (S–2444) as directed.) S–2497–3102 9 Cray Linux Environment™ (CLE) 3.1 Software Release Overview Supplement For a dual phase migration, first move from CLE 2.2 to CLE 3.1 (follow Migrating to Cray Linux Environment (CLE) 3.1 Software for Cray XT Systems.) Following a successful software migration, upgrade CLE 3.1 for SeaStar systems to CLE 3.1 for Gemini systems along with the hardware replacement (follow Upgrading a Cray XT System with Cray Linux Environment (CLE) 3.1 to a Cray XE System with CLE 3.1) (Deferred). For more information, see Field Notice Number 5673c. 1.2 Software Enhancements in CLE 3.1.UP01 1.2.1 Gemini Network Resiliency and Congestion Management Who will use these enhancements? End-users, system administrators, and site analysts. How can these enhancements help me? Network congestion is inherent in any High Performance Computing (HPC) high speed interconnect. Cray's software and hardware are designed to minimize the effects of network congestion. This release includes system software enhancements, administrative recommendations, and user documentation, which together help to prevent network congestion or to protect user jobs and data if congestion occurs. What are the network resiliency and congestion management enhancements? The CLE 3.1.UP01 and SMW 5.1.UP01 releases provide the following software enhancements and documentation to improve network resiliency and manage network congestion on Cray XE systems with a Gemini based system interconnection network. • New user documentation describes some cases where congestion can occur and how to modify PGAS and SHMEM applications to minimize the potential for inducing network congestion. • The Hardware Supervisory System (HSS) includes new software that monitors the Gemini network and, if necessary, throttles traffic to alleviate the complications due to congestion. For more information, see Field Notice Number 5673c. • System administrators can use new information about Gemini High Speed Network (HSN) routing to identify and potentially modify unrouteable network configurations. 10 S–2497–3102 Software Enhancements [1] • To improve network resiliency on the HSN, system software on the L0 automatically brings up lanes that have been downed. • System software changes include selective link speed increases and improvements to the HSN routing algorithm (zone routing). These adjustments can improve global HSN communication performance by as much as 15 to 25 percent. For more information, contact your Cray service representative. 1.2.2 Repurposed Compute Nodes Who will use this feature? System administrators and site analysts. How can repurposed compute nodes help me? You can improve performance or maintainability of some Cray system services by moving the service to a dedicated node instead of sharing resources with other services on an existing service node. You do not need to make hardware configuration changes (swapping modules) to configure additional nodes with a service node role; you can implement a flexible approach to service node configuration on an as-needed basis. The addition of service nodes can increase throughput for some functions. For example, multiple DVS servers working in parallel can increase I/O throughput; multiple MOM nodes can allow additional jobs to run in parallel. For some services, configuring additional service nodes can increase redundancy and minimize the negative impact of a failed service node. S–2497–3102 11 Cray Linux Environment™ (CLE) 3.1 Software Release Overview Supplement What does this feature do? The CLE 3.1.UP01 release includes new functionality that supports booting compute node hardware with service node images. By using this functionality to change the role of a compute node, you can add additional service nodes for services that do not require external connectivity. Some services on Cray systems have resource requirements or limitations (for example, memory, processing power or response time) that you can address by configuring a dedicated service node, such as a Cray Data Virtualization Service (Cray DVS) node or a batch system management (MOM) node. On Cray XE and Cray XT systems, service I/O node hardware (on a service blade) is equipped with Peripheral Component Interconnect (PCI) protocol card slots to support external devices. Compute node hardware (on a compute blade) does not have PCI slots. For services that do not require external connectivity, you can configure the service to run on a single, dedicated compute node and avoid using traditional service I/O node hardware. When you configure a node on a compute blade to boot a service node image and perform a service node role, that node is referred to as a repurposed compute node. Cray tests and supports the following services on repurposed compute nodes: DVS servers, Moab TORQUE MOM nodes, and PBS Professional MOM nodes. Note: Support for repurposing a compute node as a PBS Professional MOM node was originally deferred but is now supported. Where can I find more information? A new Cray publication, Repurposing Compute Nodes as Service Nodes on Cray XE and Cray XT Systems (S–0029–3101) describes the feature and provides implementation procedures. Note: The information in this new document supersedes procedures to manually configure compute nodes as compute node root servers for DSL (using DVS) in the following documentation: • Installing and Configuring Cray Linux Environment (CLE) Software (S–2444–31) • Managing System Software for Cray XE and Cray XT Systems (S–2393–31) 12 S–2497–3102 Software Enhancements [1] 1.2.3 Lustre Upgraded to Version 1.8.2 Who will use this feature? All users and administrators on Cray systems that use the Lustre file system. How can this new version of Lustre help me? A large number of existing problems are fixed with this new version of Lustre. What does upgrading to Lustre 1.8.2 do? When you update a CLE 3.0 or CLE 3.1 system to CLE 3.1.UP01, your Lustre file system software is automatically upgraded from version 1.8.1 to version 1.8.2. This change is transparent to both users and administrators. Where can I find more information? Refer to http://wiki.lustre.org. For information about upgrades to Lustre 1.8 from Lustre 1.6, see Cray Linux Environment (CLE) 3.1 Software Release Overview (S–2425–31). 1.2.4 Application Completion Reporting Who will use this feature? System administrators and site analysts. How can ACR help me? Application Completion Reporting (ACR) provides a way to track, manage, and report application data, which gives system administrators an effortless way to see how system resources are being allocated. Data provided by ACR reports can make it easier to see where system resources are being used so that recommendations can be made for application improvement, different allocation schemes, or use of different time slices. S–2497–3102 13 Cray Linux Environment™ (CLE) 3.1 Software Release Overview Supplement What does ACR do? ACR extends the data persistence schema from Cray Management Services (CMS) and provides three commands to examine and report application data. • Use the mzjob command to examine reservation data; you can specify the format of the search output so that you can use it in other programs or scripts. • Use the mzreport command to examine application completion status information; the command supports claims (applications), jobs (reservations), or user completion status. • Use mz2attr to read and display node attributes. ACR requires CLE 3.1.UP01 or later and SMW 5.1.UP01 or later; support for this feature was deferred in earlier SMW and CLE releases. Where can I find more information? Using Cray Management Services (CMS) (S–2484–5101) and the mzjob(8), mzreport(8) and mz2attr(8) man pages. 1.2.5 Topology and NID Ordering on Cray XE Systems Who will use this feature? End-users, site-analysts, and system administrators. How can this feature help me? The Application Level Placement Scheduler (ALPS) "xyz-by2" node ordering feature provides performance improvements for larger applications that run on Cray XE systems with no user interaction. System administrators can implement the new ordering with only a single configuration option change to the /etc/sysconfig/alps file on the boot node. What does this feature do? The node ordering functionality directs ALPS to use a node ordering option that results in better application performance for larger applications run on Cray XE systems. 14 S–2497–3102 Software Enhancements [1] Cray's first method for application placement on nodes along the interconnect torus was to use the node ID (NID) number list and place applications on nodes in this numerical order. Later, the simple "xyz" placement method was added, which reordered the sequence of NID numbers, used in assigning placements, in a natural 3D communications sequence (that is, following the torus, not the chassis and cabinet layout). The original numerical ordering method provides good "packing" for small numbers of nodes, but for large applications, there are "clumps" of nodes that spread across the machine, which causes inter-application traffic contention on the interconnect. With the "xyz" placement, available with previous releases of CLE, the applications are better isolated, but smaller applications are spread more thinly, losing some of the advantage. New with this release, the "xyz-by2" NID reordering method combines the incidental small node packing of the simple NID number method with the inter-application interaction reduction of the "xyz" method. To implement this node ordering, system administrators must set the ALPS_NIDORDER="-O2" parameter in the /etc/sysconfig/alps configuration file and make identical changes to the file of the same name on the shared root. If configured, a custom ordering is created when ALPS is started on the boot node, triggered in the /etc/init.d/alps script file, and used by the apbridge component of ALPS. You should change the ALPS_NIDORDER parameter only at system reboot time. Previously, this feature was supplied to selected customers via a software patch. While the internal mechanism used to implement this feature remains the same, the configuration method has changed. The preferred method is to set the ALPS_NIDORDER parameter in the /etc/sysconfig/alps file. The configuration instructions provided with the patch are obsolete and should no longer be used. Where can I find more information? The apbridge(8) man page was updated for the CLE 3.1.UP01 release. 1.3 Software Enhancements in CLE 3.1.UP00 For information about feature content in the initial or base CLE 3.1 release (CLE 3.1.UP00), see Cray Linux Environment (CLE) 3.1 Software Release Overview (S–2425–31). S–2497–3102 15 Cray Linux Environment™ (CLE) 3.1 Software Release Overview Supplement 1.4 Bugs Addressed Since the Last Release The primary purpose of each CLE 3.1 update package is to provide fixes for the release. The list of customer-filed bug reports that were closed with CLE 3.1 releases is included in the CLE 3.1 Errata; this document is provided with the release package. 1.4.1 Resolved Limitations The following features or capabilities were identified as limitations with an earlier CLE 3.1 release and were described in the CLE 3.1 Limitations document. These issues are now resolved in the update release indicated. 1.4.1.1 CLE 3.1.UP02 1.4.1.1.1 Checkpoint/Restart Supported with Cray XT Systems Running CLE 3.1 The CLE 3.1.UP01 Limitations document states that with CLE 3.1, Checkpoint/Restart is not functional on Cray XT systems. The CLE 3.1.UP02 update release removes this limitation. For additional information, see Managing System Software for Cray XE and Cray XT Systems for more information on Checkpoint/Restart. 1.4.1.2 CLE 3.1.UP01 1.4.1.2.1 LSI RDAC Support Beginning with the CLE 3.1.UP01 update release, the integrated LSI Redundant Disk Array Controller (RDAC) code is updated to version V09.03.0C00.0214 to support all models and firmware releases of LSI hardware. For additional information about support for multipathing I/O in the CLE 3.1 release, see Cray Linux Environment (CLE) 3.1 Software Release Overview (S–2425–31), Section 2.15.1 "Support for Multipathing I/O"; This document is also provided with the release package. Note: Replace the last paragraph in the referenced section with the following, corrected, statement regarding performance impact: Multipathing functionality does not negatively impact I/O performance during normal system operations. In the event of a failure, Multipathing functionality may impact performance depending on the multipath configuration, failure scenario and I/O workload on your system. 16 S–2497–3102 Software Enhancements [1] 1.4.1.2.2 Number of Cores Support for Core Specialization The Cray Linux Environment (CLE) 3.1 Software Release Overview (S–2425–31) states that with CLE 3.1, only r=1 is supported for the Core Specialization feature. The CLE 3.1.UP01 update release removes this limitation. You can specify a value for cores greater than or equal to one for the aprun -r cores option to enable core specialization. For additional information, see the Workload Management and Application Placement for the Cray Linux Environment and the aprun(1) man page. 1.4.1.2.3 Huge Page Support for Both Dynamically and Statically Linked SHMEM, UPC and CAF Applications on Cray XE Systems Beginning with CLE 3.1.UP01 release, SHMEM, Unified Parallel C (UPC), and Co-Array Fortran (CAF) applications on Cray XE systems may be either statically or dynamically linked with libhugetlbfs. For additional information, see the intro_hugepages(1) man page that is released in the xt-asyncpe 4.2 package. 1.5 Compatibilities and Differences The README document that is included with the release package describes compatibility issues and functionality changes that you should be aware of after you install a CLE 3.1 update release on a Cray system that was running an earlier version of the CLE 3.1 release; for example, a CLE 3.1.UP01 to CLE 3.1.UP02 update installation. For information about compatibility issues and functionality changes after you migrate or upgrade from the CLE 2.2 or CLE 3.0 releases, see the Compatibilities and Differences chapter in Cray Linux Environment (CLE) 3.1 Software Release Overview (S–2425–31). The README document also includes additional documentation or changes to the documentation identified after the documentation for this release was packaged. S–2497–3102 17 Cray Linux Environment™ (CLE) 3.1 Software Release Overview Supplement 18 S–2497–3102 Support Requirements [2] 2.1 Supported Cray System Hardware Platforms The CLE 3.1.UP02 update release supports Cray XE6, Cray XE5, Cray XE5m, Cray XT6, Cray XT6m, Cray XT5, Cray XT5m, Cray XT4 systems. 2.2 Supported Software Upgrade Path 2.2.1 System Management Workstation (SMW) Requirements The System Management Workstation (SMW) must be Restriction of Hazardous Substances (RoHS) compliant and must be running the SMW 5.1.UP02 release before you install the CLE 3.1.UP02 update release package. For additional information, see the SMW README document included with the SMW release package. Note: If you have a Cray XT system, you must run release version CLE 2.2.UP03 or later to install the SMW 5.1 release on your SMW. To run SMW 5.1 with CLE 2.2.UP02, contact your Cray service representative and request a special CLE 2.2.UP02 release package that includes modifications to the CLE installation software. 2.2.2 Supported CLE Software Installations The CLE 3.1 release supports four types of software installations. CLE software installation types are described in Cray Linux Environment (CLE) 3.1 Software Release Overview (S–2425–31). Figure 1 visualizes the upgrade, update, migration, and installation processes for various Cray hardware paths. The CLE 3.1.UP02 update release supports the following software installations: S–2497–3102 19 Cray Linux Environment™ (CLE) 3.1 Software Release Overview Supplement Figure 1. CLE Migration Roadmap H40-6064 Maintenance and Replacement Procedures for Cray XT5, XT5m, XE5, and XE5m Systems H40-6075 Maintenance and Replacement Procedures for Cray XT6, XT6m, XE6, and XE6m Systems S-0020 Upgrading a Cray XT System with Cray Linux Environment (CLE) 3.1 to a Cray XE System with CLE 3.1 S-0033 Migrating from System Management Workstation (SMW) 4.0 to SMW 5.1 Software S-0035 Migrating a Cray XT System with Cray Linux Environment (CLE) 2.2 to a Cray XE System with CLE 3.1 S-2444 Installing and Configuring Cray Linux Environment (CLE) Software S-2480 Installing Cray System Management Workstation (SMW) Software S-2489 Migrating to Cray Linux Environment (CLE) 3.1 Software for Cray XT Systems S-0033 S-2489 S-0033 S-0035 S-2444 H40-6064 S-2480 S-2444 S-0020 H40-6064 for XT5 H40-6075 for XT6 CLE 2.2 / SMW 4.0 SeaStar Interconnect SIO Service Blades CLE 3.0 / SMW 5.0 SeaStar Interconnect SIO Service Blades CLE 3.1 / SMW 5.1 SeaStar Interconnect SIO Service Blades CLE 3.1 / SMW 5.1 Gemini Interconnect XIO Service Blades CLE 3.1 / SMW 5.1 SeaStar Interconnect SIO Service Blades CLE 3.1 / SMW 5.1 SeaStar Interconnect SIO Service Blades CLE 3.1 / SMW 5.1 Gemini Interconnect XIO Service Blades Start Finish Books Needed • Perform SMW upgrade first • Requires a CLE upgrade installation • Lustre is preserved • Requires a CLE upgrade installation after hardware change • Lustre is preserved • Perform SMW migration first (Includes a fresh SMW installation) • Requires a fresh CLE installation • Lustre is preserved Future XE hardware upgrade Book Number Book Title • Initial or new software installations on the following platforms: Cray XE6, Cray XE6m, Cray XT6, Cray XT6m, Cray XE5, Cray XE5m, Cray XT5, and Cray XT4 systems. • Update installations from CLE 3.1.UP00 and CLE 3.1.UP01. • Upgrade installations from CLE 3.0 on Cray XT6 systems. • Upgrade software installations on Cray XT5 or Cray XT6 systems running CLE 3.1 that have been hardware upgraded to Cray XE systems. Note: This type of installation requires that you install 3.1.UP02 or later on your Cray XE system after upgrading the hardware. 20 S–2497–3102 Support Requirements [2] • (Deferred Implementation) Migration installations from CLE 2.2 on Cray XT4 and Cray XT5 systems. Note: This type of installation requires an initial installation with modifications to preserve Lustre file systems. • Migration installations from CLE 2.2 on Cray XT5 systems that have been hardware upgraded to Cray XE6 or Cray XE5 systems. Note: This type of installation requires an initial installation with modifications to preserve Lustre file systems and reconfigure CLE for Gemini. System administrators may want to advise users on compatibility of binaries between the various migration, update, and upgrade paths. Please refer to Cray Linux Environment (CLE) Software Release Overview for more information on binary compatibility when installing CLE 3.1. 2.3 Binary Compatibility A previous version of the Cray Linux Environment (CLE) Software Release Overview stated that application binaries built using dynamic libraries did not require recompiling. The language in that statement remains accurate. However, applications written in C++ or Partitioned Global Address Space (PGAS) programming models must be recompiled when migrating applications. 2.4 Additional Software Requirements 2.4.1 Release Level Requirements for Other Cray Software Products Cray Linux Environment (CLE) 3.1 Software Release Overview (S–2425–31) includes a table of other Cray software products supported with CLE 3.1 and the minimum release level required for verified compatibility. The information in this table applies to CLE 3.1 update releases with the following exceptions: • The System Management Workstation (SMW) must be upgraded to the SMW 5.1.UP02 release before you install the CLE 3.1.UP02 update release package. For more information, see the README document included with the SMW release package. • The System Management Workstation (SMW) must be upgraded to the SMW 5.1.UP01 release before you install the CLE 3.1.UP01 update release package. For more information, see the README document included with the SMW release package. S–2497–3102 21 Cray Linux Environment™ (CLE) 3.1 Software Release Overview Supplement • Cray Application Developer's Environment (CADE) must be upgraded to the most current version (at time of release this is 5.06) to ensure compatibility with the CLE 3.1.UP02 release package. For CADE release information, see Cray Application Developer's Environment Installation Guide (S–2465). Support for other Cray software products is provided in the form of updates to the latest released version only. Unless otherwise noted in the associated release documentation, Cray recommends that you continue to upgrade these releases as updates become available. 2.4.1.1 Programming Environments Requirements for Migrations Perform the following programming environments changes after migrations involving upgrade installations of Cray XT systems from CLE 3.0 to 3.1 or Cray XT to Cray XE systems hardware running CLE 3.1: Uninstall and Reinstall: • PGI • Intel • Cray Performance Tools • Cray MPT (Message Passing Toolkit) — Cray MPT 5.0 or later is required for Gemini-based systems • Chapel For upgrades to the Gemini-based interconnect, Cray recommends you remove the following: • SeaStar RPMs for Cray MPT • Portals RPMs Cray also recommends that you use the most recent version level of the Cray Compiling Environment as the default. 2.4.2 Third-party Software Requirements Cray Linux Environment (CLE) 3.1 Software Release Overview (S–2425–31) includes a section that lists third-party software requirements for the CLE 3.1 release; this information applies to CLE 3.1 update releases. Information regarding supported and certified batch system software release levels is available on the CrayPort website at http://crayport.cray.com. Click on 3rd Party Batch SW in the menu bar. Cray recommends that you continue to upgrade these products as updates become available. 22 S–2497–3102 Documentation [3] 3.1 Cray-developed Books Provided with This Release Table 1 list the books provided with the CLE 3.1 release and indicates which books are new or revised with this update release. The most recent version of each book is provided with the release package. For information about additional documentation resources and accessing documentation, see Cray Linux Environment (CLE) 3.1 Software Release Overview (S–2425–31), which is also provided with the release package. Table 1. Books Provided with This Release Book Title Number Updated Cray Linux Environment (CLE) Software Release Overview Supplement (this document) S–2497–3102 Yes Cray Linux Environment (CLE) Software Release Overview S–2425–31 No Installing and Configuring Cray Linux Environment (CLE) Software S–2444–3102 Yes Managing System Software for Cray XE and Cray XT Systems S–2393–3102 Yes Managing Lustre for the Cray Linux Environment (CLE) S–0010–31 No Introduction to Cray Data Virtualization Service S–0005–3102 Yes Writing a Node Health Checker (NHC) Plugin Test S–0023–3101 No Workload Management and Application Placement for the Cray Linux Environment S–2496–3102 Yes Using the GNI and DMAPP APIs S–2446–3102 Yes Repurposing Compute Nodes as Service Nodes on Cray XE and Cray XT Systems S–0029–3101 No Lustre Operations Manual S–6540–1813 No CrayDoc Installation and Administration Guide S–2340–411 No S–2497–3102 23 Cray Linux Environment™ (CLE) 3.1 Software Release Overview Supplement Book Title Number Updated Upgrading a Cray XT System with Cray Linux Environment (CLE) 3.1 to a Cray XE System with CLE 3.1 S–0020–3102 New Migrating a Cray XT System with Cray Linux Environment (CLE) 2.2 to a Cray XE System with CLE 3.1 S–0035–3102 New Migrating to Cray Linux Environment (CLE) 3.1 Software for Cray XT Systems S–2489–3102 New 3.1.1 CLE Migration Publications The following publications provide procedures for CLE 3.1 upgrade or migration scenarios. • If you have a new system, follow Installing and Configuring Cray Linux Environment (CLE) Software (S–2444) to perform an initial installation of the CLE software. • If you have a system that is already running CLE 3.1, follow Installing and Configuring Cray Linux Environment (CLE) Software (S–2444) to install a new CLE 3.1 update release. • If you have a Cray XT6 system running CLE 3.0, follow Installing and Configuring Cray Linux Environment (CLE) Software (S–2444) to upgrade to CLE 3.1. • If you have a Cray XT5 or Cray XT6 system running CLE 3.1 and your system is being hardware upgraded to a Cray XE system, follow Upgrading a Cray XT System with Cray Linux Environment (CLE) 3.1 to a Cray XE System with CLE 3.1 (S–0020). • (Deferred Implementation) If you have a Cray XT4 or Cray XT5 system running CLE 2.2, follow Migrating to Cray Linux Environment (CLE) 3.1 Software for Cray XT Systems (S–2489) to install CLE 3.1 for SeaStar systems. • If you have a Cray XT5 system running CLE 2.2 and your system is being hardware upgraded to a Cray XE6 or Cray XE5 system, follow Migrating a Cray XT System with Cray Linux Environment (CLE) 2.2 to a Cray XE System with CLE 3.1 (S–0035) and specific sections in Installing and Configuring Cray Linux Environment (CLE) Software (S–2444) as directed. 24 S–2497–3102 Documentation [3] 3.2 Changes to Man Pages 3.2.1 New Cray Man Pages CLE 3.1.UP02 • xtverifydefaults(8): Verifies and fixes inconsistent system default links within the shared root. 3.2.2 New Cray Man Pages CLE 3.1.UP01 • xtorder2db(8): Converts a text file to values in the od_allocator_id field of the processor table in the Service Database (SDB). • xtdb2order(8): Converts the processor table od_allocator_id field in the SDB to a text file. 3.2.3 Changed Cray Man Pages 3.2.3.1 CLE3.1.UP02 • apbridge(8): Updated to add -Oy and other NID ordering clarifications from CLE3.1.UP01 • aprun(1): Updated hugepage information; updated -r option; added text for -T; revised -t option text • apstat(1): Updated with text to describe unbound and pendCancel States. • xtopview(8): Invokes xtverifydefaults during exit processing. • xtcheckhealth(8): Updated new and changed exit codes in the RETURN VALUES section • xtcleanup_after(8): Updated new and changed exit codes in the RETURN VALUES section • rsipd.conf(5): Changed initial_lease_time option default setting to 60; system administrators may wish to change the value of initial_lease_time in /etc/opt/cray/rsipd/rsipd.conf from its default of 60 seconds to a larger value (600 seconds is suggested) to strike a balance between clients renewing their leases too often and clients taking a long time to recover after an RSIP server outage. S–2497–3102 25 Cray Linux Environment™ (CLE) 3.1 Software Release Overview Supplement 3.2.3.2 CLE 3.1.UP01 • aprun(1): Includes descriptions of the following environment variables: APRUN_DEFAULT_MEMORY, APRUN_XFER_LIMITS and APRUN_SYNC_TTY. • apbridge(8): Includes a new -O2 option for the ALPS_NIDORDER parameter to provide the "xyz-by2" NID reordering method. • basil(7): Corrected ENGINE query output; marked -s1, -sn, and -S option not for exclusive use; changed reservation_mode default to unset, use apsched default; changed gemini in 1.1 to router_id. • CLEinstall.conf(5): Includes new default network mask for Cray XE systems; clarifies that sdbnode_failover_interface must be set even if SDB node failover is not configured. • xtcleanup_after(8): Includes a new -i option to allow callers to specify a string in place of the APID when printing to the console. • xtnodestat(8): Includes a detailed description of the legend; explains how repurposed compute nodes are displayed. 26 S–2497–3102