To Next Page

To Previous Page

Chapter 4. Central processor complex I/O

system structure

This chapter describes the I/O system structure and connectivity options that are available on

z13s servers.

This chapter includes the following sections:

򐂰 Introduction to the InfiniBand and PCIe for I/O infrastructure

򐂰 I/O system overview

򐂰 I/O drawer

򐂰 PCIe I/O drawer

򐂰 PCIe I/O drawer and I/O drawer offerings

򐂰 Fanouts

򐂰 I/O features (cards)

򐂰 Connectivity

򐂰 Parallel Sysplex connectivity

򐂰 Cryptographic functions

򐂰 Integrated firmware processor

򐂰 Flash Express

򐂰 10GbE RoCE Express

򐂰 zEDC Express

Default Chapter1
- Front Cover1
- Table of Contents5
Figures17
Notices21
- Trademarks22
IBM Redbooks Promotions23
- Authors25
Preface25
- Comments Welcome27
- Now You Can Become a Published Author, too27
- Stay Connected to IBM Redbooks28
Chapter 1. Introducing IBM Z13S Servers29
- Overview of IBM Z13S Servers30
- Z13S Servers Highlights31
- Z13S Technical Overview39
- Hardware Management Consoles and Support Elements55
- IBM Z Bladecenter Extension (Zbx) Model 00455
  - Blades56
  - IBM Websphere Datapower Integration Appliance XI50 for Zenterprise56
- IBM Z Unified Resource Manager57
- IBM Dynamic Partition Manager57
- Operating Systems and Software58
  - Supported Operating Systems58
  - IBM Compilers61
Chapter 2. Central Processor Complex Hardware Components63
- Frame and Drawers64
  - The Z13S Frame64
- Z13S Frame: Rear and Front View64
  - Pcie I/O Drawer and I/O Drawer Features66
- Z13S (One CPC Drawer) I/O Drawer Configurations66
- Z13S (Two CPC Drawers) I/O Drawer Configurations66
- Processor Drawer Concept67
- Model N10 Components (Top View)68
- Model N20 Components (Top View)69
- CPC Drawer (Front View)69
- CPC Drawer Logical Structure70
  - CPC Drawer Interconnect Topology71
  - Oscillator71
- Drawer to Drawer Communication71
  - System Control72
- Oscillators Cards72
- Conceptual Overview of System Control Elements73
  - CPC Drawer Power74
- Single Chip Modules74
- Redundant Dcas and Blowers for CPC Drawers74
  - Processor Units and System Control Chips75
  - Processor Unit (PU) Chip75
- Single Chip Modules (PU SCM and SC SCM) N20 CPC Drawer75
- PU Chip Floorplan76
  - Processor Unit (Core)77
  - PU Characterization78
- PU Core Floorplan78
  - System Control Chip79
- SC Chip Diagram79
  - Cache Level Structure80
- Memory81
  - Memory Subsystem Topology82
- CPC Drawer Memory Topology82
  - Redundant Array of Independent Memory83
- RAIM Configuration Per Node83
  - Memory Configurations84
- Model N10 Memory Plug Locations84
- Model N20 One Drawer Memory Plug Locations86
- Model N20 Two Drawer Memory Plug Locations88
- Memory Allocation Diagram90
  - Memory Upgrades91
  - Preplanned Memory91
- Reliability, Availability, and Serviceability92
  - RAS in the CPC Memory Subsystem92
  - General Z13S RAS Features93
- Connectivity93
- Model N10 Drawer: Location of the Pcie and IFB Fanouts94
- Model N20 Two CPC Drawer: Locations of the Pcie and IFB Fanouts94
  - Redundant I/O Interconnect95
- Redundant I/O Interconnect for I/O Drawer96
- Model Configurations97
- Redundant I/O Interconnect for Pcie I/O Drawer97
  - Upgrades99
- Z13S Upgrade Paths99
  - Concurrent PU Conversions100
  - Model Capacity Identifier100
  - Model Capacity Identifier and MSU Values101
  - Capacity Backup102
  - On/Off Capacity on Demand and Cps104
- Power and Cooling105
  - Power Considerations105
  - High-Voltage DC Power106
  - Internal Battery Feature106
  - Power Capping106
  - Power Estimation Tool106
  - Cooling Requirements107
- Summary of Z13S Structure107
Chapter 3. Central Processor Complex System Design109
- Overview110
- Design Highlights110
- CPC Drawer Design112
  - Cache Levels and Memory Structure112
- Z13S Cache Levels and Memory Hierarchy112
- Z13S Cache Topology113
- Z13S and Zbc12 Cache Level Comparison114
  - CPC Drawer Interconnect Topology115
- Z13S CPC Drawer Communication Topology115
- Processor Unit Design116
- Point-To-Point Topology Z13S Two CPC Drawers Communication116
  - Simultaneous Multithreading117
  - Single-Instruction Multiple-Data118
- Two Threads Running Simultaneously on the same Processor Core118
- Schematic Representation of Add SIMD Instruction with 16 Elements in each Vector119
- Floating Point Registers Overlaid by Vector Registers119
  - Out-Of-Order Execution120
- Z13S PU Core Logical Diagram121
- Z13S In-Order and Out-Of-Order Core Execution Improvements122
  - Superscalar Processor123
  - Compression and Cryptography Accelerators on a Chip123
  - Decimal Floating Point Accelerator124
- Compression and Cryptography Accelerators on a Core in the Chip124
  - IEEE Floating Point125
  - Processor Error Detection and Recovery125
  - Branch Prediction126
  - Wild Branch126
- PU Error Detection and Recovery126
  - Translation Lookaside Buffer127
  - Instruction Fetching, Decoding, and Grouping127
  - Extended Translation Facility127
  - Instruction Set Extensions128
  - Transactional Execution128
  - Runtime Instrumentation128
- Processor Unit Functions128
  - Overview128
  - Central Processors130
  - Integrated Facility for Linux131
  - Internal Coupling Facility131
  - IBM Z Integrated Information Processor133
- ICF Options: Shared Icfs133
- Logical Flow of Java Code Execution on a Ziip134
  - System Assist Processors137
  - Reserved Processors138
  - Integrated Firmware Processor138
  - Processor Unit Assignment138
  - Sparing Rules139
  - Increased Flexibility with Z/VM Mode Partitions140
- Memory Design140
  - Overview140
  - Main Storage143
  - Expanded Storage143
  - Hardware System Area (HSA)144
- Logical Partitioning144
  - Overview145
  - Storage Operations152
  - Reserved Storage155
  - Logical Partition Storage Granularity156
  - LPAR Dynamic Storage Reconfiguration157
- Intelligent Resource Director157
- IRD LPAR Cluster Example157
- Clustering Technology159
- Sysplex Hardware Overview159
  - Coupling Facility Control Code160
  - Coupling Thin Interrupts161
  - Dynamic CF Dispatching161
- Dynamic CF Dispatching (Shared Cps or Shared ICF Pus)162
  - CFCC and Flash Express Use163
Chapter 4. Central Processor Complex I/O System Structure165
- Introduction to the Infiniband and Pcie for I/O Infrastructure166
  - Infiniband I/O Infrastructure166
  - Pcie I/O Infrastructure166
  - Infiniband Specifications167
  - Pcie Generation 3167
- I/O System Overview168
  - Characteristics168
  - Summary of Supported I/O Features169
- I/O Drawer170
- Pcie I/O Drawer171
- I/O Domains of an I/O Drawer171
- Pcie I/O Drawer172
- Z13S (N20) Connectivity to Pcie I/O Drawers173
- Pcie I/O Drawer and I/O Drawer Offerings174
- Pcie I/O Drawer with 32 Pcie Slots and Four I/O Domains174
- Fanouts175
- Infrastructure for Pcie and Infiniband Coupling Links176
  - Pcie Generation 3 Fanout (FC 0173)177
  - HCA2-C Fanout (FC 0162)177
  - Integrated Coupling Adapter (FC 0172)177
  - HCA3-O (12X IFB) Fanout (FC 0171)178
- OM3 50/125 ΜM Multimode Fiber Cable with MPO Connectors178
  - HCA3-O LR (1X IFB) Fanout (FC 0170)179
  - Fanout Considerations180
- I/O Features (Cards)182
  - I/O Feature Card Ordering Information183
  - Physical Channel Report184
- Connectivity185
  - I/O Feature Support and Configuration Rules186
  - FICON Channels188
  - OSA-Express5S194
  - OSA-Express4S Features196
  - OSA-Express for Ensemble Connectivity199
  - Hipersockets200
- Parallel Sysplex Connectivity202
  - Coupling Links202
- Z13 Parallel Sysplex Coupling Connectivity204
  - Migration Considerations206
- CPC Drawer Front View: Coupling Links207
- HCA3-O Fanouts: Z13S Versus Z114 / Zbc12 Servers208
  - Pulse Per Second Input211
- Cryptographic Functions211
  - CPACF Functions (FC 3863)211
  - Crypto Express5S Feature (FC 0890)211
- Integrated Firmware Processor211
- Flash Express212
  - IBM Flash Express Read/Write Cache213
- 10Gbe Roce Express213
- Pcie I/O Drawer that Is Fully Populated with Flash Express Cards213
- Zedc Express214
Chapter 5. Central Processor Complex Channel Subsystem215
- Channel Subsystem216
- Multiple Channel Subsystems and Multiple Subchannel Sets216
- Output for Display Ios,Config(All) Command221
- Systems CSS: Channel Subsystems with Channel Spanning222
- CSS, LPAR, and Identifier Example223
- I/O Configuration Management224
- Channel Subsystem Summary224
Chapter 6. Cryptography227
- Cryptography in IBM Z13 and Z13S Servers228
- Some Fundamentals on Cryptography228
- Three Levels of Protection with Three Levels of Speed231
- Cryptography on IBM Z13S Servers232
- Cryptographic Hardware Supported on IBM Z13S Servers232
- Z13S Cryptographic Support in Z/Os234
- CP Assist for Cryptographic Functions235
- The Cryptographic Coprocessor CPACF235
  - Cryptographic Synchronous Functions236
  - CPACF Protected Key237
- CPACF Key Wrapping238
- Crypto Express5S239
- Customize Image Profiles: Crypto248
- SE: View LPAR Cryptographic Controls249
- TKE Workstation250
- Cryptographic Functions Comparison253
- Cryptographic Software Support255
Chapter 7. Software Support257
- Operating Systems Summary258
- Support by Operating System258
- Support by Function271
- Cryptographic Support313
- GDPS Virtual Appliance317
- Z/Os Migration Considerations317
- Result of the Display Core Command324
- Simultaneous Multithreading325
- Single-Instruction Multiple-Data325
- The MIDAW Facility326
- IDAW Usage327
  - MIDAW Technical Description327
- MIDAW Format328
- MIDAW Usage328
  - Extended Format Data Sets329
- Iocp330
- Worldwide Port Name Tool331
- IBM Z Bladecenter Extension (Zbx) Model 004 Software Support332
- Software Licensing333
Chapter 8. System Upgrades339
- Upgrade Types340
  - Terminology Related to Cod for Z13S Systems341
  - Permanent Upgrades343
  - Temporary Upgrades344
- Concurrent Upgrades345
  - Customer Initiated Upgrade Facility347
- The Provisioning Architecture348
- Example of Temporary Upgrade Activation Sequence349
  - Summary of Concurrent Upgrade Functions350
- Miscellaneous Equipment Specification Upgrades351
  - MES Upgrade for Processors352
- Memory Sizes and Upgrades for the N10353
  - Preplanned Memory Feature353
- Memory Sizes and Upgrades for the Single Drawer N20354
- Memory Sizes and Upgrades for the Two Drawer N20354
- Feature on Demand Window for Zbx Blades Features Hwms355
  - MES Upgrades for the Zbx355
- Permanent Upgrade through the CIU Facility356
- Permanent Upgrade Order Example357
- CIU-Eligible Order Activation Example358
  - Ordering358
- Machine Profile359
  - Retrieval and Activation359
- IBM Z13S Perform Model Conversion Window360
- Customer Initiated Upgrade Order Activation Number Window360
  - On/Off Capacity on Demand360
  - Overview361
  - Ordering362
- Order On/Off Cod Record Window364
- On/Off Cod Order Example365
  - On/Off Cod Testing365
  - Activation and Deactivation366
- The Capacity Provisioning Infrastructure367
  - IBM Z/Os Capacity Provisioning367
- A Capacity Provisioning Domain369
- Capacity for Planned Event371
  - Capacity Backup373
  - CBU Activation and Deactivation375
- Example of C02 with Three CBU Features376
  - Automatic CBU for Geographically Dispersed Parallel Sysplex377
  - Processors378
- STSI Output on Z13S Server379
- Summary of Capacity On-Demand Offerings381
Chapter 9. Reliability, Availability, and Serviceability383
- The RAS Strategy384
- RAS Functions386
  - Scheduled Outages387
- Enhanced Driver Maintenance388
- RAS Capability for the HMC and SE390
- RAS Capability for Zbx Model 004391
  - Zbx Firmware392
- Considerations for Powerha in Zbx Environment394
- Typical Powerha Cluster Diagram395
- IBM Z Advanced Workload Analysis Reporter395
- Flash Express RAS Components396
- RAS Capability for Flash Express396
Chapter 10. Environmental Requirements397
- IBM Z13S Cabling Options398
- IBM Z13S Power and Cooling398
- Top Exit I/O Cabling Feature399
- Recommended Environmental Conditions402
- Rear Door Heat Exchanger (Left) and Functional Diagram407
  - IBM Zbx Physical Specifications407
- Top Exit Support for the Zbx408
- Energy Management408
  - Power Estimation Tool409
- Maximum Potential Power410
- Power Usage on the System Activity Display410
Chapter 11 . Hardware Management Console and Support Elements416
- Introduction to the HMC and SE416
  - Driver Level 27 HMC and SE Enhancements and Changes417
- Diagnostic Sampling Authorization Control418
- Change LPAR Security419
- Rack-Mounted HMC420
- Ses Location421
  - New Support Elements421
- Configure Backup FTP Server422
- Backup Critical Console Data Destinations422
- Backup Critical Data Destinations of Ses423
- Scheduled Operation for HMC Backup424
- Scheduled Operation for Ses Backup424
  - SE Driver Support with the HMC Driver424
  - HMC Feature Codes425
  - Tree Style User Interface and Classic Style User Interface426
- HMC and SE Connectivity427
- SE Physical Connection427
  - Network Planning for the HMC and SE428
- HMC Connectivity Examples429
  - Hardware Prerequisite Changes429
  - RSF Is Broadband-Only430
- Remote Support Facility431
  - RSF Connections to IBM and Enhanced IBM Service Support System432
- HMC and SE Key Capabilities433
  - Logical Partition Management434
- Change LPAR Group Controls - Group Absolute Capping435
  - Operating System Communication435
- Manage Trusted Signing Certificates436
- Import Remote Certificate Example436
- Configure 3270 Emulators436
  - HMC and SE Microcode437
- Microcode Terms and Interaction438
- System Information: Bundle Level439
  - Monitoring439
- HMC Monitor Task Group440
- Monitors Dashboard Task441
- Display the Activity for an LPAR by Processor Type441
- Display the SMT Usage441
- Monitors Dashboard: Crypto Function Integration442
- Monitors Dashboard - Flash Express Function Integration442
- Environmental Efficiency Statistics443
  - Capacity on Demand Support443
  - Features on Demand Support444
- Customize Console Date and Time446
- Timing Network Window with Scheduled DST and Scheduled Leap Second Offset447
  - NTP Client and Server Support on the HMC448
- HMC NTP Broadband Authentication Support449
- Time Coordination for Zbx Components450
  - Security and User ID Management450
  - System Input/Output Configuration Analyzer on the SE and HMC452
  - Cryptographic Support453
- Cryptographic Configuration Window454
  - Installation Support for Z/VM Using the HMC454
- Enabling Dynamic Partition Manager455
- HMC Welcome Window (CPC in DPM Mode)456
- HMC in an Ensemble456
- Unified Resource Manager Functions and Suites457
  - Ensemble Definition and Management460
  - HMC Availability462
- Ensemble Example with Primary and Alternate Hmcs463
Chapter 12. Performance465
- Z13S to Zbc12, Z114, Z10 BC, and Z9 BC Performance Comparison466
- Performance Information466
- LSPR Workload Suite467
  - Instruction Complexity468
- Memory Hierarchy on the Z13S One CPC Drawer System (Two Nodes)469
- Relative Nest Intensity469
- Relative Nest Intensity470
- The Traditional Factors that Were Used to Categorize Workloads470
- LSPR Workload Categories Based on Relative Nest Intensity471
- New Z/Os Workload Categories Defined472
- Relating Production Workloads to LSPR Workloads472
- Workload Performance Variation473
  - Main Performance Improvement Drivers with Z13S474
Appendix A. IBM Z Appliance Container Infrastructure478
- A.1 What Is Zaci478
- Zaci Framework Basic Outline479
  - Example: Deploying IBM Zaware479
- IBM Zaware Image Profile Based on Zaci480
- Zaci Icon in the HMC Interface480
Appendix B. IBM Z Systems Advanced Workload Analysis Reporter (IBM Zaware) . 453482
- B.1 Troubleshooting in Complex IT Environments482
- B.2 Introducing IBM Zaware483
- IBM Zaware Complements an Existing Environment484
  - B.2.1 Hardware Requirements Overview484
- IBM Zaware Shortens the Business Impact of a Problem485
  - B.2.3 IBM Z/Os Solutions to Improve Problem Diagnostic Procedures485
- B.3 Understanding IBM Zaware Technology486
- Basic Components of the IBM Zaware Environment487
- HMC Image Profile for an IBM Zaware LPAR488
- IBM Zaware Heat Map View Analysis489
- IBM Zaware Bar Score with Intervals490
- Feature on Demand Window for IBM Zaware Feature494
  - B.4.2 IBM Zaware Operating Requirements495
- B.5 Configuring and Using IBM Zaware Virtual Appliance496
Appendix C. Channel Options500
- C.1 Channel Options Supported on Z13S Servers500
- C.2 Maximum Unrepeated Distance for FICON SX Features501
Appendix D. Shared Memory Communications504
- D.1 Shared Memory Communications Overview504
- RDMA Technology Overview505
  - D.2.2 Shared Memory Communications over RDMA505
- Dynamic Transition from TCP to SMC-R506
  - D.2.3 Single Root I/O Virtualization506
- Shared Roce Mode Concepts507
  - D.2.4 Hardware507
- 10Gbe Roce Express508
- 10Gbe Roce Express Sample Configuration509
  - D.2.6 10Gbe Roce Express Configuration Example509
- RNIC and OSD Interaction510
  - D.2.7 Hardware Configuration Definitions511
- Physical Network ID Example512
  - D.2.8 Software Exploitation of SMC-R512
- Reduced Latency and Improved Wall Clock Time with SMC-R513
  - D.2.9 SMC-R Support Overview513
  - D.2.10 SMC-R Use Cases for Z/Os to Z/Os514
- Sysplex Distributor before Roce515
- Sysplex Distributor after Roce516
  - D.2.11 Enabling SMC-R Support in Z/Os Communications Server516
- D.3 Shared Memory Communications - Direct Memory Access517
- Clustered Systems: Multitier Application Solution. RDMA, and DMA518
  - D.3.1 Internal Shared Memory Technology Overview518
- Connecting Two Lpars on the same CPC Using SMC-D518
- Dynamic Transition from TCP to SMC-D by Using Two OSA-Express Adapters519
  - D.3.3 Internal Shared Memory - Introduction520
- Concept of Vpci Adapter Implementation521
- SMC-D Configuration that Uses Ethernet to Provide Connectivity522
- SMC-D Configuration that Uses Hipersockets to Provide Connectivity523
  - D.3.5 Planning Considerations523
- ISM Adapters Shared between Lpars524
  - D.3.7 Sample IOCP FUNCTION Statements524
- Multiple Lpars Connected through Multiple Vlans525
  - D.3.8 Software Exploitation of ISM526
- SMCD Parameter in GLOBALCONFIG527
  - D.3.10 Enabling SMC-D Support in Z/Os Communications Server527
  - D.3.11 SMC-D Support Overview528
- E.1 What Is IBM Dynamic Partition Manager530
Appendix E. IBM Dynamic Partition Manager531
- High-Level View of DPM Implementation531
- E.4 Setting up the DPM Environment532
- Enabling DPM Mode of Operation from the SE CPC Configuration Options532
- Entering the OSA Ports that will be Used by the Management Network533
- DPM Mode Welcome Window534
- Traditional HMC Welcome Window When no Cpcs Are Running in DPM Mode535
- User Options When the HMC Presents the DPM Welcome Window535
  - E.4.1 Defining Partitions in DPM Mode535
- DPM Wizard Welcome Window Options537
  - E.4.2 Summary538
- F.1 Why KVM for IBM Z Systems540
Appendix F. KVM for IBM Z Systems541
- KVM Running in Z Systems Lpars542
  - F.2.1 Storage Connectivity542
  - F.2.3 Hardware Management Console543
- Open Source Virtualization (KVM for IBM Z Systems)544
  - F.2.5 What Comes with KVM for IBM Z Systems544
- F.3 Managing the KVM for IBM Z Systems Environment546
- KVM for IBM Z Systems Management Interface546
- KVM Management by Using the Libvirt API Layers547
  - F.3.1 Hypervisor Performance Manager548
- G.1 Design of Native Pcie I/O Adapter Management550
Appendix G. Native Peripheral Component Interconnect Express551
- G.1.4 Management Tasks552
- I/O Domains and Resource Groups Managed by the IFP551
- G.3 Native Pcie Feature Definitions553
- Sample Output of AO Data or PCHID Report553
- Example of IOCP Statements for Zedc Express and 10Gbe Roce Express555
  - G.3.1 FUNCTION Identifier555
Appendix H. Flash Express558
- H.1 Flash Express Overview558
- Systems Storage Hierarchy558
- Flash Express Pcie Adapter559
- Pcie I/O Drawer that Is Fully Populated with Flash Express Cards559
- H.2 Using Flash Express560
- Sample SE/HMC Window for Flash Express Allocation to LPAR561
- Flash Express Allocation in Z/Os Lpars562
- H.3 Security on Flash Express563
- Integrated Key Controller564
  - H.3.2 Key Serving Topology565
- Key Serving Topology566
  - H.3.3 Error Recovery Scenarios566
Appendix I. GDPS Virtual Appliance570
- GDPS Offerings570
- I.1 GDPS Overview570
- I.2 Overview of GDPS Virtual Appliance572
- Positioning a Virtual Appliance572
- GDPS Virtual Appliance Architecture Overview574
- I.3 GDPS Virtual Appliance Recovery Scenarios575
- GDPS Storage Failover576
  - I.3.2 Unplanned Disk Outage576
  - I.3.3 Disaster Recovery577
Appendix J. IBM Zenterprise Data Compression Express580
- J.1 Overview580
  - Groups581
- J.3 Software Support581
  - J.3.1 Z/VM V6R3 Support with Ptfs582
Related Publications583

IBM z13s User Manual

Table of Contents

Questions and Answers:

IBM z13s Specifications

Related product manuals

Brand	IBM
Model	z13s
Category	Server
Language	English