cxl2.0协议-CXL-2.0-Specification-r1.0-2020Oct26_CXL一致性模型资源-CSDN文库

需积分: 1 199 浏览量 2024-01-26 18:06:57 上传评论收藏 9.28MB PDF 举报

资源推荐

资源详情

资源评论

Compute Express Link

(CXL

)

Specification

October 2020

Revision 2.0

- 1 -

C6124.0003 BN/FCURCI 39895222v1

LEGAL NOTICE FOR THIS PUBLICLY-AVAILABLE SPECIFICATION FROM COMPUTE EXPRESS LINK CONSORTIUM, INC.

This CXL Specification Revision 1.1 (this “CXL Specification” or this “document”) is owned by and is proprietary to Compute Express Link

Consortium, Inc., a Delaware nonprofit corporation (sometimes referred to as “CXL” or the “CXL Consortium” or the “Company”) and/or its

successors and assigns.

NOTICE TO USERS WHO ARE MEMBERS OF THE CXL CONSORTIUM:

If you are a Member of the CXL Consortium (sometimes referred to as a “CXL Member”), and even if you have received this publicly-available version

of this CXL Specification after agreeing to CXL Consortium’s Evaluation Copy Agreement (a copy of which is available

https://www.computeexpresslink.org/download-the-specification, each such CXL Member must also be in compliance with all of the following CXL

Consortium documents, policies and/or procedures (collectively, the “CXL Governing Documents”) in order for such CXL Member’s use and/or

implementation of this CXL Specification to receive and enjoy all of the rights, benefits, privileges and protections of CXL Consortium membership: (i)

CXL Consortium’s Intellectual Property Policy; (ii) CXL Consortium’s Bylaws; (iii) any and all other CXL Consortium policies and procedures; and (iv)

the CXL Member’s Participation Agreement.

NOTICE TO NON-MEMBERS OF THE CXL CONSORTIUM:

If you are not a CXL Member and have received this publicly-available version of this CXL Specification, your use of this document is subject to your

compliance with, and is limited by, all of the terms and conditions of the CXL Consortium’s Evaluation Copy Agreement (a copy of which is available at

https://www.computeexpresslink.org/download-the-specification).

In addition to the restrictions set forth in the CXL Consortium’s Evaluation Copy Agreement, any references or citations to this document must

acknowledge the Compute Express Link Consortium, Inc.’s sole and exclusive copyright ownership of this CXL Specification. The proper copyright

any such citation or reference to this document you are not permitted to revise, alter, modify, make any derivatives of, or otherwise amend the referenced

portion of this document in any way without the prior express written permission of the Compute Express Link Consortium, Inc.

Except for the limited rights explicitly given to a non-CXL Member pursuant to the explicit provisions of the CXL Consortium’s Evaluation Copy

Agreement which governs the publicly-available version of this CXL Specification, nothing contained in this CXL Specification shall be deemed as

granting (either expressly or impliedly) to any party that is not a CXL Member: (ii) any kind of license to implement or use this CXL Specification or any

portion or content described or contained therein, or any kind of license in or to any other intellectual property owned or controlled by the CXL

Consortium, including without limitation any trademarks of the CXL Consortium.; or (ii) any benefits and/or rights as a CXL Member under any CXL

Governing Documents.

LEGAL DISCLAIMERS FOR ALL PARTIES:

THIS DOCUMENT AND ALL SPECIFICATIONS AND/OR OTHER CONTENT PROVIDED HEREIN IS PROVIDED ON AN “AS IS” BASIS. TO

THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, COMPUTE EXPRESS LINK CONSORTIUM, INC. (ALONG WITH THE

CONTRIBUTORS TO THIS DOCUMENT) HEREBY DISCLAIM ALL REPRESENTATIONS, WARRANTIES AND/OR COVENANTS, EITHER

EXPRESS OR IMPLIED, STATUTORY OR AT COMMON LAW, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF

MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE, VALIDITY, AND/OR NON-INFRINGEMENT.

In the event this CXL Specification makes any references (including without limitation any incorporation by reference) to another standard’s setting

organization’s or any other party’s (“Third Party”) content or work, including without limitation any specifications or standards of any such Third Party

(“Third Party Specification”), you are hereby notified that your use or implementation of any Third Party Specification: (i) is not governed by any of

the CXL Governing Documents; (ii) may require your use of a Third Party’s patents, copyrights or other intellectual property rights, which in turn may

require you to independently obtain a license or other consent from that Third Party in order to have full rights to implement or use that Third Party

Specification; and/or (iii) may be governed by the intellectual property policy or other policies or procedures of the Third Party which owns the Third

Party Specification. Any trademarks or service marks of any Third Party which may be referenced in this CXL Specification is owned by the respective

owner of such marks.

NOTICE TO ALL PARTIES REGARDING THE PCI-SIG UNIQUE VALUE PROVIDED IN THIS CXL SPECIFICATION:

NOTICE TO USERS: THE UNIQUE VALUE THAT IS PROVIDED IN THIS CXL SPECIFICATION IS FOR USE IN VENDOR DEFINED

MESSAGE FIELDS, DESIGNATED VENDOR SPECIFIC EXTENDED CAPABILITIES, AND ALTERNATE PROTOCOL NEGOTIATION ONLY

AND MAY NOT BE USED IN ANY OTHER MANNER, AND A USER OF THE UNIQUE VALUE MAY NOT USE THE UNIQUE VALUE IN A

MANNER THAT (A) ALTERS, MODIFIES, HARMS OR DAMAGES THE TECHNICAL FUNCTIONING, SAFETY OR SECURITY OF THE PCI-

SIG ECOSYSTEM OR ANY PORTION THEREOF, OR (B) COULD OR WOULD REASONABLY BE DETERMINED TO ALTER, MODIFY,

HARM OR DAMAGE THE TECHNICAL FUNCTIONING, SAFETY OR SECURITY OF THE PCI-SIG ECOSYSTEM OR ANY PORTION

THEREOF (FOR PURPOSES OF THIS NOTICE, “PCI-SIG ECOSYSTEM” MEANS THE PCI-SIG SPECIFICATIONS, MEMBERS OF PCI-SIG

AND THEIR ASSOCIATED PRODUCTS AND SERVICES THAT INCORPORATE ALL OR A PORTION OF A PCI-SIG SPECIFICATION AND

EXTENDS TO THOSE PRODUCTS AND SERVICES INTERFACING WITH PCI-SIG MEMBER PRODUCTS AND SERVICES).

Contents

Compute Express Link Specification

October 26, 2020 3

Revision 2.0, Version 1.0

Contents

1.0 Introduction ............................................................................................................................................................................................27

1.1 Audience ....................................................................................................................................................................................27

1.2 Terminology / Acronyms ....................................................................................................................................................27

1.3 Reference Documents..........................................................................................................................................................30

1.4 Motivation and Overview....................................................................................................................................................30

1.4.1 Compute Express Link.......................................................................................................................................30

1.4.2 Flex Bus....................................................................................................................................................................32

1.5 Flex Bus Link Features .........................................................................................................................................................35

1.6 Flex Bus Layering Overview...............................................................................................................................................35

1.7 Document Scope....................................................................................................................................................................37

2.0 Compute Express Link System Architecture...........................................................................................................................39

2.1 Type 1 CXL Device.................................................................................................................................................................39

2.2 Type 2 CXL Device.................................................................................................................................................................40

2.2.1 Bias Based Coherency Model .........................................................................................................................41

2.2.1.1 Host Bias ...........................................................................................................................................42

2.2.1.2 Device Bias .......................................................................................................................................42

2.2.1.3 Mode Management.......................................................................................................................43

2.2.1.4 Software Assisted Bias Mode Management.......................................................................43

2.2.1.5 Hardware Autonomous Bias Mode Management............................................................43

2.3 Type 3 CXL Device.................................................................................................................................................................44

2.4 Multi Logical Device ..............................................................................................................................................................44

2.4.1 LD-ID for CXL.io and CXL.mem ......................................................................................................................45

2.4.1.1 LD-ID for CXL.mem.......................................................................................................................45

2.4.1.2 LD-ID for CXL.io .............................................................................................................................45

2.4.2 Pooled Memory Device Configuration Registers ...................................................................................45

2.5 CXL Device Scaling ................................................................................................................................................................47

3.0 Compute Express Link Transaction Layer................................................................................................................................48

3.1 CXL.io...........................................................................................................................................................................................48

3.1.1 CXL.io Endpoint....................................................................................................................................................49

3.1.2 CXL Power Management VDM Format.......................................................................................................50

3.1.2.1 Credit and PM Initialization.......................................................................................................54

3.1.3 CXL Error VDM Format ......................................................................................................................................55

3.1.4 Optional PCIe Features Required for CXL.................................................................................................56

3.1.5 Error Propagation................................................................................................................................................56

3.1.6 Memory Type Indication on ATS...................................................................................................................56

3.1.7 Deferrable Writes.................................................................................................................................................57

3.2 CXL.cache ..................................................................................................................................................................................58

3.2.1 Overview..................................................................................................................................................................58

3.2.2 CXL.cache Channel Description.....................................................................................................................59

3.2.2.1 Channel Ordering..........................................................................................................................59

3.2.2.2 Channel Crediting .........................................................................................................................59

3.2.3 CXL.cache Wire Description............................................................................................................................60

3.2.3.1 D2H Request ...................................................................................................................................60

3.2.3.2 D2H Response ................................................................................................................................61

3.2.3.3 D2H Data...........................................................................................................................................61

3.2.3.4 H2D Request ...................................................................................................................................62

3.2.3.5 H2D Response ................................................................................................................................62

3.2.3.6 H2D Data...........................................................................................................................................63

3.2.4 CXL.cache Transaction Description .............................................................................................................63

3.2.4.1 Device to Host Requests ............................................................................................................63

3.2.4.2 Device to Host Response...........................................................................................................74

Contents

Compute Express Link Specification

October 26, 2020 4

Revision 2.0, Version 1.0

3.2.4.3 Host to Device Requests ............................................................................................................76

3.2.4.4 Host to Device Response...........................................................................................................77

3.2.5 Cacheability Details and Request Restrictions........................................................................................79

3.2.5.1 GO-M Responses...........................................................................................................................79

3.2.5.2 Device/Host Snoop-GO-Data Assumptions ......................................................................79

3.2.5.3 Device/Host Snoop/WritePull Assumptions .....................................................................79

3.2.5.4 Snoop Responses and Data Transfer on CXL.cache Evicts.........................................80

3.2.5.5 Multiple Snoops to the Same Address ................................................................................80

3.2.5.6 Multiple Reads to the Same Cache Line..............................................................................80

3.2.5.7 Multiple Evicts to the Same Cache Line ..............................................................................80

3.2.5.8 Multiple Write Requests to the Same Cache Line...........................................................80

3.2.5.9 Normal Global Observation (GO)............................................................................................80

3.2.5.10 Relaxed Global Observation (FastGO)..................................................................................81

3.2.5.11 Evict to Device-Attached Memory .........................................................................................81

3.2.5.12 Memory Type on CXL.cache.....................................................................................................81

3.2.5.13 General Assumptions ..................................................................................................................81

3.2.5.14 Buried Cache State Rules...........................................................................................................82

3.3 CXL.mem....................................................................................................................................................................................83

3.3.1 Introduction ...........................................................................................................................................................83

3.3.2 QoS Telemetry for Memory.............................................................................................................................84

3.3.2.1 QoS Telemetry Overview...........................................................................................................84

3.3.2.2 Reference Model for Host Support of QoS Telemetry..................................................85

3.3.2.3 Memory Device Support for QoS Telemetry.....................................................................86

3.3.3 M2S Request (Req) ..............................................................................................................................................94

3.3.4 M2S Request with Data (RwD) ........................................................................................................................97

3.3.5 S2M No Data Response (NDR)........................................................................................................................98

3.3.6 S2M Data Response (DRS) ...............................................................................................................................99

3.3.7 Forward Progress and Ordering Rules .................................................................................................... 100

3.4 Transaction Ordering Summary ................................................................................................................................... 101

3.5 Transaction Flows to Device-Attached Memory.................................................................................................... 103

3.5.1 Flows for Type 1 and Type 2 Devices ...................................................................................................... 103

3.5.1.1 Notes and Assumptions .......................................................................................................... 103

3.5.1.2 Requests from Host................................................................................................................... 104

3.5.1.3 Requests from Device in Host and Device Bias............................................................. 109

3.5.2 Type 2 and Type 3 Memory Flows ............................................................................................................ 112

3.5.2.1 Speculative Memory Read...................................................................................................... 112

3.6 Flows for Type 3 Devices................................................................................................................................................. 113

4.0 Compute Express Link Link Layers........................................................................................................................................... 115

4.1 CXL.io Link Layer ................................................................................................................................................................. 115

4.2 CXL.mem and CXL.cache Common Link Layer....................................................................................................... 117

4.2.1 Introduction ........................................................................................................................................................117

4.2.2 High-Level CXL.cache/CXL.mem Flit Overview ................................................................................... 119

4.2.3 Slot Format Definition .................................................................................................................................... 124

4.2.3.1 H2D and M2S Formats............................................................................................................. 125

4.2.3.2 D2H and S2M Formats............................................................................................................. 131

4.2.4 Link Layer Registers......................................................................................................................................... 139

4.2.5 Flit Packing Rules.............................................................................................................................................. 139

4.2.6 Link Layer Control Flit..................................................................................................................................... 141

4.2.7 Link Layer Initialization................................................................................................................................... 145

4.2.8 CXL.cache/CXL.mem Link Layer Retry..................................................................................................... 146

4.2.8.1 LLR Variables................................................................................................................................ 147

4.2.8.2 LLCRD Forcing ............................................................................................................................. 149

4.2.8.3 LLR Control Flits ......................................................................................................................... 151

4.2.8.4 RETRY Framing Sequences.................................................................................................... 151

4.2.8.5 LLR State Machines ................................................................................................................... 152

4.2.8.6 Interaction with Physical Layer Reinitialization............................................................. 156

Contents

Compute Express Link Specification

October 26, 2020 5

Revision 2.0, Version 1.0

4.2.8.7 CXL.cache/CXL.mem Flit CRC ............................................................................................... 157

4.2.9 Poison and Viral ................................................................................................................................................158

4.2.9.1 Viral .................................................................................................................................................. 158

5.0 Compute Express Link ARB/MUX.............................................................................................................................................. 160

5.1 Virtual LSM States............................................................................................................................................................... 161

5.1.1 Additional Rules for Local vLSM Transitions ........................................................................................ 164

5.1.2 Rules for Virtual LSM State Transitions Across Link.......................................................................... 164

5.1.2.1 General Rules............................................................................................................................... 164

5.1.2.2 Entry to Active Exchange Protocol ..................................................................................... 164

5.1.2.3 Status Synchronization Protocol.........................................................................................165

5.1.2.4 State Request ALMP.................................................................................................................. 167

5.1.2.5 State Status ALMP ..................................................................................................................... 169

5.1.2.6 Unexpected ALMPs ................................................................................................................... 171

5.1.3 Applications of the vLSM State Transition Rules................................................................................ 172

5.1.3.1 Initial Link Training .................................................................................................................... 172

5.1.3.2 Status Exchange Snapshot Example.................................................................................. 175

5.1.3.3 L1 Abort Example....................................................................................................................... 176

5.2 ARB/MUX Link Management Packets......................................................................................................................... 177

5.2.1 ARB/MUX Bypass Feature............................................................................................................................. 178

5.3 Arbitration and Data Multiplexing/Demultiplexing .............................................................................................. 179

6.0 Flex Bus Physical Layer .................................................................................................................................................................. 180

6.1 Overview ................................................................................................................................................................................. 180

6.2 Flex Bus.CXL Framing and Packet Layout.................................................................................................................181

6.2.1 Ordered Set Blocks and Data Blocks........................................................................................................181

6.2.2 Protocol ID[15:0]............................................................................................................................................... 182

6.2.3 x16 Packet Layout ............................................................................................................................................ 183

6.2.4 x8 Packet Layout............................................................................................................................................... 184

6.2.5 x4 Packet Layout............................................................................................................................................... 187

6.2.6 x2 Packet Layout............................................................................................................................................... 187

6.2.7 x1 Packet Layout............................................................................................................................................... 187

6.2.8 Special Case: CXL.io -- When a TLP Ends on a Flit Boundary........................................................ 187

6.2.9 Framing Errors ................................................................................................................................................... 188

6.3 Link Training.......................................................................................................................................................................... 190

6.3.1 PCIe vs Flex Bus.CXL Mode Selection ...................................................................................................... 190

6.3.1.1 Hardware Autonomous Mode Negotiation..................................................................... 190

6.3.1.2 CXL 2.0 Versus CXL 1.1 Negotiation.................................................................................. 194

6.3.1.3 Flex Bus.CXL Negotiation with Maximum Supported Link

Speed of 8GT/s or 16GT/s ..................................................................................................... 196

6.3.1.4 Link Width Degradation and Speed Downgrade.......................................................... 197

6.4 Recovery.Idle and Config.Idle Transitions to L0 .................................................................................................... 197

6.5 L1 Abort Scenario ............................................................................................................................................................... 197

6.6 Exit from Recovery.............................................................................................................................................................. 197

6.7 Retimers and Low Latency Mode ................................................................................................................................. 197

6.7.1 SKP Ordered Set Frequency and L1/Recovery Entry........................................................................ 198

7.0 Switching...............................................................................................................................................................................................201

7.1 Overview ................................................................................................................................................................................. 201

7.1.1 Single VCS Switch............................................................................................................................................. 201

7.1.2 Multiple VCS Switch ........................................................................................................................................202

7.1.3 Multiple VCS Switch with MLD Ports........................................................................................................203

7.2 Switch Configuration and Composition..................................................................................................................... 204

7.2.1 CXL Switch Initialization Options............................................................................................................... 204

7.2.1.1 Static Initialization ..................................................................................................................... 204

7.2.1.2 Fabric Manager Boots First .................................................................................................... 205

7.2.1.3 Fabric Manager and Host Boot Simultaneously............................................................ 207