Code

Add FP8 MoE for turbomind#3601

Code

Open

lzhangzz wants to merge 137 commits intoInternLM:mainInternLM/lmdeploy:mainfrom

lzhangzz:gemm3lzhangzz/lmdeploy:gemm3Copy head branch name to clipboard

Commits

Commits on Mar 27, 2025

• low level abstraction

  lzhangzz

  committed
  404263d

  Copy full SHA for 404263d

Commits on Apr 2, 2025

• refactor

  lzhangzz

  committed
  81bfa75

  Copy full SHA for 81bfa75

Commits on Apr 7, 2025

• eliminate template

  lzhangzz

  committed
  770b85d

  Copy full SHA for 770b85d

• remove unused

  lzhangzz

  committed
  e3a9619

  Copy full SHA for e3a9619

• refactor bindings

  lzhangzz

  committed
  6b9a433

  Copy full SHA for 6b9a433

• simplify lm head

  lzhangzz

  committed
  613aeec

  Copy full SHA for 613aeec

Commits on Apr 8, 2025

• refactor weight

  lzhangzz

  committed
  e3fe34c

  Copy full SHA for e3fe34c

• fix tp

  lzhangzz

  committed
  1e057d1

  Copy full SHA for 1e057d1

• cublas

  lzhangzz

  committed
  40e9097

  Copy full SHA for 40e9097

Commits on Apr 9, 2025

• Merge remote-tracking branch 'origin/main' into core

  lzhangzz

  committed
  6fc9cc9

  Copy full SHA for 6fc9cc9

Commits on Apr 10, 2025

• refactor sampling

  lzhangzz

  committed
  ff3b5f7

  Copy full SHA for ff3b5f7

• remove unused

  lzhangzz

  committed
  06ff641

  Copy full SHA for 06ff641

Commits on Apr 11, 2025

• simplify

  lzhangzz

  committed
  14a7f45

  Copy full SHA for 14a7f45

• fix AWQ support

  lzhangzz

  committed
  096155c

  Copy full SHA for 096155c

• fix moe

  lzhangzz

  committed
  5fd35ae

  Copy full SHA for 5fd35ae

• fix nccl lm_head

  lzhangzz

  committed
  0c5ef46

  Copy full SHA for 0c5ef46

• fix

  lzhangzz

  committed
  c2020b2

  Copy full SHA for c2020b2

Commits on Apr 15, 2025

• refactor data types

  lzhangzz

  committed
  510675c

  Copy full SHA for 510675c

• skip legacy ut

  lzhangzz

  committed
  00b121e

  Copy full SHA for 00b121e

• simplify

  lzhangzz

  committed
  88d17d4

  Copy full SHA for 88d17d4

• rename data types

  lzhangzz

  committed
  699c24f

  Copy full SHA for 699c24f

• refactor

  lzhangzz

  committed
  3ffd070

  Copy full SHA for 3ffd070

Commits on Apr 16, 2025

• refactor runtime states

  lzhangzz

  committed
  eed6bfb

  Copy full SHA for eed6bfb

• fix msvc build

  lzhangzz

  committed
  d2ec3af

  Copy full SHA for d2ec3af

• fix msvc build

  lzhangzz

  committed
  2529631

  Copy full SHA for 2529631

• fix msvc build

  lzhangzz

  committed
  1d77856

  Copy full SHA for 1d77856

• fix msvc build

  lzhangzz

  committed
  6e728cf

  Copy full SHA for 6e728cf

• fix msvc build

  lzhangzz

  committed
  1b6a80d

  Copy full SHA for 1b6a80d

• fix msvc build

  lzhangzz

  committed
  0d976d3

  Copy full SHA for 0d976d3

• fix msvc build

  lzhangzz

  committed
  18e7602

  Copy full SHA for 18e7602

• fix msvc build

  lzhangzz

  committed
  7fec496

  Copy full SHA for 7fec496

• fix msvc build

  lzhangzz

  committed
  69b1841

  Copy full SHA for 69b1841

• format

  lzhangzz

  committed
  c8bc36d

  Copy full SHA for c8bc36d

• remove unused

  lzhangzz

  committed
  8161c0d

  Copy full SHA for 8161c0d

• fix msvc build

  lzhangzz

  committed
  7459992

  Copy full SHA for 7459992

• fix msvc build

  lzhangzz

  committed
  d38421f

  Copy full SHA for d38421f

• fix msvc build

  lzhangzz

  committed
  7d6ab03

  Copy full SHA for 7d6ab03

• fix msvc build

  lzhangzz

  committed
  b214a0e

  Copy full SHA for b214a0e

• fix msvc build

  lzhangzz

  committed
  3ab38ca

  Copy full SHA for 3ab38ca

• fix msvc build

  lzhangzz

  committed
  f394ef0

  Copy full SHA for f394ef0

• fix msvc build

  lzhangzz

  committed
  105f1cc

  Copy full SHA for 105f1cc

• fix msvc build

  lzhangzz

  committed
  5ccf30c

  Copy full SHA for 5ccf30c

• fix msvc build

  lzhangzz

  committed
  b59620c

  Copy full SHA for b59620c

• fix msvc build

  lzhangzz

  committed
  a243da0

  Copy full SHA for a243da0

• fix msvc build

  lzhangzz

  committed
  42172d3

  Copy full SHA for 42172d3

• fix msvc build

  lzhangzz

  committed
  4d9910a

  Copy full SHA for 4d9910a

• fix msvc build

  lzhangzz

  committed
  bf7c213

  Copy full SHA for bf7c213

• fix msvc build

  lzhangzz

  committed
  8651edd

  Copy full SHA for 8651edd

• fix msvc build

  lzhangzz

  committed
  4788b80

  Copy full SHA for 4788b80

• fix msvc build

  lzhangzz

  committed
  529225d

  Copy full SHA for 529225d

Commits on Apr 17, 2025

• fix msvc build

  lzhangzz

  committed
  6fd5f72

  Copy full SHA for 6fd5f72

• fix msvc build

  lzhangzz

  committed
  86d4e86

  Copy full SHA for 86d4e86

• fix ut & msvc build

  lzhangzz

  committed
  8ea7e20

  Copy full SHA for 8ea7e20

• fix ut & msvc build

  lzhangzz

  committed
  dcf9669

  Copy full SHA for dcf9669

• fix gcc build

  lzhangzz

  committed
  7f8974b

  Copy full SHA for 7f8974b

• fix lint & ut

  lzhangzz

  committed
  646813b

  Copy full SHA for 646813b

• fix lint

  lzhangzz

  committed
  98f4840

  Copy full SHA for 98f4840

• fetch Catch2 when building tests

  lzhangzz

  committed
  ea07957

  Copy full SHA for ea07957

• rewind msvc build

  lzhangzz

  committed
  5d69923

  Copy full SHA for 5d69923

Commits on Apr 18, 2025

• fix sampling

  lzhangzz

  committed
  d0079b5

  Copy full SHA for d0079b5

Commits on Apr 21, 2025

• fp8 round trip test

  lzhangzz

  committed
  15b4007

  Copy full SHA for 15b4007

Commits on Apr 22, 2025

• pseudo quant test

  lzhangzz

  committed
  72fa3ee

  Copy full SHA for 72fa3ee

Commits on Apr 24, 2025

• initial sm90 gemm kernel

  lzhangzz

  committed
  c4de357

  Copy full SHA for c4de357

• optimize smem desc

  lzhangzz

  committed
  26b270a

  Copy full SHA for 26b270a

• multiple warp groups

  lzhangzz

  committed
  68ac168

  Copy full SHA for 68ac168

• optimize smem desc

  lzhangzz

  committed
  4e573c5

  Copy full SHA for 4e573c5

• flush TMA ops

  lzhangzz

  committed
  4b801eb

  Copy full SHA for 4b801eb

• TMA epilogue

  lzhangzz

  committed
  97407ad

  Copy full SHA for 97407ad

• clean-up

  lzhangzz

  committed
  a97a3bb

  Copy full SHA for a97a3bb

• launch config & fence operand

  lzhangzz

  committed
  b26d7c8

  Copy full SHA for b26d7c8

• tuning

  lzhangzz

  committed
  24520ee

  Copy full SHA for 24520ee

• minor

  lzhangzz

  committed
  eb7a50e

  Copy full SHA for eb7a50e

Commits on Apr 25, 2025

• TMA multicast

  lzhangzz

  committed
  c477918

  Copy full SHA for c477918

• pipeline

  lzhangzz

  committed
  04257a6

  Copy full SHA for 04257a6

• warp specialization

  lzhangzz

  committed
  da5b22d

  Copy full SHA for da5b22d

Commits on Apr 27, 2025

• persistent kernel

  lzhangzz

  committed
  c728db2

  Copy full SHA for c728db2

Commits on Apr 28, 2025

• better TMA multicast

  lzhangzz

  committed
  7e03761

  Copy full SHA for 7e03761

Commits on Apr 29, 2025

• initial fp8 gemm

  lzhangzz

  committed
  4fb3799

  Copy full SHA for 4fb3799

• optimize

  lzhangzz

  committed
  27f1d29

  Copy full SHA for 27f1d29

Commits on Apr 30, 2025

• optimize

  lzhangzz

  committed
  876fbe7

  Copy full SHA for 876fbe7

• cluster boundary check

  lzhangzz

  committed
  316f2f8

  Copy full SHA for 316f2f8

• slow

  lzhangzz

  committed
  fb9dd9a

  Copy full SHA for fb9dd9a

• revert

  lzhangzz

  committed
  dc6f660

  Copy full SHA for dc6f660

• v2

  lzhangzz

  committed
  fd515e1

  Copy full SHA for fd515e1

Commits on May 1, 2025

• optimize

  lzhangzz

  committed
  f0e4e4e

  Copy full SHA for f0e4e4e

Commits on May 6, 2025

• fix scaling

  lzhangzz

  committed
  4bb1033

  Copy full SHA for 4bb1033

• fix scaling

  lzhangzz

  committed
  87ab9df

  Copy full SHA for 87ab9df

Commits on May 7, 2025

• optimize

  lzhangzz

  committed
  c1ec1de

  Copy full SHA for c1ec1de

• TMA store swizzle

  lzhangzz

  committed
  45b8218

  Copy full SHA for 45b8218

Commits on May 8, 2025

• better TMA store swizzle

  lzhangzz

  committed
  26b6e21

  Copy full SHA for 26b6e21

• clean up

  lzhangzz

  committed
  49e6ac9

  Copy full SHA for 49e6ac9

• prefetch U

  lzhangzz

  committed
  c2e3564

  Copy full SHA for c2e3564

• prefetch V

  lzhangzz

  committed
  42571c2

  Copy full SHA for 42571c2

Commits on May 9, 2025

• fix V for multi wg

  lzhangzz

  committed
  108e2aa

  Copy full SHA for 108e2aa

• fix U stride for TMA

  lzhangzz

  committed
  89fbbb7

  Copy full SHA for 89fbbb7

Commits on May 13, 2025

• qwen3 dense fp8

  lzhangzz

  committed
  464a2bc

  Copy full SHA for 464a2bc

• fix tma multicast

  lzhangzz

  committed
  0653c65

  Copy full SHA for 0653c65

• fix producer register count

  lzhangzz

  committed
  c14fcc4

  Copy full SHA for c14fcc4

• decouple epilogue

  lzhangzz

  committed
  dfbae20

  Copy full SHA for dfbae20

Commits on May 15, 2025

• warpspecialized pingpong

  lzhangzz

  committed
  731f2c5

  Copy full SHA for 731f2c5

• fix multicast

  lzhangzz

  committed
  77ea397

  Copy full SHA for 77ea397

• cluster layout

  lzhangzz

  committed
  2d991a1

  Copy full SHA for 2d991a1

Commits on May 16, 2025

• update

  lzhangzz

  committed
  86344c7

  Copy full SHA for 86344c7

• optimize

  lzhangzz

  committed
  56e181b

  Copy full SHA for 56e181b

• larger tiles

  lzhangzz

  committed
  0a422d3

  Copy full SHA for 0a422d3

Commits on May 19, 2025

• multicast U

  lzhangzz

  committed
  a004cd7

  Copy full SHA for a004cd7

• rename

  lzhangzz

  committed
  07c181d

  Copy full SHA for 07c181d

• v3

  lzhangzz

  committed
  b18f336

  Copy full SHA for b18f336

• optimize v3

  lzhangzz

  committed
  88018a6

  Copy full SHA for 88018a6

Commits on May 20, 2025

• tune

  lzhangzz

  committed
  5959297

  Copy full SHA for 5959297

Commits on May 21, 2025

• `tensormap.replace`

  lzhangzz

  committed
  a3022a3

  Copy full SHA for a3022a3

Commits on May 22, 2025

• refactor

  lzhangzz

  committed
  7eda414

  Copy full SHA for 7eda414

Commits on May 23, 2025

• v4

  lzhangzz

  committed
  2dc68c2

  Copy full SHA for 2dc68c2

Commits on May 26, 2025

• init moe support

  lzhangzz

  committed
  5ccf1d8

  Copy full SHA for 5ccf1d8

Commits on May 27, 2025

• fix

  lzhangzz

  committed
  d98be0f

  Copy full SHA for d98be0f

• fix v stride

  lzhangzz

  committed
  2de6d70

  Copy full SHA for 2de6d70

• fix empty tile & unaligned U

  lzhangzz

  committed
  5ef84ee

  Copy full SHA for 5ef84ee

• multicast schedule

  lzhangzz

  committed
  7eb5f69

  Copy full SHA for 7eb5f69

Commits on May 28, 2025

• scheduling

  lzhangzz

  committed
  bac2803

  Copy full SHA for bac2803

• fix scheduling

  lzhangzz

  committed
  062872d

  Copy full SHA for 062872d

Commits on May 29, 2025

• tune

  lzhangzz

  committed
  493ab50

  Copy full SHA for 493ab50

• fix non-grouped gemm

  lzhangzz

  committed
  e3f3818

  Copy full SHA for e3f3818

• tune group gemm

  lzhangzz

  committed
  e6119a3

  Copy full SHA for e6119a3

• load weight by pointers

  lzhangzz

  committed
  cfacea2

  Copy full SHA for cfacea2

Commits on May 30, 2025

• fp8 moe

  lzhangzz

  committed
  90ed985

  Copy full SHA for 90ed985

• Merge remote-tracking branch 'origin/main' into gemm3

  lzhangzz

  committed
  4a58024

  Copy full SHA for 4a58024

Commits on Jun 2, 2025

• switch to https git

  lzhangzz

  committed
  036d67a

  Copy full SHA for 036d67a

• fix cutlass tag

  lzhangzz

  committed
  8474114

  Copy full SHA for 8474114

• 90 -> 90a

  lzhangzz

  committed
  e4b4c49

  Copy full SHA for e4b4c49

• fix missing headers

  lzhangzz

  committed
  75eb1ed

  Copy full SHA for 75eb1ed

• fix cuda-11

  lzhangzz

  committed
  3493b2d

  Copy full SHA for 3493b2d

• guard sm90

  lzhangzz

  committed
  ce5ac45

  Copy full SHA for ce5ac45

• update

  lzhangzz

  committed
  863be32

  Copy full SHA for 863be32

• v5

  lzhangzz

  committed
  11289e4

  Copy full SHA for 11289e4

• update

  lzhangzz

  committed
  d31e65d

  Copy full SHA for d31e65d

• update

  lzhangzz

  committed
  2fa2ad5

  Copy full SHA for 2fa2ad5

Commits on Jun 3, 2025

• v5

  lzhangzz

  committed
  19b93d2

  Copy full SHA for 19b93d2