Channels are uncompressed, this means that when working with multiple textures/channels (height, basecolor, metallic, roughness and so on) you end up with 4 or more textures at the resolution of your document that need to do some back and forth between the software and the VRAM (the GPU memory), you end-up with a lot of bandwidth consumption. A 32bits height map will be around 190mb for example. The uncompressed map per channels adds up quickly in size, in addition to all the substances and materials than need to stays in VRAM in order to be able to compute the result of the blending. So very quickly the VRAM will be filled. Therefore Substance Painter end up doing a lot of back and forth with the GPU. A 4K by 4K texture set can consume something like 4 to 8 GB per second on very complex layer stack.
Also when using fill layers or tools, don't hesitate to disable channels that you don't need. This will allow the engine to discard the computation for these specific channels and will speed up performance in the long run. This can be done by setting the blending modes of a layer to "disable" too.