CNet: complex-valued neural networks in C++ and CUDA

CNet is a C++/CUDA framework for building and studying deep complex-valued neural networks, and for minimising complex-valued functions by gradient descent with Wirtinger derivatives. Every built-in layer ships a CPU reference and a CUDA kernel, checked against each other and against finite differences. It is open source under the MIT licence at github.com/crasmarum/CNet.

Why complex-valued networks? Complex numbers suit data whose phase carries information: radio, radar, medical imaging, acoustics. The area is underexplored, largely because real-valued networks have been so successful. The field ℂ can be identified with ℝ², but as W. Rudin noted, “in spite of the well-known and obvious identification of ℂ with ℝ², these two are entirely different as far as their vector space structure is concerned.”

Why not PyTorch or JAX?

PyTorch and JAX support complex tensors, and for mainstream work they are the right tools: far more mature, faster at scale, with large ecosystems. CNet does not compete there. It exists for research on complex-valued learning itself:

• Complex by construction. Every variable, layer, loss and the optimizer are complex. Nothing is assembled from real building blocks plus patches.
• Explicit Wirtinger gradients. Each layer works with the pair ∂L/∂z and ∂L/∂z̄ directly, so non-holomorphic functions such as |z|² are handled in the open, not behind a conjugation convention.
• Spectral layers built in. The DFT, its inverse and the element-wise (Hadamard) product are first-class layers.
• A Born-rule head. The output is read as an amplitude vector z, and class probabilities follow the quantum measurement rule p_k = |z_k|² / ‖z‖², with no softmax.
• Small and auditable. The forward and backward pass of every layer and its kernel are short, readable code.

Your own model in one file

For sequence models (text, bytes, Go moves, any token stream) you write only the model graph and the data source. app::Trainer supplies training on one GPU or many, Adam with warm-up and cosine decay, gradient clipping, validation and checkpointing; app::Infer loads a model and generates or scores.

#include "../impl/cnet_app.h"

struct MyData : app::DataSource { ... };   // vocab(), context(), sample(...)
struct MyModel : app::ModelDef { ... };    // build(net, vocab, context)

int main(int argc, char **argv) {
    FLAGS::Parse(argc, argv);
    MyData data(...); MyModel model(...);
    app::Trainer trainer(model, data);     // -steps -lr -batch -save_path ...
    return trainer.run();
}

make app SRC=examples/char_lm.cpp
./char_lm -data book.txt -steps 5000 -batch 8 -save_path lm.mod
./char_lm -data book.txt -generate true -model lm.mod -prompt "ROMEO:" -gen_len 300

examples/char_lm.cpp is a complete byte-level language model in about 100 lines.

A small network

Networks are graphs of complex functions. This one classifies MNIST digits with a Fourier transform, a learned element-wise filter, a complex GELU and a dense layer, ending in the Born-rule cross entropy:

CNet cnet;
auto inp    = cnet.add(new CInput(OutSize(28 * 28)));
auto fft    = cnet.add(new FourierTrans(InSize(28 * 28)), {inp});

auto h_data = cnet.add(new CInput(OutSize(28 * 28)));
auto hdm    = cnet.add(new Hadamard(InSize(28 * 28), InSize(28 * 28)), {fft, h_data});
auto gelu   = cnet.add(new CGelu(InSize(28 * 28)), {hdm});

auto l_data = cnet.add(new CInput(OutSize(28 * 28 * 10)));
auto lin    = cnet.add(new Linear(InSize(28 * 28), InSize(28 * 28 * 10)), {gelu, l_data});
auto outp   = cnet.add(new CrossEntropy(InSize(10)), {lin});

Learnable parameters are just inputs to other layers: h_data is the filter of the Hadamard layer and l_data the weight matrix of the dense one.

Adding your own layer

To add a function, say the complex sigmoid S(z)_i = 1 / (1 + e^−z_i), extend CFunc and implement forward(). A layer also needs a constructor taking a Uid, a clone() and a getName(), because CNet clones layers on the GPU and names them in saved models.

class CSigmoid : public CFunc {
public:
  CSigmoid(Uid uid, InSize in_size) : CFunc(uid, in_size, OutSize(in_size.value())) {}
  CSigmoid(InSize in_size) : CFunc(in_size, OutSize(in_size.value())) {}

  virtual CFunc* clone(Uid uid) { return new CSigmoid(uid, InSize(input().length_)); }
  virtual std::string getName() { return "CSigmoid_" + std::to_string(uid_); }

  virtual void forward() {
    for (int i = 0; i < no_outputs(); ++i)
      for (int k = 0; k < input().length_; ++k) {
        complex<float> g = 1.0f / (1.0f + exp(-input().z(k)));
        mutable_output(i)->real_[offset(i) + k] = g.real();
        mutable_output(i)->imag_[offset(i) + k] = g.imag();
      }
  }
};

Writing backward() by hand is usually the hard part. Instead you can give the two Wirtinger derivatives, dz() and dz_star(), and CNet builds the backward pass from them. The sigmoid depends only on z, not on its conjugate, so

∂S_j/∂z̄_i = 0,    ∂S_j/∂z_i = g(z_i)(1 − g(z_i)) if i = j, else 0

virtual complex<float> dz(int out_indx, int in_indx) {
  if (out_indx != in_indx) return 0;
  auto g = 1.0f / (1.0f + exp(-input().z(in_indx)));
  return g * (1.0f - g);
}
virtual complex<float> dz_star(int out_indx, int in_indx) { return 0; }

Most useful functions are not holomorphic, and then dz_star() is not zero. CNet's own activation, CGelu, applies a real function g to the real and imaginary parts separately, f(z) = g(x) + i g(y), so

∂f/∂z = ½(g′(x) + g′(y)),    ∂f/∂z̄ = ½(g′(x) − g′(y))

A falling loss only shows that the gradient points downhill. To check the derivatives themselves, compare the backward pass with central finite differences of the forward pass, as CNet's own layer tests do. The generic backward evaluates dz() for every pair of input and output, which is fine for small layers; for large element-wise layers, override backward() directly. A new CFunc runs on the CPU; running it inside a GPU graph also needs a hand-written CUDA kernel.

How it runs on a GPU

To run the forward and backward passes, CNet orders the graph by depth. Forward runs depth by depth from the inputs; backward runs in reverse.

On an NVIDIA GPU, CNet clones the network once per example in the batch and runs each depth level across all clones at once. A batch of 32 becomes 32 clones of the CPU network.

Memory therefore grows linearly with the batch while the time per step stays nearly flat: on an H100, a six-block complex FNet takes about 0.33 ms forward and 0.32 ms backward at every batch size from 16 to 256. Because the device pointers never move after allocation, the whole step can be captured once as a CUDA graph and replayed, which halves the step time and gives bit-identical results. Across several GPUs, CNet trains data-parallel, one process per GPU, with gradients summed by NCCL.

The layers

Training uses plain gradient descent, momentum, or full Adam with gradient clipping, on CPU and GPU. The Born-rule loss is ill-conditioned when the correct class has a near-zero amplitude, so Adam with clipping is recommended, and the sequence loss has an optional probability floor that steadies training of deep models.

Building

git clone https://github.com/crasmarum/CNet.git
cd CNet
make

The Makefile builds with CUDA when it finds nvcc, for Ampere and Hopper GPUs by default, and falls back to a CPU-only build with g++ otherwise.

Limits

• Multi-GPU training is data-parallel only; there is no model or tensor parallelism yet.
• Adding a layer to the GPU path means writing its CUDA kernel by hand; there is no automatic CPU-to-GPU code generation.
• Linux and macOS only.