Rationale

The biggest strength of {kindling} when modelling neural networks is its versatility — it inherits {torch}’s versatility while being human friendly, including the ability to apply custom optimizer functions, loss functions, and per-layer activation functions. Learn more: https://kindling.joshuamarie.com/articles/special-cases.

With act_funs(), you are not limited to the activation functions available in {torch}’s namespace. Use new_act_fn() to wrap any compatible function into a validated custom activation. This feature, however, is only available on version 0.3.0 and above.

Function to use

To do this, use new_act_fn(). It takes a user-supplied function, validates it against a small dummy tensor at definition time (a dry-run probe), and wraps it in a call-time type guard. This means errors surface early — before your model ever starts training.

The function you supply must:

Accept at least one argument (the input tensor).
Return a torch_tensor.

Basic Usage

Currently, nnf_tanh doesn’t exist in the {torch} namespace, so tanh is not a valid argument to act_funs(). With new_act_fn(), you can wrap torch::torch_tanh() to make it usable.

Here’s a basic example that wraps torch::torch_tanh() as a custom activation:

hyper_tan = new_act_fn(\(x) torch::torch_tanh(x))

You can also pass it directly into act_funs(), just like any built-in activation:

act_funs(relu, elu, new_act_fn(\(x) torch::torch_tanh(x)))

Using Custom Activations in a Model

Naturally, functions for modelling, like ffnn(), accept act_funs() into the activations argument. Again, you can pass a custom activation function within new_act_fn(), then pass it through act_funs().

Here’s a basic example:

model = ffnn(
    Sepal.Length ~ .,
    data = iris[, 1:4],
    hidden_neurons = c(64, 32, 16),
    activations = act_funs(
        relu,
        silu,
        new_act_fn(\(x) torch::torch_tanh(x))
    ),
    epochs = 50
)
model

Each element of act_funs() corresponds to one hidden layer, in order. Here, the first hidden layer uses ReLU, the second uses SiLU (Swish), and the third uses Tanh.

You can also use a single custom activation recycled across all layers:

ffnn(
    Sepal.Length ~ .,
    data = iris[, 1:4],
    hidden_neurons = c(64, 32),
    activations = act_funs(new_act_fn(\(x) torch::torch_tanh(x))),
    epochs = 50
)

Skipping the Dry-Run Probe

By default, new_act_fn() runs a quick dry-run with a small dummy tensor to validate your function before training. You can disable this with probe = FALSE, though this is generally not recommended:

my_act = new_act_fn(\(x) torch::torch_tanh(x), probe = FALSE)

Naming Your Custom Activation

You can provide a human-readable name via .name, which is used in print output and diagnostics:

my_act = new_act_fn(\(x) torch::torch_tanh(x), .name = "my_tanh")

Here’s a simple application:

ffnn(
    Sepal.Length ~ .,
    data = iris[, 1:4],
    hidden_neurons = c(64, 32),
    activations = act_funs(
        relu, 
        new_act_fn(\(x) torch::torch_tanh(x), .name = "hyper_tanh")
    ),
    epochs = 50
)

Error Handling

new_act_fn() is designed to fail loudly and early. Common errors include:

Function returns a non-tensor. This will error at definition time:
```
new_act_fn(\(x) as.numeric(x))
```
Function accepts no arguments. This will error immediately:
```
new_act_fn(function() torch::torch_zeros(2))
```

These checks ensure your model’s architecture is valid before any data ever flows through it.

Summary

Feature	Details
Wraps any R function	Must accept a tensor, return a tensor
Dry-run probe	Validates at definition time (`probe = TRUE` by default)
Call-time guard	Type-checks output on every forward pass
Compatible with `act_funs()`	Use alongside built-in activations freely
Closures supported	Parametric activations work naturally

Custom Activation Function