Dropout in Quantum Neural Networks

Francesco Scala

Demos/
Quantum Machine Learning/
Dropout in Quantum Neural Networks

Dropout in Quantum Neural Networks

Francesco Scala

Published: March 11, 2024. Last updated: November 05, 2024.

Are you struggling with overfitting while training Quantum Neural Networks (QNNs)?

In this demo, we show how to exploit the quantum version of the dropout technique to avoid the problem of overfitting in overparametrized QNNs. What follows is based on the paper “A General Approach to Dropout in Quantum Neural Networks” by F. Scala, et al. 1.

demos/_static/demonstration_assets/quantum_dropout/socialthumbnail_large_QuantumDropout_2024-03-07.png

What is overfitting and dropout?

Neural Networks (NNs) usually require highly flexible models with lots of trainable parameters in order to learn a certain underlying function (or data distribution). However, being able to learn with low in-sample error is not enough; generalization — the ability to provide good predictions on previously unseen data — is also desirable.

Highly expressive models may suffer from overfitting, which means that they are trained too well on the training data, and as a result perform poorly on new, unseen data. This happens because the model has learned the noise in the training data, rather than the underlying pattern that is generalizable to new data.

Dropout is a common technique for classical Deep Neural Networks (DNNs) preventing computational units from becoming too specialized and reducing the risk of overfitting 2, 3. It consists of randomly removing neurons or connections only during training to block the flow of information. Once the model is trained, the DNN is employed in its original form.

Why dropout for Quantum Neural Networks?

Recently, it has been shown that the use of overparametrized QNN models changes the optimization landscape by removing lots of local minima 4, 5. On the one hand, this increased number of parameters leads to faster and easier training, but on the other hand, it may drive the model to overfit the data. This is also strictly related to the repeated encoding of classical data to achieve nonlinearity in the computation. This is why, inspired from classical DNNs, one can think of applying some sort of dropout to QNNs. This would correspond to randomly dropping some (groups of) parameterized gates during training to achieve better generalization.

Quantum dropout of rotations in a sine regression

In this demo we will exploit quantum dropout to avoid overfitting during the regression of noisy data originally coming from the sinusoidal function. In particular, we will randomly “drop” rotations during the training phase. In practice, this will correspond to temporarily setting parameters to a value of 0.

Let’s start by importing Pennylane and numpy and fixing the random seed for reproducibility:

import numpy as np
import pennylane as qml

seed = 12345
np.random.seed(seed=seed)

The circuit

Now we define the embedding of classical data and the variational ansatz that will then be combined to construct our QNN. Dropout will happen inside the variational ansatz. Obtaining dropout with standard Pennylane would be quite straightforward by means of some “if statements”, but the training procedure will take ages. Here we will leverage JAX in order to speed up the training process with Just In Time (JIT) compilation. The drawback is that the definition of the variational ansatz becomes a little elaborated, since JAX has its own language for conditional statements. For this purpose we define two functions true_cond and false_cond to work with jax.lax.cond`, which is the JAX conditional statement. See this demo for additional insights on how to optimize QNNs with JAX.

Practically speaking, rotation dropout will be performed by passing a list to the ansatz. The single qubit rotations are applied depending on the values stored in this list: if the value is negative the rotation is dropped (rotation dropout), otherwise it is applied. How to produce this list will be explained later in this demo (see the make_dropout function).

import jax  # require for Just In Time (JIT) compilation
import jax.numpy as jnp

jax.config.update("jax_platform_name", "cpu")
jax.config.update("jax_enable_x64", True)


def embedding(x, wires):
    # Encodes the datum multiple times in the register,
    # employing also nonlinear functions
    assert len(x) == 1  # check feature is 1-D
    for i in wires:
        qml.RY(jnp.arcsin(x), wires=i)
    for i in wires:
        qml.RZ(jnp.arccos(x ** 2), wires=i)


def true_cond(angle):
    # necessary for using an if statement within jitted function
    # exploiting jax.lax.cond
    # if this function is assessed the rotation is dropped
    return 0.0


def false_cond(angle):
    # necessary for using an if statement within jitted function
    # exploiting jax.lax.cond
    # if this function is assessed the rotation is kept
    return angle


def var_ansatz(
    theta, wires, rotations=[qml.RX, qml.RZ, qml.RX], entangler=qml.CNOT, keep_rotation=None
):

    """Single layer of the variational ansatz for our QNN.
    We have a single qubit rotation per each qubit (wire) followed by
    a linear chain of entangling gates (entangler). This structure is repeated per each rotation in `rotations`
    (defining `inner_layers`).
    The single qubit rotations are applied depending on the values stored in `keep_rotation`:
    if the value is negative the rotation is dropped (rotation dropout), otherwise it is applied.

    Params:
    - theta: variational angles that will undergo optimization
    - wires: list of qubits (wires)
    - rotations: list of rotation kind per each `inner_layer`
    - entangler: entangling gate
    - keep_rotation: list of lists. There is one list per each `inner_layer`.
                    In each list there are indexes of the rotations that we want to apply.
                    Some of these values may be substituted by -1 value
                    which means that the rotation gate wont be applied (dropout).
    """

    # the length of `rotations` defines the number of inner layers
    N = len(wires)
    assert len(theta) == 3 * N
    wires = list(wires)

    counter = 0
    # keep_rotations contains a list per each inner_layer
    for rots in keep_rotation:
        # we cicle over the elements of the lists inside keep_rotation
        for qb, keep_or_drop in enumerate(rots):
            rot = rotations[counter]  # each inner layer can have a different rotation

            angle = theta[counter * N + qb]
            # conditional statement implementing dropout
            # if `keep_or_drop` is negative the rotation is dropped
            angle_drop = jax.lax.cond(keep_or_drop < 0, true_cond, false_cond, angle)
            rot(angle_drop, wires=wires[qb])
        for qb in wires[:-1]:
            entangler(wires=[wires[qb], wires[qb + 1]])
        counter += 1

And then we define the hyperparameters of our QNN, namely the number of qubits, the number of sublayers in the variational ansatz (inner_layers) and the resulting number of parameters per layer:

n_qubits = 5
inner_layers = 3
params_per_layer = n_qubits * inner_layers

Now we actually build the QNN:

def create_circuit(n_qubits, layers):
    device = qml.device("default.qubit", wires=n_qubits)

    @qml.qnode(device)
    def circuit(x, theta, keep_rot):
        # print(x)
        # print(theta)

        for i in range(layers):
            embedding(x, wires=range(n_qubits))

            keep_rotation = keep_rot[i]

            var_ansatz(
                theta[i * params_per_layer : (i + 1) * params_per_layer],
                wires=range(n_qubits),
                entangler=qml.CNOT,
                keep_rotation=keep_rotation,
            )

        return qml.expval(qml.PauliZ(wires=0))  # we measure only the first qubit

    return circuit

Let’s have a look at a single layer of our QNN:

import matplotlib.pyplot as plt


plt.style.use("pennylane.drawer.plot")  # set pennylane theme, which is nice to see

# create the circuit with given number of qubits and layers
layers = 1
circ = create_circuit(n_qubits, layers=layers)

# for the moment let's keep all the rotations in all sublayers
keep_all_rot = [
    [list(range((n_qubits))) for j in range(1, inner_layers + 1)],
]
# we count the parameters
numbered_params = np.array(range(params_per_layer * layers), dtype=float)
# we encode a single coordinate
single_sample = np.array([0])

qml.draw_mpl(circ, decimals=2,)(single_sample, numbered_params, keep_all_rot)

plt.show()

We now build the model that we will employ for the regression task. Since we want to have an overparametrized QNN, we will add 10 layers and we exploit JAX to speed the training up:

layers = 10
qnn_tmp = create_circuit(n_qubits, layers)
qnn_tmp = jax.jit(qnn_tmp)
qnn_batched = jax.vmap(
    qnn_tmp, (0, None, None)
)  # we want to vmap on 0-axis of the first circuit param
# in this way we process in parallel all the inputs
# We jit for faster execution
qnn = jax.jit(qnn_batched)

Dropping rotations

As anticipated, we need to set some random parameters to 0 at each optimization step. Given a layer dropout rate \(p_L\) (this will be called layer_drop_rate) and the gate dropout rate \(p_G\) (this will be called rot_drop_rate), the probability \(p\) that a gate is dropped in a layer can be calculated with the conditioned probability law:

\[p=p(A\cap B)=p(A|B)p(B)=p_Gp_L\]

where \(B\) represents the selection of a specific layer and \(A\) the selection of a specific gate within the chosen layer.

In the following cell we define a function that produces the list of the indices of rotation gates that are kept. For gates which are dropped, the value -1 is assigned to the corresponding index. The structure of the list is nested; we have one list per inner_layer inside one list per each layer, all contained in another list. This function will be called at each iteration.

def make_dropout(key):
    drop_layers = []

    for lay in range(layers):
        # each layer has prob p_L=layer_drop_rate of being dropped
        # according to that for every layer we sample
        # if we have to appy dropout in it or not
        out = jax.random.choice(
            key, jnp.array(range(2)), p=jnp.array([1 - layer_drop_rate, layer_drop_rate])
        )
        key = jax.random.split(key)[0]  # update the random key

        if out == 1:  # if it has to be dropped
            drop_layers.append(lay)

    keep_rot = []
    # we make list of indexes corresponding to the rotations gates
    # that are kept in the computation during a single train step
    for i in range(layers):
        # each list is divded in layers and then in "inner layers"
        # this is strictly related to the QNN architecture that we use
        keep_rot_layer = [list(range((n_qubits))) for j in range(1, inner_layers + 1)]

        if i in drop_layers:  # if dropout has to be applied in this layer
            keep_rot_layer = []  # list of indexes for a single layer
            inner_keep_r = []  # list of indexes for a single inner layer
            for param in range(params_per_layer):
                # each rotation within the layer has prob p=rot_drop_rate of being dropped
                # according to that for every parameter (rotation) we sample
                # if we have to drop it or not
                out = jax.random.choice(
                    key, jnp.array(range(2)), p=jnp.array([1 - rot_drop_rate, rot_drop_rate])
                )
                key = jax.random.split(key)[0]  # update the random key

                if out == 0:  # if we have to keep it
                    inner_keep_r.append(param % n_qubits)  # % is required because we work
                    # inner layer by inner layer
                else:  # if the rotation has to be dropped
                    inner_keep_r.append(-1)  # we assign the value -1

                if param % n_qubits == n_qubits - 1:  # if it's the last qubit of the register
                    # append the inner layer list
                    keep_rot_layer.append(inner_keep_r)
                    # and reset it
                    inner_keep_r = []

        keep_rot.append(keep_rot_layer)

    return jnp.array(keep_rot)

We can check the output of the make_dropout function:

# setting the drop probability
layer_drop_rate, rot_drop_rate = 0.5, 0.3  # 15% probability of dropping a gate

# JAX random key
key = jax.random.PRNGKey(12345)
# create the list of indexes,
# -1 implies we are dropping a gate
keep_rot = make_dropout(key)

# let's just print the list for first layer
print(keep_rot[0])

[[ 0  1  2  3  4]
 [ 0  1 -1  3  4]
 [ 0  1  2 -1  4]]

Noisy sinusoidal function

To test the effectiveness of the dropout technique, we will use a prototypical dataset with which it is very easy to overfit: the sinusoidal function. We produce some points according to the \(\sin\) function and then we add some white Gaussian noise (noise that follows a normal distribution) \(\epsilon.\) The noise is essential to obtain overfitting; when our model is extremely expressive, it is capable of exactly fit each point and some parameters become hyper-specialized in recognizing the noisy features. This makes predictions on new unseen data difficult, since the overfitting model did not learn the true underlying data distribution. The dropout technique will help in avoiding co-adaptation and hyper-specialization, effectively reducing overfitting.

from sklearn.model_selection import train_test_split


def make_sin_dataset(dataset_size=100, test_size=0.4, noise_value=0.4, plot=False):
    """1D regression problem y=sin(x*\pi)"""
    # x-axis
    x_ax = np.linspace(-1, 1, dataset_size)
    y = [[np.sin(x * np.pi)] for x in x_ax]
    np.random.seed(123)
    # noise vector
    noise = np.array([np.random.normal(0, 0.5, 1) for i in y]) * noise_value
    X = np.array(x_ax)
    y = np.array(y + noise)  # apply noise

    # split the dataset
    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=test_size, random_state=40, shuffle=True
    )

    X_train = X_train.reshape(-1, 1)
    X_test = X_test.reshape(-1, 1)

    y_train = y_train.reshape(-1, 1)
    y_test = y_test.reshape(-1, 1)

    return X_train, X_test, y_train, y_test


from matplotlib import ticker

X, X_test, y, y_test = make_sin_dataset(dataset_size=20, test_size=0.25)


fig, ax = plt.subplots()
plt.plot(X, y, "o", label="Training")
plt.plot(X_test, y_test, "o", label="Test")

plt.plot(
    np.linspace(-1, 1, 100),
    [np.sin(x * np.pi) for x in np.linspace(-1, 1, 100)],
    linestyle="dotted",
    label=r"$\sin(x)$",
)
plt.ylabel(r"$y = \sin(\pi\cdot x) + \epsilon$")
plt.xlabel(r"$x$")
ax.xaxis.set_major_locator(ticker.MultipleLocator(0.5))
ax.yaxis.set_major_locator(ticker.MultipleLocator(0.5))
plt.legend()

plt.show()

Since our circuit is only able to provide outputs in the range \([-1,1],\) we rescale all the noisy data within this range. To do this we leverage the MinMaxScaler from sklearn. It is common practice to fit the scaler only from training data and then apply it also to the test. The reason behind this is that in general one only has knowledge about the training dataset. (If the training dataset is not exhaustively representative of the underlying distribution, this preprocessing may lead to some outliers in the test set to be scaled out of the desired range.)

from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler(feature_range=(-1, 1))
y = scaler.fit_transform(y)
y_test = scaler.transform(y_test)

# reshaping for computation
y = y.reshape(-1,)
y_test = y_test.reshape(-1,)

Optimization

At this point we have to set the hyperparameters of the optimization, namely the number of epochs, the learning rate, and the optimizer:

import optax  # optimization using jax

epochs = 700
optimizer = optax.adam(learning_rate=0.01)

We define the cost function as the Mean Square Error:

@jax.jit
def calculate_mse_cost(X, y, theta, keep_rot):
    yp = qnn(X, theta, keep_rot)
    # depending on your version of Pennylane you may require the following line
    #####
    yp = jnp.array(yp).T
    #####
    cost = jnp.mean((yp - y) ** 2)

    return cost


# Optimization update step
@jax.jit
def optimizer_update(opt_state, params, x, y, keep_rot):
    loss, grads = jax.value_and_grad(lambda theta: calculate_mse_cost(x, y, theta, keep_rot))(
        params
    )
    updates, opt_state = optimizer.update(grads, opt_state)

    params = optax.apply_updates(params, updates)
    return params, opt_state, loss

Training the model

And now we can try to train the model. We execute different runs of the training to understand the average behaviour of quantum dropout. To see the effect of dropout we can set different values of layer_drop_rate and rot_drop_rate:

n_run = 3
drop_rates = [(0.0, 0.0), (0.3, 0.2), (0.7, 0.7)]

train_history = {}
test_history = {}
opt_params = {}


for layer_drop_rate, rot_drop_rate in drop_rates:
    # initialization of some lists to store data
    costs_per_comb = []
    test_costs_per_comb = []
    opt_params_per_comb = []
    # we execute multiple runs in order to see the average behaviour
    for tmp_seed in range(seed, seed + n_run):
        key = jax.random.PRNGKey(tmp_seed)
        assert len(X.shape) == 2  # X must be a matrix
        assert len(y.shape) == 1  # y must be an array
        assert X.shape[0] == y.shape[0]  # compatibility check

        # parameters initialization with gaussian ditribution
        initial_params = jax.random.normal(key, shape=(layers * params_per_layer,))
        # update the random key
        key = jax.random.split(key)[0]

        params = jnp.copy(initial_params)

        # optimizer initialization
        opt_state = optimizer.init(initial_params)

        # lists for saving single run training and test cost trend
        costs = []
        test_costs = []

        for epoch in range(epochs):
            # generate the list for dropout
            keep_rot = make_dropout(key)
            # update the random key
            key = jax.random.split(key)[0]

            # optimization step
            params, opt_state, cost = optimizer_update(opt_state, params, X, y, keep_rot)

            ############## performance evaluation #############
            # inference is done with the original model
            # with all the gates
            keep_rot = jnp.array(
                [
                    [list(range((n_qubits))) for j in range(1, inner_layers + 1)]
                    for i in range(layers)
                ]
            )
            # inference on train set
            cost = calculate_mse_cost(X, y, params, keep_rot)

            costs.append(cost)

            # inference on test set
            test_cost = calculate_mse_cost(X_test, y_test, params, keep_rot)
            test_costs.append(test_cost)

            # we print updates every 5 iterations
            if epoch % 5 == 0:
                print(
                    f"{layer_drop_rate:.1f}-{rot_drop_rate:.1f}",
                    f"run {tmp_seed-seed} - epoch {epoch}/{epochs}",
                    f"--- Train cost:{cost:.5f}",
                    f"--- Test cost:{test_cost:.5f}",
                    end="\r",
                )

        costs_per_comb.append(costs)
        test_costs_per_comb.append(test_costs)
        opt_params_per_comb.append(params)
        print()
    costs_per_comb = np.array(costs_per_comb)
    test_costs_per_comb = np.array(test_costs_per_comb)
    opt_params_per_comb = np.array(opt_params_per_comb)

    train_history[(layer_drop_rate, rot_drop_rate)] = costs_per_comb
    test_history[(layer_drop_rate, rot_drop_rate)] = test_costs_per_comb
    opt_params[(layer_drop_rate, rot_drop_rate)] = opt_params_per_comb

0.0-0.0 run 0 - epoch 0/700 --- Train cost:0.41948 --- Test cost:0.15604
0.0-0.0 run 0 - epoch 5/700 --- Train cost:0.17625 --- Test cost:0.06937
0.0-0.0 run 0 - epoch 10/700 --- Train cost:0.08036 --- Test cost:0.03687
0.0-0.0 run 0 - epoch 15/700 --- Train cost:0.04314 --- Test cost:0.02814
0.0-0.0 run 0 - epoch 20/700 --- Train cost:0.02818 --- Test cost:0.02979
0.0-0.0 run 0 - epoch 25/700 --- Train cost:0.02180 --- Test cost:0.03449
0.0-0.0 run 0 - epoch 30/700 --- Train cost:0.01806 --- Test cost:0.03888
0.0-0.0 run 0 - epoch 35/700 --- Train cost:0.01544 --- Test cost:0.04073
0.0-0.0 run 0 - epoch 40/700 --- Train cost:0.01343 --- Test cost:0.04056
0.0-0.0 run 0 - epoch 45/700 --- Train cost:0.01194 --- Test cost:0.04051
0.0-0.0 run 0 - epoch 50/700 --- Train cost:0.01096 --- Test cost:0.04126
0.0-0.0 run 0 - epoch 55/700 --- Train cost:0.01028 --- Test cost:0.04203
0.0-0.0 run 0 - epoch 60/700 --- Train cost:0.00970 --- Test cost:0.04225
0.0-0.0 run 0 - epoch 65/700 --- Train cost:0.00920 --- Test cost:0.04211
0.0-0.0 run 0 - epoch 70/700 --- Train cost:0.00876 --- Test cost:0.04196
0.0-0.0 run 0 - epoch 75/700 --- Train cost:0.00839 --- Test cost:0.04192
0.0-0.0 run 0 - epoch 80/700 --- Train cost:0.00804 --- Test cost:0.04194
0.0-0.0 run 0 - epoch 85/700 --- Train cost:0.00773 --- Test cost:0.04201
0.0-0.0 run 0 - epoch 90/700 --- Train cost:0.00744 --- Test cost:0.04207
0.0-0.0 run 0 - epoch 95/700 --- Train cost:0.00717 --- Test cost:0.04200
0.0-0.0 run 0 - epoch 100/700 --- Train cost:0.00691 --- Test cost:0.04184
0.0-0.0 run 0 - epoch 105/700 --- Train cost:0.00667 --- Test cost:0.04179
0.0-0.0 run 0 - epoch 110/700 --- Train cost:0.00644 --- Test cost:0.04194
0.0-0.0 run 0 - epoch 115/700 --- Train cost:0.00621 --- Test cost:0.04219
0.0-0.0 run 0 - epoch 120/700 --- Train cost:0.00600 --- Test cost:0.04245
0.0-0.0 run 0 - epoch 125/700 --- Train cost:0.00580 --- Test cost:0.04272
0.0-0.0 run 0 - epoch 130/700 --- Train cost:0.00560 --- Test cost:0.04305
0.0-0.0 run 0 - epoch 135/700 --- Train cost:0.00541 --- Test cost:0.04343
0.0-0.0 run 0 - epoch 140/700 --- Train cost:0.00522 --- Test cost:0.04384
0.0-0.0 run 0 - epoch 145/700 --- Train cost:0.00505 --- Test cost:0.04427
0.0-0.0 run 0 - epoch 150/700 --- Train cost:0.00487 --- Test cost:0.04471
0.0-0.0 run 0 - epoch 155/700 --- Train cost:0.00471 --- Test cost:0.04515
0.0-0.0 run 0 - epoch 160/700 --- Train cost:0.00454 --- Test cost:0.04563
0.0-0.0 run 0 - epoch 165/700 --- Train cost:0.00438 --- Test cost:0.04613
0.0-0.0 run 0 - epoch 170/700 --- Train cost:0.00423 --- Test cost:0.04665
0.0-0.0 run 0 - epoch 175/700 --- Train cost:0.00408 --- Test cost:0.04718
0.0-0.0 run 0 - epoch 180/700 --- Train cost:0.00394 --- Test cost:0.04774
0.0-0.0 run 0 - epoch 185/700 --- Train cost:0.00379 --- Test cost:0.04831
0.0-0.0 run 0 - epoch 190/700 --- Train cost:0.00366 --- Test cost:0.04890
0.0-0.0 run 0 - epoch 195/700 --- Train cost:0.00352 --- Test cost:0.04949
0.0-0.0 run 0 - epoch 200/700 --- Train cost:0.00339 --- Test cost:0.05009
0.0-0.0 run 0 - epoch 205/700 --- Train cost:0.00327 --- Test cost:0.05070
0.0-0.0 run 0 - epoch 210/700 --- Train cost:0.00315 --- Test cost:0.05131
0.0-0.0 run 0 - epoch 215/700 --- Train cost:0.00303 --- Test cost:0.05192
0.0-0.0 run 0 - epoch 220/700 --- Train cost:0.00291 --- Test cost:0.05253
0.0-0.0 run 0 - epoch 225/700 --- Train cost:0.00280 --- Test cost:0.05313
0.0-0.0 run 0 - epoch 230/700 --- Train cost:0.00270 --- Test cost:0.05372
0.0-0.0 run 0 - epoch 235/700 --- Train cost:0.00260 --- Test cost:0.05429
0.0-0.0 run 0 - epoch 240/700 --- Train cost:0.00250 --- Test cost:0.05486
0.0-0.0 run 0 - epoch 245/700 --- Train cost:0.00241 --- Test cost:0.05541
0.0-0.0 run 0 - epoch 250/700 --- Train cost:0.00232 --- Test cost:0.05594
0.0-0.0 run 0 - epoch 255/700 --- Train cost:0.00223 --- Test cost:0.05645
0.0-0.0 run 0 - epoch 260/700 --- Train cost:0.00215 --- Test cost:0.05695
0.0-0.0 run 0 - epoch 265/700 --- Train cost:0.00207 --- Test cost:0.05743
0.0-0.0 run 0 - epoch 270/700 --- Train cost:0.00199 --- Test cost:0.05788
0.0-0.0 run 0 - epoch 275/700 --- Train cost:0.00192 --- Test cost:0.05832
0.0-0.0 run 0 - epoch 280/700 --- Train cost:0.00185 --- Test cost:0.05874
0.0-0.0 run 0 - epoch 285/700 --- Train cost:0.00179 --- Test cost:0.05915
0.0-0.0 run 0 - epoch 290/700 --- Train cost:0.00173 --- Test cost:0.05954
0.0-0.0 run 0 - epoch 295/700 --- Train cost:0.00167 --- Test cost:0.05993
0.0-0.0 run 0 - epoch 300/700 --- Train cost:0.00161 --- Test cost:0.06030
0.0-0.0 run 0 - epoch 305/700 --- Train cost:0.00155 --- Test cost:0.06067
0.0-0.0 run 0 - epoch 310/700 --- Train cost:0.00150 --- Test cost:0.06103
0.0-0.0 run 0 - epoch 315/700 --- Train cost:0.00145 --- Test cost:0.06140
0.0-0.0 run 0 - epoch 320/700 --- Train cost:0.00140 --- Test cost:0.06176
0.0-0.0 run 0 - epoch 325/700 --- Train cost:0.00135 --- Test cost:0.06213
0.0-0.0 run 0 - epoch 330/700 --- Train cost:0.00130 --- Test cost:0.06250
0.0-0.0 run 0 - epoch 335/700 --- Train cost:0.00125 --- Test cost:0.06288
0.0-0.0 run 0 - epoch 340/700 --- Train cost:0.00121 --- Test cost:0.06327
0.0-0.0 run 0 - epoch 345/700 --- Train cost:0.00117 --- Test cost:0.06366
0.0-0.0 run 0 - epoch 350/700 --- Train cost:0.00112 --- Test cost:0.06406
0.0-0.0 run 0 - epoch 355/700 --- Train cost:0.00108 --- Test cost:0.06447
0.0-0.0 run 0 - epoch 360/700 --- Train cost:0.00104 --- Test cost:0.06489
0.0-0.0 run 0 - epoch 365/700 --- Train cost:0.00100 --- Test cost:0.06532
0.0-0.0 run 0 - epoch 370/700 --- Train cost:0.00096 --- Test cost:0.06575
0.0-0.0 run 0 - epoch 375/700 --- Train cost:0.00093 --- Test cost:0.06619
0.0-0.0 run 0 - epoch 380/700 --- Train cost:0.00089 --- Test cost:0.06664
0.0-0.0 run 0 - epoch 385/700 --- Train cost:0.00086 --- Test cost:0.06709
0.0-0.0 run 0 - epoch 390/700 --- Train cost:0.00083 --- Test cost:0.06755
0.0-0.0 run 0 - epoch 395/700 --- Train cost:0.00080 --- Test cost:0.06801
0.0-0.0 run 0 - epoch 400/700 --- Train cost:0.00077 --- Test cost:0.06847
0.0-0.0 run 0 - epoch 405/700 --- Train cost:0.00074 --- Test cost:0.06894
0.0-0.0 run 0 - epoch 410/700 --- Train cost:0.00071 --- Test cost:0.06940
0.0-0.0 run 0 - epoch 415/700 --- Train cost:0.00069 --- Test cost:0.06986
0.0-0.0 run 0 - epoch 420/700 --- Train cost:0.00066 --- Test cost:0.07032
0.0-0.0 run 0 - epoch 425/700 --- Train cost:0.00064 --- Test cost:0.07078
0.0-0.0 run 0 - epoch 430/700 --- Train cost:0.00062 --- Test cost:0.07123
0.0-0.0 run 0 - epoch 435/700 --- Train cost:0.00060 --- Test cost:0.07167
0.0-0.0 run 0 - epoch 440/700 --- Train cost:0.00058 --- Test cost:0.07211
0.0-0.0 run 0 - epoch 445/700 --- Train cost:0.00056 --- Test cost:0.07254
0.0-0.0 run 0 - epoch 450/700 --- Train cost:0.00054 --- Test cost:0.07296
0.0-0.0 run 0 - epoch 455/700 --- Train cost:0.00053 --- Test cost:0.07338
0.0-0.0 run 0 - epoch 460/700 --- Train cost:0.00051 --- Test cost:0.07378
0.0-0.0 run 0 - epoch 465/700 --- Train cost:0.00050 --- Test cost:0.07418
0.0-0.0 run 0 - epoch 470/700 --- Train cost:0.00048 --- Test cost:0.07456
0.0-0.0 run 0 - epoch 475/700 --- Train cost:0.00047 --- Test cost:0.07494
0.0-0.0 run 0 - epoch 480/700 --- Train cost:0.00045 --- Test cost:0.07531
0.0-0.0 run 0 - epoch 485/700 --- Train cost:0.00044 --- Test cost:0.07567
0.0-0.0 run 0 - epoch 490/700 --- Train cost:0.00043 --- Test cost:0.07602
0.0-0.0 run 0 - epoch 495/700 --- Train cost:0.00042 --- Test cost:0.07636
0.0-0.0 run 0 - epoch 500/700 --- Train cost:0.00041 --- Test cost:0.07669
0.0-0.0 run 0 - epoch 505/700 --- Train cost:0.00040 --- Test cost:0.07701
0.0-0.0 run 0 - epoch 510/700 --- Train cost:0.00039 --- Test cost:0.07733
0.0-0.0 run 0 - epoch 515/700 --- Train cost:0.00038 --- Test cost:0.07763
0.0-0.0 run 0 - epoch 520/700 --- Train cost:0.00037 --- Test cost:0.07793
0.0-0.0 run 0 - epoch 525/700 --- Train cost:0.00036 --- Test cost:0.07822
0.0-0.0 run 0 - epoch 530/700 --- Train cost:0.00035 --- Test cost:0.07851
0.0-0.0 run 0 - epoch 535/700 --- Train cost:0.00034 --- Test cost:0.07879
0.0-0.0 run 0 - epoch 540/700 --- Train cost:0.00033 --- Test cost:0.07906
0.0-0.0 run 0 - epoch 545/700 --- Train cost:0.00032 --- Test cost:0.07932
0.0-0.0 run 0 - epoch 550/700 --- Train cost:0.00032 --- Test cost:0.07958
0.0-0.0 run 0 - epoch 555/700 --- Train cost:0.00031 --- Test cost:0.07983
0.0-0.0 run 0 - epoch 560/700 --- Train cost:0.00030 --- Test cost:0.08008
0.0-0.0 run 0 - epoch 565/700 --- Train cost:0.00029 --- Test cost:0.08032
0.0-0.0 run 0 - epoch 570/700 --- Train cost:0.00029 --- Test cost:0.08056
0.0-0.0 run 0 - epoch 575/700 --- Train cost:0.00028 --- Test cost:0.08079
0.0-0.0 run 0 - epoch 580/700 --- Train cost:0.00027 --- Test cost:0.08102
0.0-0.0 run 0 - epoch 585/700 --- Train cost:0.00027 --- Test cost:0.08124
0.0-0.0 run 0 - epoch 590/700 --- Train cost:0.00026 --- Test cost:0.08146
0.0-0.0 run 0 - epoch 595/700 --- Train cost:0.00026 --- Test cost:0.08167
0.0-0.0 run 0 - epoch 600/700 --- Train cost:0.00025 --- Test cost:0.08188
0.0-0.0 run 0 - epoch 605/700 --- Train cost:0.00024 --- Test cost:0.08209
0.0-0.0 run 0 - epoch 610/700 --- Train cost:0.00024 --- Test cost:0.08229
0.0-0.0 run 0 - epoch 615/700 --- Train cost:0.00023 --- Test cost:0.08249
0.0-0.0 run 0 - epoch 620/700 --- Train cost:0.00023 --- Test cost:0.08269
0.0-0.0 run 0 - epoch 625/700 --- Train cost:0.00022 --- Test cost:0.08288
0.0-0.0 run 0 - epoch 630/700 --- Train cost:0.00022 --- Test cost:0.08307
0.0-0.0 run 0 - epoch 635/700 --- Train cost:0.00021 --- Test cost:0.08326
0.0-0.0 run 0 - epoch 640/700 --- Train cost:0.00021 --- Test cost:0.08344
0.0-0.0 run 0 - epoch 645/700 --- Train cost:0.00020 --- Test cost:0.08363
0.0-0.0 run 0 - epoch 650/700 --- Train cost:0.00020 --- Test cost:0.08380
0.0-0.0 run 0 - epoch 655/700 --- Train cost:0.00019 --- Test cost:0.08398
0.0-0.0 run 0 - epoch 660/700 --- Train cost:0.00019 --- Test cost:0.08415
0.0-0.0 run 0 - epoch 665/700 --- Train cost:0.00019 --- Test cost:0.08432
0.0-0.0 run 0 - epoch 670/700 --- Train cost:0.00018 --- Test cost:0.08449
0.0-0.0 run 0 - epoch 675/700 --- Train cost:0.00018 --- Test cost:0.08466
0.0-0.0 run 0 - epoch 680/700 --- Train cost:0.00017 --- Test cost:0.08482
0.0-0.0 run 0 - epoch 685/700 --- Train cost:0.00017 --- Test cost:0.08498
0.0-0.0 run 0 - epoch 690/700 --- Train cost:0.00017 --- Test cost:0.08514
0.0-0.0 run 0 - epoch 695/700 --- Train cost:0.00016 --- Test cost:0.08529
0.0-0.0 run 1 - epoch 0/700 --- Train cost:0.24656 --- Test cost:0.08702
0.0-0.0 run 1 - epoch 5/700 --- Train cost:0.08913 --- Test cost:0.04158
0.0-0.0 run 1 - epoch 10/700 --- Train cost:0.04125 --- Test cost:0.03336
0.0-0.0 run 1 - epoch 15/700 --- Train cost:0.02747 --- Test cost:0.03105
0.0-0.0 run 1 - epoch 20/700 --- Train cost:0.02192 --- Test cost:0.02927
0.0-0.0 run 1 - epoch 25/700 --- Train cost:0.01856 --- Test cost:0.02839
0.0-0.0 run 1 - epoch 30/700 --- Train cost:0.01619 --- Test cost:0.02840
0.0-0.0 run 1 - epoch 35/700 --- Train cost:0.01447 --- Test cost:0.02881
0.0-0.0 run 1 - epoch 40/700 --- Train cost:0.01311 --- Test cost:0.02902
0.0-0.0 run 1 - epoch 45/700 --- Train cost:0.01186 --- Test cost:0.02896
0.0-0.0 run 1 - epoch 50/700 --- Train cost:0.01078 --- Test cost:0.02889
0.0-0.0 run 1 - epoch 55/700 --- Train cost:0.00986 --- Test cost:0.02908
0.0-0.0 run 1 - epoch 60/700 --- Train cost:0.00904 --- Test cost:0.02942
0.0-0.0 run 1 - epoch 65/700 --- Train cost:0.00832 --- Test cost:0.02980
0.0-0.0 run 1 - epoch 70/700 --- Train cost:0.00768 --- Test cost:0.03032
0.0-0.0 run 1 - epoch 75/700 --- Train cost:0.00712 --- Test cost:0.03113
0.0-0.0 run 1 - epoch 80/700 --- Train cost:0.00661 --- Test cost:0.03226
0.0-0.0 run 1 - epoch 85/700 --- Train cost:0.00615 --- Test cost:0.03358
0.0-0.0 run 1 - epoch 90/700 --- Train cost:0.00573 --- Test cost:0.03490
0.0-0.0 run 1 - epoch 95/700 --- Train cost:0.00535 --- Test cost:0.03619
0.0-0.0 run 1 - epoch 100/700 --- Train cost:0.00501 --- Test cost:0.03755
0.0-0.0 run 1 - epoch 105/700 --- Train cost:0.00469 --- Test cost:0.03904
0.0-0.0 run 1 - epoch 110/700 --- Train cost:0.00440 --- Test cost:0.04061
0.0-0.0 run 1 - epoch 115/700 --- Train cost:0.00413 --- Test cost:0.04219
0.0-0.0 run 1 - epoch 120/700 --- Train cost:0.00388 --- Test cost:0.04374
0.0-0.0 run 1 - epoch 125/700 --- Train cost:0.00365 --- Test cost:0.04523
0.0-0.0 run 1 - epoch 130/700 --- Train cost:0.00343 --- Test cost:0.04665
0.0-0.0 run 1 - epoch 135/700 --- Train cost:0.00323 --- Test cost:0.04802
0.0-0.0 run 1 - epoch 140/700 --- Train cost:0.00303 --- Test cost:0.04937
0.0-0.0 run 1 - epoch 145/700 --- Train cost:0.00285 --- Test cost:0.05070
0.0-0.0 run 1 - epoch 150/700 --- Train cost:0.00268 --- Test cost:0.05199
0.0-0.0 run 1 - epoch 155/700 --- Train cost:0.00252 --- Test cost:0.05326
0.0-0.0 run 1 - epoch 160/700 --- Train cost:0.00237 --- Test cost:0.05451
0.0-0.0 run 1 - epoch 165/700 --- Train cost:0.00223 --- Test cost:0.05574
0.0-0.0 run 1 - epoch 170/700 --- Train cost:0.00210 --- Test cost:0.05694
0.0-0.0 run 1 - epoch 175/700 --- Train cost:0.00198 --- Test cost:0.05813
0.0-0.0 run 1 - epoch 180/700 --- Train cost:0.00186 --- Test cost:0.05929
0.0-0.0 run 1 - epoch 185/700 --- Train cost:0.00175 --- Test cost:0.06042
0.0-0.0 run 1 - epoch 190/700 --- Train cost:0.00166 --- Test cost:0.06153
0.0-0.0 run 1 - epoch 195/700 --- Train cost:0.00156 --- Test cost:0.06260
0.0-0.0 run 1 - epoch 200/700 --- Train cost:0.00148 --- Test cost:0.06364
0.0-0.0 run 1 - epoch 205/700 --- Train cost:0.00140 --- Test cost:0.06466
0.0-0.0 run 1 - epoch 210/700 --- Train cost:0.00133 --- Test cost:0.06564
0.0-0.0 run 1 - epoch 215/700 --- Train cost:0.00126 --- Test cost:0.06659
0.0-0.0 run 1 - epoch 220/700 --- Train cost:0.00119 --- Test cost:0.06751
0.0-0.0 run 1 - epoch 225/700 --- Train cost:0.00113 --- Test cost:0.06840
0.0-0.0 run 1 - epoch 230/700 --- Train cost:0.00108 --- Test cost:0.06926
0.0-0.0 run 1 - epoch 235/700 --- Train cost:0.00103 --- Test cost:0.07010
0.0-0.0 run 1 - epoch 240/700 --- Train cost:0.00098 --- Test cost:0.07091
0.0-0.0 run 1 - epoch 245/700 --- Train cost:0.00093 --- Test cost:0.07169
0.0-0.0 run 1 - epoch 250/700 --- Train cost:0.00089 --- Test cost:0.07244
0.0-0.0 run 1 - epoch 255/700 --- Train cost:0.00085 --- Test cost:0.07317
0.0-0.0 run 1 - epoch 260/700 --- Train cost:0.00081 --- Test cost:0.07388
0.0-0.0 run 1 - epoch 265/700 --- Train cost:0.00078 --- Test cost:0.07456
0.0-0.0 run 1 - epoch 270/700 --- Train cost:0.00075 --- Test cost:0.07522
0.0-0.0 run 1 - epoch 275/700 --- Train cost:0.00072 --- Test cost:0.07586
0.0-0.0 run 1 - epoch 280/700 --- Train cost:0.00069 --- Test cost:0.07647
0.0-0.0 run 1 - epoch 285/700 --- Train cost:0.00066 --- Test cost:0.07707
0.0-0.0 run 1 - epoch 290/700 --- Train cost:0.00063 --- Test cost:0.07764
0.0-0.0 run 1 - epoch 295/700 --- Train cost:0.00061 --- Test cost:0.07819
0.0-0.0 run 1 - epoch 300/700 --- Train cost:0.00058 --- Test cost:0.07873
0.0-0.0 run 1 - epoch 305/700 --- Train cost:0.00056 --- Test cost:0.07924
0.0-0.0 run 1 - epoch 310/700 --- Train cost:0.00054 --- Test cost:0.07974
0.0-0.0 run 1 - epoch 315/700 --- Train cost:0.00052 --- Test cost:0.08022
0.0-0.0 run 1 - epoch 320/700 --- Train cost:0.00050 --- Test cost:0.08068
0.0-0.0 run 1 - epoch 325/700 --- Train cost:0.00048 --- Test cost:0.08113
0.0-0.0 run 1 - epoch 330/700 --- Train cost:0.00046 --- Test cost:0.08156
0.0-0.0 run 1 - epoch 335/700 --- Train cost:0.00045 --- Test cost:0.08198
0.0-0.0 run 1 - epoch 340/700 --- Train cost:0.00043 --- Test cost:0.08238
0.0-0.0 run 1 - epoch 345/700 --- Train cost:0.00042 --- Test cost:0.08277
0.0-0.0 run 1 - epoch 350/700 --- Train cost:0.00040 --- Test cost:0.08314
0.0-0.0 run 1 - epoch 355/700 --- Train cost:0.00039 --- Test cost:0.08351
0.0-0.0 run 1 - epoch 360/700 --- Train cost:0.00038 --- Test cost:0.08386
0.0-0.0 run 1 - epoch 365/700 --- Train cost:0.00036 --- Test cost:0.08420
0.0-0.0 run 1 - epoch 370/700 --- Train cost:0.00035 --- Test cost:0.08453
0.0-0.0 run 1 - epoch 375/700 --- Train cost:0.00034 --- Test cost:0.08485
0.0-0.0 run 1 - epoch 380/700 --- Train cost:0.00033 --- Test cost:0.08515
0.0-0.0 run 1 - epoch 385/700 --- Train cost:0.00032 --- Test cost:0.08545
0.0-0.0 run 1 - epoch 390/700 --- Train cost:0.00031 --- Test cost:0.08574
0.0-0.0 run 1 - epoch 395/700 --- Train cost:0.00030 --- Test cost:0.08602
0.0-0.0 run 1 - epoch 400/700 --- Train cost:0.00029 --- Test cost:0.08629
0.0-0.0 run 1 - epoch 405/700 --- Train cost:0.00028 --- Test cost:0.08656
0.0-0.0 run 1 - epoch 410/700 --- Train cost:0.00028 --- Test cost:0.08681
0.0-0.0 run 1 - epoch 415/700 --- Train cost:0.00027 --- Test cost:0.08706
0.0-0.0 run 1 - epoch 420/700 --- Train cost:0.00026 --- Test cost:0.08730
0.0-0.0 run 1 - epoch 425/700 --- Train cost:0.00025 --- Test cost:0.08754
0.0-0.0 run 1 - epoch 430/700 --- Train cost:0.00025 --- Test cost:0.08777
0.0-0.0 run 1 - epoch 435/700 --- Train cost:0.00024 --- Test cost:0.08799
0.0-0.0 run 1 - epoch 440/700 --- Train cost:0.00023 --- Test cost:0.08821
0.0-0.0 run 1 - epoch 445/700 --- Train cost:0.00023 --- Test cost:0.08842
0.0-0.0 run 1 - epoch 450/700 --- Train cost:0.00022 --- Test cost:0.08863
0.0-0.0 run 1 - epoch 455/700 --- Train cost:0.00022 --- Test cost:0.08883
0.0-0.0 run 1 - epoch 460/700 --- Train cost:0.00021 --- Test cost:0.08902
0.0-0.0 run 1 - epoch 465/700 --- Train cost:0.00020 --- Test cost:0.08922
0.0-0.0 run 1 - epoch 470/700 --- Train cost:0.00020 --- Test cost:0.08940
0.0-0.0 run 1 - epoch 475/700 --- Train cost:0.00020 --- Test cost:0.08959
0.0-0.0 run 1 - epoch 480/700 --- Train cost:0.00019 --- Test cost:0.08977
0.0-0.0 run 1 - epoch 485/700 --- Train cost:0.00019 --- Test cost:0.08995
0.0-0.0 run 1 - epoch 490/700 --- Train cost:0.00018 --- Test cost:0.09012
0.0-0.0 run 1 - epoch 495/700 --- Train cost:0.00018 --- Test cost:0.09029
0.0-0.0 run 1 - epoch 500/700 --- Train cost:0.00017 --- Test cost:0.09045
0.0-0.0 run 1 - epoch 505/700 --- Train cost:0.00017 --- Test cost:0.09062
0.0-0.0 run 1 - epoch 510/700 --- Train cost:0.00017 --- Test cost:0.09078
0.0-0.0 run 1 - epoch 515/700 --- Train cost:0.00016 --- Test cost:0.09094
0.0-0.0 run 1 - epoch 520/700 --- Train cost:0.00016 --- Test cost:0.09109
0.0-0.0 run 1 - epoch 525/700 --- Train cost:0.00015 --- Test cost:0.09124
0.0-0.0 run 1 - epoch 530/700 --- Train cost:0.00015 --- Test cost:0.09139
0.0-0.0 run 1 - epoch 535/700 --- Train cost:0.00015 --- Test cost:0.09154
0.0-0.0 run 1 - epoch 540/700 --- Train cost:0.00014 --- Test cost:0.09169
0.0-0.0 run 1 - epoch 545/700 --- Train cost:0.00014 --- Test cost:0.09183
0.0-0.0 run 1 - epoch 550/700 --- Train cost:0.00014 --- Test cost:0.09197
0.0-0.0 run 1 - epoch 555/700 --- Train cost:0.00014 --- Test cost:0.09211
0.0-0.0 run 1 - epoch 560/700 --- Train cost:0.00013 --- Test cost:0.09225
0.0-0.0 run 1 - epoch 565/700 --- Train cost:0.00013 --- Test cost:0.09239
0.0-0.0 run 1 - epoch 570/700 --- Train cost:0.00013 --- Test cost:0.09252
0.0-0.0 run 1 - epoch 575/700 --- Train cost:0.00012 --- Test cost:0.09265
0.0-0.0 run 1 - epoch 580/700 --- Train cost:0.00012 --- Test cost:0.09279
0.0-0.0 run 1 - epoch 585/700 --- Train cost:0.00012 --- Test cost:0.09292
0.0-0.0 run 1 - epoch 590/700 --- Train cost:0.00012 --- Test cost:0.09304
0.0-0.0 run 1 - epoch 595/700 --- Train cost:0.00011 --- Test cost:0.09317
0.0-0.0 run 1 - epoch 600/700 --- Train cost:0.00011 --- Test cost:0.09330
0.0-0.0 run 1 - epoch 605/700 --- Train cost:0.00011 --- Test cost:0.09342
0.0-0.0 run 1 - epoch 610/700 --- Train cost:0.00011 --- Test cost:0.09355
0.0-0.0 run 1 - epoch 615/700 --- Train cost:0.00011 --- Test cost:0.09367
0.0-0.0 run 1 - epoch 620/700 --- Train cost:0.00010 --- Test cost:0.09379
0.0-0.0 run 1 - epoch 625/700 --- Train cost:0.00010 --- Test cost:0.09391
0.0-0.0 run 1 - epoch 630/700 --- Train cost:0.00010 --- Test cost:0.09403
0.0-0.0 run 1 - epoch 635/700 --- Train cost:0.00010 --- Test cost:0.09415
0.0-0.0 run 1 - epoch 640/700 --- Train cost:0.00010 --- Test cost:0.09426
0.0-0.0 run 1 - epoch 645/700 --- Train cost:0.00009 --- Test cost:0.09438
0.0-0.0 run 1 - epoch 650/700 --- Train cost:0.00009 --- Test cost:0.09450
0.0-0.0 run 1 - epoch 655/700 --- Train cost:0.00009 --- Test cost:0.09461
0.0-0.0 run 1 - epoch 660/700 --- Train cost:0.00009 --- Test cost:0.09472
0.0-0.0 run 1 - epoch 665/700 --- Train cost:0.00009 --- Test cost:0.09484
0.0-0.0 run 1 - epoch 670/700 --- Train cost:0.00008 --- Test cost:0.09495
0.0-0.0 run 1 - epoch 675/700 --- Train cost:0.00008 --- Test cost:0.09506
0.0-0.0 run 1 - epoch 680/700 --- Train cost:0.00008 --- Test cost:0.09517
0.0-0.0 run 1 - epoch 685/700 --- Train cost:0.00008 --- Test cost:0.09528
0.0-0.0 run 1 - epoch 690/700 --- Train cost:0.00008 --- Test cost:0.09539
0.0-0.0 run 1 - epoch 695/700 --- Train cost:0.00008 --- Test cost:0.09550
0.0-0.0 run 2 - epoch 0/700 --- Train cost:0.32211 --- Test cost:0.14482
0.0-0.0 run 2 - epoch 5/700 --- Train cost:0.15515 --- Test cost:0.07197
0.0-0.0 run 2 - epoch 10/700 --- Train cost:0.07803 --- Test cost:0.04644
0.0-0.0 run 2 - epoch 15/700 --- Train cost:0.04212 --- Test cost:0.04713
0.0-0.0 run 2 - epoch 20/700 --- Train cost:0.02660 --- Test cost:0.06245
0.0-0.0 run 2 - epoch 25/700 --- Train cost:0.01997 --- Test cost:0.07354
0.0-0.0 run 2 - epoch 30/700 --- Train cost:0.01639 --- Test cost:0.07600
0.0-0.0 run 2 - epoch 35/700 --- Train cost:0.01419 --- Test cost:0.07577
0.0-0.0 run 2 - epoch 40/700 --- Train cost:0.01266 --- Test cost:0.07584
0.0-0.0 run 2 - epoch 45/700 --- Train cost:0.01150 --- Test cost:0.07567
0.0-0.0 run 2 - epoch 50/700 --- Train cost:0.01058 --- Test cost:0.07488
0.0-0.0 run 2 - epoch 55/700 --- Train cost:0.00986 --- Test cost:0.07427
0.0-0.0 run 2 - epoch 60/700 --- Train cost:0.00924 --- Test cost:0.07422
0.0-0.0 run 2 - epoch 65/700 --- Train cost:0.00872 --- Test cost:0.07441
0.0-0.0 run 2 - epoch 70/700 --- Train cost:0.00828 --- Test cost:0.07487
0.0-0.0 run 2 - epoch 75/700 --- Train cost:0.00789 --- Test cost:0.07585
0.0-0.0 run 2 - epoch 80/700 --- Train cost:0.00754 --- Test cost:0.07722
0.0-0.0 run 2 - epoch 85/700 --- Train cost:0.00722 --- Test cost:0.07862
0.0-0.0 run 2 - epoch 90/700 --- Train cost:0.00692 --- Test cost:0.07992
0.0-0.0 run 2 - epoch 95/700 --- Train cost:0.00664 --- Test cost:0.08116
0.0-0.0 run 2 - epoch 100/700 --- Train cost:0.00638 --- Test cost:0.08232
0.0-0.0 run 2 - epoch 105/700 --- Train cost:0.00613 --- Test cost:0.08338
0.0-0.0 run 2 - epoch 110/700 --- Train cost:0.00589 --- Test cost:0.08434
0.0-0.0 run 2 - epoch 115/700 --- Train cost:0.00566 --- Test cost:0.08524
0.0-0.0 run 2 - epoch 120/700 --- Train cost:0.00544 --- Test cost:0.08614
0.0-0.0 run 2 - epoch 125/700 --- Train cost:0.00522 --- Test cost:0.08703
0.0-0.0 run 2 - epoch 130/700 --- Train cost:0.00502 --- Test cost:0.08789
0.0-0.0 run 2 - epoch 135/700 --- Train cost:0.00482 --- Test cost:0.08876
0.0-0.0 run 2 - epoch 140/700 --- Train cost:0.00463 --- Test cost:0.08965
0.0-0.0 run 2 - epoch 145/700 --- Train cost:0.00445 --- Test cost:0.09055
0.0-0.0 run 2 - epoch 150/700 --- Train cost:0.00427 --- Test cost:0.09145
0.0-0.0 run 2 - epoch 155/700 --- Train cost:0.00410 --- Test cost:0.09238
0.0-0.0 run 2 - epoch 160/700 --- Train cost:0.00394 --- Test cost:0.09332
0.0-0.0 run 2 - epoch 165/700 --- Train cost:0.00378 --- Test cost:0.09428
0.0-0.0 run 2 - epoch 170/700 --- Train cost:0.00362 --- Test cost:0.09526
0.0-0.0 run 2 - epoch 175/700 --- Train cost:0.00347 --- Test cost:0.09625
0.0-0.0 run 2 - epoch 180/700 --- Train cost:0.00332 --- Test cost:0.09727
0.0-0.0 run 2 - epoch 185/700 --- Train cost:0.00318 --- Test cost:0.09832
0.0-0.0 run 2 - epoch 190/700 --- Train cost:0.00304 --- Test cost:0.09939
0.0-0.0 run 2 - epoch 195/700 --- Train cost:0.00290 --- Test cost:0.10047
0.0-0.0 run 2 - epoch 200/700 --- Train cost:0.00277 --- Test cost:0.10156
0.0-0.0 run 2 - epoch 205/700 --- Train cost:0.00264 --- Test cost:0.10266
0.0-0.0 run 2 - epoch 210/700 --- Train cost:0.00252 --- Test cost:0.10375
0.0-0.0 run 2 - epoch 215/700 --- Train cost:0.00240 --- Test cost:0.10484
0.0-0.0 run 2 - epoch 220/700 --- Train cost:0.00229 --- Test cost:0.10591
0.0-0.0 run 2 - epoch 225/700 --- Train cost:0.00218 --- Test cost:0.10694
0.0-0.0 run 2 - epoch 230/700 --- Train cost:0.00208 --- Test cost:0.10795
0.0-0.0 run 2 - epoch 235/700 --- Train cost:0.00199 --- Test cost:0.10891
0.0-0.0 run 2 - epoch 240/700 --- Train cost:0.00189 --- Test cost:0.10982
0.0-0.0 run 2 - epoch 245/700 --- Train cost:0.00181 --- Test cost:0.11068
0.0-0.0 run 2 - epoch 250/700 --- Train cost:0.00173 --- Test cost:0.11148
0.0-0.0 run 2 - epoch 255/700 --- Train cost:0.00165 --- Test cost:0.11223
0.0-0.0 run 2 - epoch 260/700 --- Train cost:0.00158 --- Test cost:0.11292
0.0-0.0 run 2 - epoch 265/700 --- Train cost:0.00151 --- Test cost:0.11356
0.0-0.0 run 2 - epoch 270/700 --- Train cost:0.00145 --- Test cost:0.11414
0.0-0.0 run 2 - epoch 275/700 --- Train cost:0.00139 --- Test cost:0.11468
0.0-0.0 run 2 - epoch 280/700 --- Train cost:0.00134 --- Test cost:0.11516
0.0-0.0 run 2 - epoch 285/700 --- Train cost:0.00129 --- Test cost:0.11561
0.0-0.0 run 2 - epoch 290/700 --- Train cost:0.00124 --- Test cost:0.11602
0.0-0.0 run 2 - epoch 295/700 --- Train cost:0.00119 --- Test cost:0.11639
0.0-0.0 run 2 - epoch 300/700 --- Train cost:0.00115 --- Test cost:0.11673
0.0-0.0 run 2 - epoch 305/700 --- Train cost:0.00111 --- Test cost:0.11705
0.0-0.0 run 2 - epoch 310/700 --- Train cost:0.00107 --- Test cost:0.11734
0.0-0.0 run 2 - epoch 315/700 --- Train cost:0.00104 --- Test cost:0.11760
0.0-0.0 run 2 - epoch 320/700 --- Train cost:0.00100 --- Test cost:0.11785
0.0-0.0 run 2 - epoch 325/700 --- Train cost:0.00097 --- Test cost:0.11808
0.0-0.0 run 2 - epoch 330/700 --- Train cost:0.00094 --- Test cost:0.11830
0.0-0.0 run 2 - epoch 335/700 --- Train cost:0.00091 --- Test cost:0.11850
0.0-0.0 run 2 - epoch 340/700 --- Train cost:0.00088 --- Test cost:0.11869
0.0-0.0 run 2 - epoch 345/700 --- Train cost:0.00085 --- Test cost:0.11887
0.0-0.0 run 2 - epoch 350/700 --- Train cost:0.00082 --- Test cost:0.11904
0.0-0.0 run 2 - epoch 355/700 --- Train cost:0.00080 --- Test cost:0.11920
0.0-0.0 run 2 - epoch 360/700 --- Train cost:0.00077 --- Test cost:0.11936
0.0-0.0 run 2 - epoch 365/700 --- Train cost:0.00075 --- Test cost:0.11951
0.0-0.0 run 2 - epoch 370/700 --- Train cost:0.00073 --- Test cost:0.11966
0.0-0.0 run 2 - epoch 375/700 --- Train cost:0.00070 --- Test cost:0.11980
0.0-0.0 run 2 - epoch 380/700 --- Train cost:0.00068 --- Test cost:0.11994
0.0-0.0 run 2 - epoch 385/700 --- Train cost:0.00066 --- Test cost:0.12008
0.0-0.0 run 2 - epoch 390/700 --- Train cost:0.00064 --- Test cost:0.12021
0.0-0.0 run 2 - epoch 395/700 --- Train cost:0.00062 --- Test cost:0.12035
0.0-0.0 run 2 - epoch 400/700 --- Train cost:0.00060 --- Test cost:0.12048
0.0-0.0 run 2 - epoch 405/700 --- Train cost:0.00058 --- Test cost:0.12061
0.0-0.0 run 2 - epoch 410/700 --- Train cost:0.00056 --- Test cost:0.12074
0.0-0.0 run 2 - epoch 415/700 --- Train cost:0.00054 --- Test cost:0.12087
0.0-0.0 run 2 - epoch 420/700 --- Train cost:0.00053 --- Test cost:0.12100
0.0-0.0 run 2 - epoch 425/700 --- Train cost:0.00051 --- Test cost:0.12113
0.0-0.0 run 2 - epoch 430/700 --- Train cost:0.00049 --- Test cost:0.12126
0.0-0.0 run 2 - epoch 435/700 --- Train cost:0.00048 --- Test cost:0.12139
0.0-0.0 run 2 - epoch 440/700 --- Train cost:0.00046 --- Test cost:0.12152
0.0-0.0 run 2 - epoch 445/700 --- Train cost:0.00045 --- Test cost:0.12165
0.0-0.0 run 2 - epoch 450/700 --- Train cost:0.00043 --- Test cost:0.12179
0.0-0.0 run 2 - epoch 455/700 --- Train cost:0.00042 --- Test cost:0.12192
0.0-0.0 run 2 - epoch 460/700 --- Train cost:0.00040 --- Test cost:0.12206
0.0-0.0 run 2 - epoch 465/700 --- Train cost:0.00039 --- Test cost:0.12219
0.0-0.0 run 2 - epoch 470/700 --- Train cost:0.00038 --- Test cost:0.12233
0.0-0.0 run 2 - epoch 475/700 --- Train cost:0.00036 --- Test cost:0.12247
0.0-0.0 run 2 - epoch 480/700 --- Train cost:0.00035 --- Test cost:0.12261
0.0-0.0 run 2 - epoch 485/700 --- Train cost:0.00034 --- Test cost:0.12275
0.0-0.0 run 2 - epoch 490/700 --- Train cost:0.00033 --- Test cost:0.12289
0.0-0.0 run 2 - epoch 495/700 --- Train cost:0.00032 --- Test cost:0.12303
0.0-0.0 run 2 - epoch 500/700 --- Train cost:0.00031 --- Test cost:0.12317
0.0-0.0 run 2 - epoch 505/700 --- Train cost:0.00030 --- Test cost:0.12331
0.0-0.0 run 2 - epoch 510/700 --- Train cost:0.00028 --- Test cost:0.12345
0.0-0.0 run 2 - epoch 515/700 --- Train cost:0.00028 --- Test cost:0.12360
0.0-0.0 run 2 - epoch 520/700 --- Train cost:0.00027 --- Test cost:0.12374
0.0-0.0 run 2 - epoch 525/700 --- Train cost:0.00026 --- Test cost:0.12388
0.0-0.0 run 2 - epoch 530/700 --- Train cost:0.00025 --- Test cost:0.12402
0.0-0.0 run 2 - epoch 535/700 --- Train cost:0.00024 --- Test cost:0.12417
0.0-0.0 run 2 - epoch 540/700 --- Train cost:0.00023 --- Test cost:0.12431
0.0-0.0 run 2 - epoch 545/700 --- Train cost:0.00022 --- Test cost:0.12445
0.0-0.0 run 2 - epoch 550/700 --- Train cost:0.00022 --- Test cost:0.12459
0.0-0.0 run 2 - epoch 555/700 --- Train cost:0.00021 --- Test cost:0.12473
0.0-0.0 run 2 - epoch 560/700 --- Train cost:0.00020 --- Test cost:0.12486
0.0-0.0 run 2 - epoch 565/700 --- Train cost:0.00020 --- Test cost:0.12500
0.0-0.0 run 2 - epoch 570/700 --- Train cost:0.00019 --- Test cost:0.12514
0.0-0.0 run 2 - epoch 575/700 --- Train cost:0.00018 --- Test cost:0.12527
0.0-0.0 run 2 - epoch 580/700 --- Train cost:0.00018 --- Test cost:0.12540
0.0-0.0 run 2 - epoch 585/700 --- Train cost:0.00017 --- Test cost:0.12553
0.0-0.0 run 2 - epoch 590/700 --- Train cost:0.00017 --- Test cost:0.12566
0.0-0.0 run 2 - epoch 595/700 --- Train cost:0.00016 --- Test cost:0.12578
0.0-0.0 run 2 - epoch 600/700 --- Train cost:0.00016 --- Test cost:0.12591
0.0-0.0 run 2 - epoch 605/700 --- Train cost:0.00015 --- Test cost:0.12603
0.0-0.0 run 2 - epoch 610/700 --- Train cost:0.00015 --- Test cost:0.12615
0.0-0.0 run 2 - epoch 615/700 --- Train cost:0.00014 --- Test cost:0.12626
0.0-0.0 run 2 - epoch 620/700 --- Train cost:0.00014 --- Test cost:0.12638
0.0-0.0 run 2 - epoch 625/700 --- Train cost:0.00013 --- Test cost:0.12649
0.0-0.0 run 2 - epoch 630/700 --- Train cost:0.00013 --- Test cost:0.12660
0.0-0.0 run 2 - epoch 635/700 --- Train cost:0.00013 --- Test cost:0.12671
0.0-0.0 run 2 - epoch 640/700 --- Train cost:0.00012 --- Test cost:0.12681
0.0-0.0 run 2 - epoch 645/700 --- Train cost:0.00012 --- Test cost:0.12691
0.0-0.0 run 2 - epoch 650/700 --- Train cost:0.00012 --- Test cost:0.12701
0.0-0.0 run 2 - epoch 655/700 --- Train cost:0.00011 --- Test cost:0.12710
0.0-0.0 run 2 - epoch 660/700 --- Train cost:0.00011 --- Test cost:0.12720
0.0-0.0 run 2 - epoch 665/700 --- Train cost:0.00011 --- Test cost:0.12729
0.0-0.0 run 2 - epoch 670/700 --- Train cost:0.00011 --- Test cost:0.12737
0.0-0.0 run 2 - epoch 675/700 --- Train cost:0.00010 --- Test cost:0.12746
0.0-0.0 run 2 - epoch 680/700 --- Train cost:0.00010 --- Test cost:0.12754
0.0-0.0 run 2 - epoch 685/700 --- Train cost:0.00010 --- Test cost:0.12762
0.0-0.0 run 2 - epoch 690/700 --- Train cost:0.00010 --- Test cost:0.12769
0.0-0.0 run 2 - epoch 695/700 --- Train cost:0.00009 --- Test cost:0.12777
0.3-0.2 run 0 - epoch 0/700 --- Train cost:0.47606 --- Test cost:0.17619
0.3-0.2 run 0 - epoch 5/700 --- Train cost:0.40491 --- Test cost:0.15495
0.3-0.2 run 0 - epoch 10/700 --- Train cost:0.36241 --- Test cost:0.13692
0.3-0.2 run 0 - epoch 15/700 --- Train cost:0.31689 --- Test cost:0.12580
0.3-0.2 run 0 - epoch 20/700 --- Train cost:0.27189 --- Test cost:0.11358
0.3-0.2 run 0 - epoch 25/700 --- Train cost:0.24435 --- Test cost:0.11085
0.3-0.2 run 0 - epoch 30/700 --- Train cost:0.23198 --- Test cost:0.11171
0.3-0.2 run 0 - epoch 35/700 --- Train cost:0.20060 --- Test cost:0.10482
0.3-0.2 run 0 - epoch 40/700 --- Train cost:0.15768 --- Test cost:0.09010
0.3-0.2 run 0 - epoch 45/700 --- Train cost:0.11749 --- Test cost:0.07223
0.3-0.2 run 0 - epoch 50/700 --- Train cost:0.08208 --- Test cost:0.05499
0.3-0.2 run 0 - epoch 55/700 --- Train cost:0.06563 --- Test cost:0.04828
0.3-0.2 run 0 - epoch 60/700 --- Train cost:0.05824 --- Test cost:0.04427
0.3-0.2 run 0 - epoch 65/700 --- Train cost:0.05473 --- Test cost:0.04176
0.3-0.2 run 0 - epoch 70/700 --- Train cost:0.05463 --- Test cost:0.04203
0.3-0.2 run 0 - epoch 75/700 --- Train cost:0.05668 --- Test cost:0.04220
0.3-0.2 run 0 - epoch 80/700 --- Train cost:0.05745 --- Test cost:0.04136
0.3-0.2 run 0 - epoch 85/700 --- Train cost:0.05734 --- Test cost:0.04099
0.3-0.2 run 0 - epoch 90/700 --- Train cost:0.05854 --- Test cost:0.04100
0.3-0.2 run 0 - epoch 95/700 --- Train cost:0.06390 --- Test cost:0.04238
0.3-0.2 run 0 - epoch 100/700 --- Train cost:0.06861 --- Test cost:0.04367
0.3-0.2 run 0 - epoch 105/700 --- Train cost:0.06949 --- Test cost:0.04351
0.3-0.2 run 0 - epoch 110/700 --- Train cost:0.06846 --- Test cost:0.04309
0.3-0.2 run 0 - epoch 115/700 --- Train cost:0.06571 --- Test cost:0.04127
0.3-0.2 run 0 - epoch 120/700 --- Train cost:0.06714 --- Test cost:0.04178
0.3-0.2 run 0 - epoch 125/700 --- Train cost:0.06910 --- Test cost:0.04450
0.3-0.2 run 0 - epoch 130/700 --- Train cost:0.06950 --- Test cost:0.04580
0.3-0.2 run 0 - epoch 135/700 --- Train cost:0.07205 --- Test cost:0.04737
0.3-0.2 run 0 - epoch 140/700 --- Train cost:0.07827 --- Test cost:0.05209
0.3-0.2 run 0 - epoch 145/700 --- Train cost:0.08319 --- Test cost:0.05399
0.3-0.2 run 0 - epoch 150/700 --- Train cost:0.08677 --- Test cost:0.05451
0.3-0.2 run 0 - epoch 155/700 --- Train cost:0.08372 --- Test cost:0.05102
0.3-0.2 run 0 - epoch 160/700 --- Train cost:0.07929 --- Test cost:0.04767
0.3-0.2 run 0 - epoch 165/700 --- Train cost:0.07403 --- Test cost:0.04636
0.3-0.2 run 0 - epoch 170/700 --- Train cost:0.06885 --- Test cost:0.04365
0.3-0.2 run 0 - epoch 175/700 --- Train cost:0.06322 --- Test cost:0.04118
0.3-0.2 run 0 - epoch 180/700 --- Train cost:0.05602 --- Test cost:0.03614
0.3-0.2 run 0 - epoch 185/700 --- Train cost:0.04753 --- Test cost:0.03105
0.3-0.2 run 0 - epoch 190/700 --- Train cost:0.04127 --- Test cost:0.02818
0.3-0.2 run 0 - epoch 195/700 --- Train cost:0.03718 --- Test cost:0.02850
0.3-0.2 run 0 - epoch 200/700 --- Train cost:0.03546 --- Test cost:0.03204
0.3-0.2 run 0 - epoch 205/700 --- Train cost:0.03469 --- Test cost:0.03423
0.3-0.2 run 0 - epoch 210/700 --- Train cost:0.03399 --- Test cost:0.03465
0.3-0.2 run 0 - epoch 215/700 --- Train cost:0.03328 --- Test cost:0.03534
0.3-0.2 run 0 - epoch 220/700 --- Train cost:0.03072 --- Test cost:0.03649
0.3-0.2 run 0 - epoch 225/700 --- Train cost:0.02834 --- Test cost:0.03625
0.3-0.2 run 0 - epoch 230/700 --- Train cost:0.02672 --- Test cost:0.03533
0.3-0.2 run 0 - epoch 235/700 --- Train cost:0.02428 --- Test cost:0.03428
0.3-0.2 run 0 - epoch 240/700 --- Train cost:0.02221 --- Test cost:0.03201
0.3-0.2 run 0 - epoch 245/700 --- Train cost:0.02076 --- Test cost:0.02863
0.3-0.2 run 0 - epoch 250/700 --- Train cost:0.01993 --- Test cost:0.02707
0.3-0.2 run 0 - epoch 255/700 --- Train cost:0.01938 --- Test cost:0.02727
0.3-0.2 run 0 - epoch 260/700 --- Train cost:0.01960 --- Test cost:0.02682
0.3-0.2 run 0 - epoch 265/700 --- Train cost:0.01931 --- Test cost:0.02692
0.3-0.2 run 0 - epoch 270/700 --- Train cost:0.01890 --- Test cost:0.02738
0.3-0.2 run 0 - epoch 275/700 --- Train cost:0.01884 --- Test cost:0.02744
0.3-0.2 run 0 - epoch 280/700 --- Train cost:0.01797 --- Test cost:0.02864
0.3-0.2 run 0 - epoch 285/700 --- Train cost:0.01717 --- Test cost:0.02974
0.3-0.2 run 0 - epoch 290/700 --- Train cost:0.01682 --- Test cost:0.03020
0.3-0.2 run 0 - epoch 295/700 --- Train cost:0.01683 --- Test cost:0.02911
0.3-0.2 run 0 - epoch 300/700 --- Train cost:0.01660 --- Test cost:0.02998
0.3-0.2 run 0 - epoch 305/700 --- Train cost:0.01617 --- Test cost:0.03147
0.3-0.2 run 0 - epoch 310/700 --- Train cost:0.01655 --- Test cost:0.02997
0.3-0.2 run 0 - epoch 315/700 --- Train cost:0.01699 --- Test cost:0.02838
0.3-0.2 run 0 - epoch 320/700 --- Train cost:0.01758 --- Test cost:0.02794
0.3-0.2 run 0 - epoch 325/700 --- Train cost:0.01870 --- Test cost:0.02761
0.3-0.2 run 0 - epoch 330/700 --- Train cost:0.01911 --- Test cost:0.02595
0.3-0.2 run 0 - epoch 335/700 --- Train cost:0.02007 --- Test cost:0.02274
0.3-0.2 run 0 - epoch 340/700 --- Train cost:0.02188 --- Test cost:0.02069
0.3-0.2 run 0 - epoch 345/700 --- Train cost:0.02271 --- Test cost:0.01895
0.3-0.2 run 0 - epoch 350/700 --- Train cost:0.02297 --- Test cost:0.01911
0.3-0.2 run 0 - epoch 355/700 --- Train cost:0.02420 --- Test cost:0.01979
0.3-0.2 run 0 - epoch 360/700 --- Train cost:0.02649 --- Test cost:0.01930
0.3-0.2 run 0 - epoch 365/700 --- Train cost:0.02836 --- Test cost:0.02010
0.3-0.2 run 0 - epoch 370/700 --- Train cost:0.02728 --- Test cost:0.02165
0.3-0.2 run 0 - epoch 375/700 --- Train cost:0.02535 --- Test cost:0.02276
0.3-0.2 run 0 - epoch 380/700 --- Train cost:0.02468 --- Test cost:0.02406
0.3-0.2 run 0 - epoch 385/700 --- Train cost:0.02509 --- Test cost:0.02421
0.3-0.2 run 0 - epoch 390/700 --- Train cost:0.02393 --- Test cost:0.02529
0.3-0.2 run 0 - epoch 395/700 --- Train cost:0.02267 --- Test cost:0.02655
0.3-0.2 run 0 - epoch 400/700 --- Train cost:0.02049 --- Test cost:0.02755
0.3-0.2 run 0 - epoch 405/700 --- Train cost:0.01956 --- Test cost:0.02625
0.3-0.2 run 0 - epoch 410/700 --- Train cost:0.01878 --- Test cost:0.02549
0.3-0.2 run 0 - epoch 415/700 --- Train cost:0.01875 --- Test cost:0.02229
0.3-0.2 run 0 - epoch 420/700 --- Train cost:0.01866 --- Test cost:0.02048
0.3-0.2 run 0 - epoch 425/700 --- Train cost:0.01779 --- Test cost:0.01958
0.3-0.2 run 0 - epoch 430/700 --- Train cost:0.01675 --- Test cost:0.02042
0.3-0.2 run 0 - epoch 435/700 --- Train cost:0.01615 --- Test cost:0.02255
0.3-0.2 run 0 - epoch 440/700 --- Train cost:0.01546 --- Test cost:0.02485
0.3-0.2 run 0 - epoch 445/700 --- Train cost:0.01523 --- Test cost:0.02668
0.3-0.2 run 0 - epoch 450/700 --- Train cost:0.01568 --- Test cost:0.02932
0.3-0.2 run 0 - epoch 455/700 --- Train cost:0.01658 --- Test cost:0.03298
0.3-0.2 run 0 - epoch 460/700 --- Train cost:0.01744 --- Test cost:0.03585
0.3-0.2 run 0 - epoch 465/700 --- Train cost:0.01803 --- Test cost:0.03597
0.3-0.2 run 0 - epoch 470/700 --- Train cost:0.01955 --- Test cost:0.03434
0.3-0.2 run 0 - epoch 475/700 --- Train cost:0.02067 --- Test cost:0.03210
0.3-0.2 run 0 - epoch 480/700 --- Train cost:0.02037 --- Test cost:0.03026
0.3-0.2 run 0 - epoch 485/700 --- Train cost:0.01895 --- Test cost:0.02938
0.3-0.2 run 0 - epoch 490/700 --- Train cost:0.01774 --- Test cost:0.02958
0.3-0.2 run 0 - epoch 495/700 --- Train cost:0.01723 --- Test cost:0.03170
0.3-0.2 run 0 - epoch 500/700 --- Train cost:0.01692 --- Test cost:0.03297
0.3-0.2 run 0 - epoch 505/700 --- Train cost:0.01671 --- Test cost:0.03376
0.3-0.2 run 0 - epoch 510/700 --- Train cost:0.01683 --- Test cost:0.03293
0.3-0.2 run 0 - epoch 515/700 --- Train cost:0.01770 --- Test cost:0.03231
0.3-0.2 run 0 - epoch 520/700 --- Train cost:0.01787 --- Test cost:0.03141
0.3-0.2 run 0 - epoch 525/700 --- Train cost:0.01720 --- Test cost:0.03014
0.3-0.2 run 0 - epoch 530/700 --- Train cost:0.01619 --- Test cost:0.03057
0.3-0.2 run 0 - epoch 535/700 --- Train cost:0.01601 --- Test cost:0.03083
0.3-0.2 run 0 - epoch 540/700 --- Train cost:0.01612 --- Test cost:0.03070
0.3-0.2 run 0 - epoch 545/700 --- Train cost:0.01574 --- Test cost:0.03290
0.3-0.2 run 0 - epoch 550/700 --- Train cost:0.01625 --- Test cost:0.03325
0.3-0.2 run 0 - epoch 555/700 --- Train cost:0.01698 --- Test cost:0.03418
0.3-0.2 run 0 - epoch 560/700 --- Train cost:0.01668 --- Test cost:0.03583
0.3-0.2 run 0 - epoch 565/700 --- Train cost:0.01682 --- Test cost:0.03725
0.3-0.2 run 0 - epoch 570/700 --- Train cost:0.01708 --- Test cost:0.03564
0.3-0.2 run 0 - epoch 575/700 --- Train cost:0.01755 --- Test cost:0.03299
0.3-0.2 run 0 - epoch 580/700 --- Train cost:0.01787 --- Test cost:0.03198
0.3-0.2 run 0 - epoch 585/700 --- Train cost:0.01694 --- Test cost:0.03235
0.3-0.2 run 0 - epoch 590/700 --- Train cost:0.01641 --- Test cost:0.03314
0.3-0.2 run 0 - epoch 595/700 --- Train cost:0.01679 --- Test cost:0.03300
0.3-0.2 run 0 - epoch 600/700 --- Train cost:0.01819 --- Test cost:0.03303
0.3-0.2 run 0 - epoch 605/700 --- Train cost:0.01947 --- Test cost:0.03337
0.3-0.2 run 0 - epoch 610/700 --- Train cost:0.01828 --- Test cost:0.03462
0.3-0.2 run 0 - epoch 615/700 --- Train cost:0.01640 --- Test cost:0.03760
0.3-0.2 run 0 - epoch 620/700 --- Train cost:0.01539 --- Test cost:0.03958
0.3-0.2 run 0 - epoch 625/700 --- Train cost:0.01417 --- Test cost:0.04136
0.3-0.2 run 0 - epoch 630/700 --- Train cost:0.01333 --- Test cost:0.04329
0.3-0.2 run 0 - epoch 635/700 --- Train cost:0.01219 --- Test cost:0.04554
0.3-0.2 run 0 - epoch 640/700 --- Train cost:0.01101 --- Test cost:0.04733
0.3-0.2 run 0 - epoch 645/700 --- Train cost:0.01051 --- Test cost:0.04716
0.3-0.2 run 0 - epoch 650/700 --- Train cost:0.01033 --- Test cost:0.04391
0.3-0.2 run 0 - epoch 655/700 --- Train cost:0.01065 --- Test cost:0.04019
0.3-0.2 run 0 - epoch 660/700 --- Train cost:0.01158 --- Test cost:0.03914
0.3-0.2 run 0 - epoch 665/700 --- Train cost:0.01218 --- Test cost:0.04120
0.3-0.2 run 0 - epoch 670/700 --- Train cost:0.01252 --- Test cost:0.04520
0.3-0.2 run 0 - epoch 675/700 --- Train cost:0.01322 --- Test cost:0.04921
0.3-0.2 run 0 - epoch 680/700 --- Train cost:0.01317 --- Test cost:0.05070
0.3-0.2 run 0 - epoch 685/700 --- Train cost:0.01265 --- Test cost:0.05084
0.3-0.2 run 0 - epoch 690/700 --- Train cost:0.01196 --- Test cost:0.05020
0.3-0.2 run 0 - epoch 695/700 --- Train cost:0.01165 --- Test cost:0.04836
0.3-0.2 run 1 - epoch 0/700 --- Train cost:0.29753 --- Test cost:0.10869
0.3-0.2 run 1 - epoch 5/700 --- Train cost:0.26330 --- Test cost:0.08804
0.3-0.2 run 1 - epoch 10/700 --- Train cost:0.21336 --- Test cost:0.07231
0.3-0.2 run 1 - epoch 15/700 --- Train cost:0.17060 --- Test cost:0.05948
0.3-0.2 run 1 - epoch 20/700 --- Train cost:0.12846 --- Test cost:0.05353
0.3-0.2 run 1 - epoch 25/700 --- Train cost:0.09453 --- Test cost:0.05322
0.3-0.2 run 1 - epoch 30/700 --- Train cost:0.07272 --- Test cost:0.05897
0.3-0.2 run 1 - epoch 35/700 --- Train cost:0.06275 --- Test cost:0.06653
0.3-0.2 run 1 - epoch 40/700 --- Train cost:0.05683 --- Test cost:0.07305
0.3-0.2 run 1 - epoch 45/700 --- Train cost:0.04927 --- Test cost:0.07256
0.3-0.2 run 1 - epoch 50/700 --- Train cost:0.03885 --- Test cost:0.06714
0.3-0.2 run 1 - epoch 55/700 --- Train cost:0.03106 --- Test cost:0.06172
0.3-0.2 run 1 - epoch 60/700 --- Train cost:0.02770 --- Test cost:0.05626
0.3-0.2 run 1 - epoch 65/700 --- Train cost:0.02680 --- Test cost:0.05032
0.3-0.2 run 1 - epoch 70/700 --- Train cost:0.02594 --- Test cost:0.04432
0.3-0.2 run 1 - epoch 75/700 --- Train cost:0.02597 --- Test cost:0.04153
0.3-0.2 run 1 - epoch 80/700 --- Train cost:0.02653 --- Test cost:0.04166
0.3-0.2 run 1 - epoch 85/700 --- Train cost:0.02755 --- Test cost:0.04234
0.3-0.2 run 1 - epoch 90/700 --- Train cost:0.02812 --- Test cost:0.04202
0.3-0.2 run 1 - epoch 95/700 --- Train cost:0.02861 --- Test cost:0.04093
0.3-0.2 run 1 - epoch 100/700 --- Train cost:0.02843 --- Test cost:0.04032
0.3-0.2 run 1 - epoch 105/700 --- Train cost:0.02706 --- Test cost:0.04262
0.3-0.2 run 1 - epoch 110/700 --- Train cost:0.02685 --- Test cost:0.04394
0.3-0.2 run 1 - epoch 115/700 --- Train cost:0.02726 --- Test cost:0.04233
0.3-0.2 run 1 - epoch 120/700 --- Train cost:0.02671 --- Test cost:0.04016
0.3-0.2 run 1 - epoch 125/700 --- Train cost:0.02577 --- Test cost:0.04100
0.3-0.2 run 1 - epoch 130/700 --- Train cost:0.02352 --- Test cost:0.04145
0.3-0.2 run 1 - epoch 135/700 --- Train cost:0.02132 --- Test cost:0.04209
0.3-0.2 run 1 - epoch 140/700 --- Train cost:0.01952 --- Test cost:0.04329
0.3-0.2 run 1 - epoch 145/700 --- Train cost:0.01841 --- Test cost:0.04427
0.3-0.2 run 1 - epoch 150/700 --- Train cost:0.01800 --- Test cost:0.04643
0.3-0.2 run 1 - epoch 155/700 --- Train cost:0.01809 --- Test cost:0.04260
0.3-0.2 run 1 - epoch 160/700 --- Train cost:0.01875 --- Test cost:0.03847
0.3-0.2 run 1 - epoch 165/700 --- Train cost:0.01914 --- Test cost:0.03827
0.3-0.2 run 1 - epoch 170/700 --- Train cost:0.01847 --- Test cost:0.03662
0.3-0.2 run 1 - epoch 175/700 --- Train cost:0.01780 --- Test cost:0.03229
0.3-0.2 run 1 - epoch 180/700 --- Train cost:0.01785 --- Test cost:0.02940
0.3-0.2 run 1 - epoch 185/700 --- Train cost:0.01835 --- Test cost:0.02859
0.3-0.2 run 1 - epoch 190/700 --- Train cost:0.01878 --- Test cost:0.02969
0.3-0.2 run 1 - epoch 195/700 --- Train cost:0.02021 --- Test cost:0.02947
0.3-0.2 run 1 - epoch 200/700 --- Train cost:0.02121 --- Test cost:0.02781
0.3-0.2 run 1 - epoch 205/700 --- Train cost:0.02254 --- Test cost:0.02726
0.3-0.2 run 1 - epoch 210/700 --- Train cost:0.02484 --- Test cost:0.02858
0.3-0.2 run 1 - epoch 215/700 --- Train cost:0.02534 --- Test cost:0.02795
0.3-0.2 run 1 - epoch 220/700 --- Train cost:0.02506 --- Test cost:0.02560
0.3-0.2 run 1 - epoch 225/700 --- Train cost:0.02613 --- Test cost:0.02338
0.3-0.2 run 1 - epoch 230/700 --- Train cost:0.02696 --- Test cost:0.02419
0.3-0.2 run 1 - epoch 235/700 --- Train cost:0.02761 --- Test cost:0.02587
0.3-0.2 run 1 - epoch 240/700 --- Train cost:0.02758 --- Test cost:0.02627
0.3-0.2 run 1 - epoch 245/700 --- Train cost:0.02569 --- Test cost:0.02712
0.3-0.2 run 1 - epoch 250/700 --- Train cost:0.02528 --- Test cost:0.02652
0.3-0.2 run 1 - epoch 255/700 --- Train cost:0.02580 --- Test cost:0.02681
0.3-0.2 run 1 - epoch 260/700 --- Train cost:0.02652 --- Test cost:0.02926
0.3-0.2 run 1 - epoch 265/700 --- Train cost:0.02866 --- Test cost:0.03026
0.3-0.2 run 1 - epoch 270/700 --- Train cost:0.03103 --- Test cost:0.02837
0.3-0.2 run 1 - epoch 275/700 --- Train cost:0.03371 --- Test cost:0.02630
0.3-0.2 run 1 - epoch 280/700 --- Train cost:0.03535 --- Test cost:0.02468
0.3-0.2 run 1 - epoch 285/700 --- Train cost:0.03386 --- Test cost:0.02466
0.3-0.2 run 1 - epoch 290/700 --- Train cost:0.02953 --- Test cost:0.02733
0.3-0.2 run 1 - epoch 295/700 --- Train cost:0.02535 --- Test cost:0.03228
0.3-0.2 run 1 - epoch 300/700 --- Train cost:0.02308 --- Test cost:0.03782
0.3-0.2 run 1 - epoch 305/700 --- Train cost:0.02196 --- Test cost:0.04193
0.3-0.2 run 1 - epoch 310/700 --- Train cost:0.02277 --- Test cost:0.03882
0.3-0.2 run 1 - epoch 315/700 --- Train cost:0.02445 --- Test cost:0.03420
0.3-0.2 run 1 - epoch 320/700 --- Train cost:0.02630 --- Test cost:0.03248
0.3-0.2 run 1 - epoch 325/700 --- Train cost:0.02795 --- Test cost:0.03070
0.3-0.2 run 1 - epoch 330/700 --- Train cost:0.02930 --- Test cost:0.03133
0.3-0.2 run 1 - epoch 335/700 --- Train cost:0.02918 --- Test cost:0.03339
0.3-0.2 run 1 - epoch 340/700 --- Train cost:0.02873 --- Test cost:0.03402
0.3-0.2 run 1 - epoch 345/700 --- Train cost:0.02699 --- Test cost:0.03452
0.3-0.2 run 1 - epoch 350/700 --- Train cost:0.02491 --- Test cost:0.03589
0.3-0.2 run 1 - epoch 355/700 --- Train cost:0.02182 --- Test cost:0.04081
0.3-0.2 run 1 - epoch 360/700 --- Train cost:0.01841 --- Test cost:0.04855
0.3-0.2 run 1 - epoch 365/700 --- Train cost:0.01629 --- Test cost:0.05739
0.3-0.2 run 1 - epoch 370/700 --- Train cost:0.01607 --- Test cost:0.05921
0.3-0.2 run 1 - epoch 375/700 --- Train cost:0.01552 --- Test cost:0.05969
0.3-0.2 run 1 - epoch 380/700 --- Train cost:0.01467 --- Test cost:0.06039
0.3-0.2 run 1 - epoch 385/700 --- Train cost:0.01440 --- Test cost:0.05962
0.3-0.2 run 1 - epoch 390/700 --- Train cost:0.01477 --- Test cost:0.05665
0.3-0.2 run 1 - epoch 395/700 --- Train cost:0.01496 --- Test cost:0.05206
0.3-0.2 run 1 - epoch 400/700 --- Train cost:0.01515 --- Test cost:0.04723
0.3-0.2 run 1 - epoch 405/700 --- Train cost:0.01550 --- Test cost:0.04108
0.3-0.2 run 1 - epoch 410/700 --- Train cost:0.01594 --- Test cost:0.03604
0.3-0.2 run 1 - epoch 415/700 --- Train cost:0.01745 --- Test cost:0.03178
0.3-0.2 run 1 - epoch 420/700 --- Train cost:0.01907 --- Test cost:0.03074
0.3-0.2 run 1 - epoch 425/700 --- Train cost:0.02072 --- Test cost:0.03100
0.3-0.2 run 1 - epoch 430/700 --- Train cost:0.02258 --- Test cost:0.02925
0.3-0.2 run 1 - epoch 435/700 --- Train cost:0.02556 --- Test cost:0.02682
0.3-0.2 run 1 - epoch 440/700 --- Train cost:0.02780 --- Test cost:0.02608
0.3-0.2 run 1 - epoch 445/700 --- Train cost:0.02890 --- Test cost:0.02605
0.3-0.2 run 1 - epoch 450/700 --- Train cost:0.02778 --- Test cost:0.02578
0.3-0.2 run 1 - epoch 455/700 --- Train cost:0.02589 --- Test cost:0.02465
0.3-0.2 run 1 - epoch 460/700 --- Train cost:0.02313 --- Test cost:0.02453
0.3-0.2 run 1 - epoch 465/700 --- Train cost:0.02230 --- Test cost:0.02470
0.3-0.2 run 1 - epoch 470/700 --- Train cost:0.02221 --- Test cost:0.02607
0.3-0.2 run 1 - epoch 475/700 --- Train cost:0.02244 --- Test cost:0.02747
0.3-0.2 run 1 - epoch 480/700 --- Train cost:0.02321 --- Test cost:0.02905
0.3-0.2 run 1 - epoch 485/700 --- Train cost:0.02427 --- Test cost:0.03002
0.3-0.2 run 1 - epoch 490/700 --- Train cost:0.02487 --- Test cost:0.03213
0.3-0.2 run 1 - epoch 495/700 --- Train cost:0.02497 --- Test cost:0.03094
0.3-0.2 run 1 - epoch 500/700 --- Train cost:0.02365 --- Test cost:0.03063
0.3-0.2 run 1 - epoch 505/700 --- Train cost:0.02153 --- Test cost:0.02974
0.3-0.2 run 1 - epoch 510/700 --- Train cost:0.02013 --- Test cost:0.02918
0.3-0.2 run 1 - epoch 515/700 --- Train cost:0.01988 --- Test cost:0.02849
0.3-0.2 run 1 - epoch 520/700 --- Train cost:0.02023 --- Test cost:0.02809
0.3-0.2 run 1 - epoch 525/700 --- Train cost:0.02155 --- Test cost:0.02827
0.3-0.2 run 1 - epoch 530/700 --- Train cost:0.02203 --- Test cost:0.02946
0.3-0.2 run 1 - epoch 535/700 --- Train cost:0.02275 --- Test cost:0.02949
0.3-0.2 run 1 - epoch 540/700 --- Train cost:0.02294 --- Test cost:0.02962
0.3-0.2 run 1 - epoch 545/700 --- Train cost:0.02214 --- Test cost:0.02953
0.3-0.2 run 1 - epoch 550/700 --- Train cost:0.01995 --- Test cost:0.03036
0.3-0.2 run 1 - epoch 555/700 --- Train cost:0.01818 --- Test cost:0.03044
0.3-0.2 run 1 - epoch 560/700 --- Train cost:0.01787 --- Test cost:0.02972
0.3-0.2 run 1 - epoch 565/700 --- Train cost:0.01949 --- Test cost:0.03088
0.3-0.2 run 1 - epoch 570/700 --- Train cost:0.01991 --- Test cost:0.03128
0.3-0.2 run 1 - epoch 575/700 --- Train cost:0.02000 --- Test cost:0.03229
0.3-0.2 run 1 - epoch 580/700 --- Train cost:0.02012 --- Test cost:0.03475
0.3-0.2 run 1 - epoch 585/700 --- Train cost:0.02029 --- Test cost:0.03853
0.3-0.2 run 1 - epoch 590/700 --- Train cost:0.02044 --- Test cost:0.04113
0.3-0.2 run 1 - epoch 595/700 --- Train cost:0.02109 --- Test cost:0.04275
0.3-0.2 run 1 - epoch 600/700 --- Train cost:0.02163 --- Test cost:0.04315
0.3-0.2 run 1 - epoch 605/700 --- Train cost:0.02199 --- Test cost:0.04196
0.3-0.2 run 1 - epoch 610/700 --- Train cost:0.02327 --- Test cost:0.03672
0.3-0.2 run 1 - epoch 615/700 --- Train cost:0.02422 --- Test cost:0.03375
0.3-0.2 run 1 - epoch 620/700 --- Train cost:0.02461 --- Test cost:0.03259
0.3-0.2 run 1 - epoch 625/700 --- Train cost:0.02398 --- Test cost:0.03313
0.3-0.2 run 1 - epoch 630/700 --- Train cost:0.02345 --- Test cost:0.03497
0.3-0.2 run 1 - epoch 635/700 --- Train cost:0.02421 --- Test cost:0.03511
0.3-0.2 run 1 - epoch 640/700 --- Train cost:0.02565 --- Test cost:0.03593
0.3-0.2 run 1 - epoch 645/700 --- Train cost:0.02613 --- Test cost:0.03628
0.3-0.2 run 1 - epoch 650/700 --- Train cost:0.02579 --- Test cost:0.03844
0.3-0.2 run 1 - epoch 655/700 --- Train cost:0.02616 --- Test cost:0.04036
0.3-0.2 run 1 - epoch 660/700 --- Train cost:0.02684 --- Test cost:0.03859
0.3-0.2 run 1 - epoch 665/700 --- Train cost:0.02739 --- Test cost:0.03761
0.3-0.2 run 1 - epoch 670/700 --- Train cost:0.02756 --- Test cost:0.03633
0.3-0.2 run 1 - epoch 675/700 --- Train cost:0.02639 --- Test cost:0.03506
0.3-0.2 run 1 - epoch 680/700 --- Train cost:0.02516 --- Test cost:0.03229
0.3-0.2 run 1 - epoch 685/700 --- Train cost:0.02443 --- Test cost:0.02950
0.3-0.2 run 1 - epoch 690/700 --- Train cost:0.02328 --- Test cost:0.02673
0.3-0.2 run 1 - epoch 695/700 --- Train cost:0.02258 --- Test cost:0.02543
0.3-0.2 run 2 - epoch 0/700 --- Train cost:0.34278 --- Test cost:0.15919
0.3-0.2 run 2 - epoch 5/700 --- Train cost:0.28132 --- Test cost:0.13218
0.3-0.2 run 2 - epoch 10/700 --- Train cost:0.23132 --- Test cost:0.11190
0.3-0.2 run 2 - epoch 15/700 --- Train cost:0.20663 --- Test cost:0.10601
0.3-0.2 run 2 - epoch 20/700 --- Train cost:0.17510 --- Test cost:0.09089
0.3-0.2 run 2 - epoch 25/700 --- Train cost:0.14803 --- Test cost:0.07539
0.3-0.2 run 2 - epoch 30/700 --- Train cost:0.13183 --- Test cost:0.06781
0.3-0.2 run 2 - epoch 35/700 --- Train cost:0.12741 --- Test cost:0.06498
0.3-0.2 run 2 - epoch 40/700 --- Train cost:0.13464 --- Test cost:0.07032
0.3-0.2 run 2 - epoch 45/700 --- Train cost:0.13521 --- Test cost:0.07230
0.3-0.2 run 2 - epoch 50/700 --- Train cost:0.11684 --- Test cost:0.06300
0.3-0.2 run 2 - epoch 55/700 --- Train cost:0.10356 --- Test cost:0.05661
0.3-0.2 run 2 - epoch 60/700 --- Train cost:0.10000 --- Test cost:0.05824
0.3-0.2 run 2 - epoch 65/700 --- Train cost:0.09885 --- Test cost:0.06258
0.3-0.2 run 2 - epoch 70/700 --- Train cost:0.09525 --- Test cost:0.06176
0.3-0.2 run 2 - epoch 75/700 --- Train cost:0.08257 --- Test cost:0.05414
0.3-0.2 run 2 - epoch 80/700 --- Train cost:0.06751 --- Test cost:0.04591
0.3-0.2 run 2 - epoch 85/700 --- Train cost:0.06228 --- Test cost:0.04132
0.3-0.2 run 2 - epoch 90/700 --- Train cost:0.06065 --- Test cost:0.04068
0.3-0.2 run 2 - epoch 95/700 --- Train cost:0.06269 --- Test cost:0.04089
0.3-0.2 run 2 - epoch 100/700 --- Train cost:0.06198 --- Test cost:0.03901
0.3-0.2 run 2 - epoch 105/700 --- Train cost:0.05928 --- Test cost:0.03481
0.3-0.2 run 2 - epoch 110/700 --- Train cost:0.05577 --- Test cost:0.03087
0.3-0.2 run 2 - epoch 115/700 --- Train cost:0.05254 --- Test cost:0.02750
0.3-0.2 run 2 - epoch 120/700 --- Train cost:0.04757 --- Test cost:0.02674
0.3-0.2 run 2 - epoch 125/700 --- Train cost:0.04103 --- Test cost:0.02500
0.3-0.2 run 2 - epoch 130/700 --- Train cost:0.03974 --- Test cost:0.02562
0.3-0.2 run 2 - epoch 135/700 --- Train cost:0.03892 --- Test cost:0.02729
0.3-0.2 run 2 - epoch 140/700 --- Train cost:0.03626 --- Test cost:0.02841
0.3-0.2 run 2 - epoch 145/700 --- Train cost:0.03162 --- Test cost:0.02885
0.3-0.2 run 2 - epoch 150/700 --- Train cost:0.02799 --- Test cost:0.02885
0.3-0.2 run 2 - epoch 155/700 --- Train cost:0.02635 --- Test cost:0.02905
0.3-0.2 run 2 - epoch 160/700 --- Train cost:0.02718 --- Test cost:0.02897
0.3-0.2 run 2 - epoch 165/700 --- Train cost:0.02886 --- Test cost:0.02912
0.3-0.2 run 2 - epoch 170/700 --- Train cost:0.02706 --- Test cost:0.03066
0.3-0.2 run 2 - epoch 175/700 --- Train cost:0.02518 --- Test cost:0.03333
0.3-0.2 run 2 - epoch 180/700 --- Train cost:0.02473 --- Test cost:0.03470
0.3-0.2 run 2 - epoch 185/700 --- Train cost:0.02495 --- Test cost:0.03578
0.3-0.2 run 2 - epoch 190/700 --- Train cost:0.02559 --- Test cost:0.03443
0.3-0.2 run 2 - epoch 195/700 --- Train cost:0.02744 --- Test cost:0.03276
0.3-0.2 run 2 - epoch 200/700 --- Train cost:0.02844 --- Test cost:0.03364
0.3-0.2 run 2 - epoch 205/700 --- Train cost:0.02869 --- Test cost:0.03450
0.3-0.2 run 2 - epoch 210/700 --- Train cost:0.02955 --- Test cost:0.03535
0.3-0.2 run 2 - epoch 215/700 --- Train cost:0.03238 --- Test cost:0.03724
0.3-0.2 run 2 - epoch 220/700 --- Train cost:0.03412 --- Test cost:0.03921
0.3-0.2 run 2 - epoch 225/700 --- Train cost:0.03353 --- Test cost:0.03855
0.3-0.2 run 2 - epoch 230/700 --- Train cost:0.03182 --- Test cost:0.03540
0.3-0.2 run 2 - epoch 235/700 --- Train cost:0.03060 --- Test cost:0.03439
0.3-0.2 run 2 - epoch 240/700 --- Train cost:0.03054 --- Test cost:0.03452
0.3-0.2 run 2 - epoch 245/700 --- Train cost:0.03235 --- Test cost:0.03604
0.3-0.2 run 2 - epoch 250/700 --- Train cost:0.03219 --- Test cost:0.03593
0.3-0.2 run 2 - epoch 255/700 --- Train cost:0.03195 --- Test cost:0.03635
0.3-0.2 run 2 - epoch 260/700 --- Train cost:0.03218 --- Test cost:0.03722
0.3-0.2 run 2 - epoch 265/700 --- Train cost:0.03181 --- Test cost:0.03760
0.3-0.2 run 2 - epoch 270/700 --- Train cost:0.03112 --- Test cost:0.03641
0.3-0.2 run 2 - epoch 275/700 --- Train cost:0.03101 --- Test cost:0.03436
0.3-0.2 run 2 - epoch 280/700 --- Train cost:0.03019 --- Test cost:0.03358
0.3-0.2 run 2 - epoch 285/700 --- Train cost:0.02781 --- Test cost:0.03371
0.3-0.2 run 2 - epoch 290/700 --- Train cost:0.02519 --- Test cost:0.03568
0.3-0.2 run 2 - epoch 295/700 --- Train cost:0.02357 --- Test cost:0.03853
0.3-0.2 run 2 - epoch 300/700 --- Train cost:0.02374 --- Test cost:0.03838
0.3-0.2 run 2 - epoch 305/700 --- Train cost:0.02319 --- Test cost:0.03819
0.3-0.2 run 2 - epoch 310/700 --- Train cost:0.02267 --- Test cost:0.03689
0.3-0.2 run 2 - epoch 315/700 --- Train cost:0.02285 --- Test cost:0.03611
0.3-0.2 run 2 - epoch 320/700 --- Train cost:0.02284 --- Test cost:0.03616
0.3-0.2 run 2 - epoch 325/700 --- Train cost:0.02083 --- Test cost:0.03639
0.3-0.2 run 2 - epoch 330/700 --- Train cost:0.01859 --- Test cost:0.03534
0.3-0.2 run 2 - epoch 335/700 --- Train cost:0.01835 --- Test cost:0.03335
0.3-0.2 run 2 - epoch 340/700 --- Train cost:0.01874 --- Test cost:0.03133
0.3-0.2 run 2 - epoch 345/700 --- Train cost:0.01900 --- Test cost:0.02970
0.3-0.2 run 2 - epoch 350/700 --- Train cost:0.01883 --- Test cost:0.02837
0.3-0.2 run 2 - epoch 355/700 --- Train cost:0.01890 --- Test cost:0.02819
0.3-0.2 run 2 - epoch 360/700 --- Train cost:0.01858 --- Test cost:0.02929
0.3-0.2 run 2 - epoch 365/700 --- Train cost:0.01810 --- Test cost:0.03308
0.3-0.2 run 2 - epoch 370/700 --- Train cost:0.01799 --- Test cost:0.03617
0.3-0.2 run 2 - epoch 375/700 --- Train cost:0.01923 --- Test cost:0.03711
0.3-0.2 run 2 - epoch 380/700 --- Train cost:0.01992 --- Test cost:0.04044
0.3-0.2 run 2 - epoch 385/700 --- Train cost:0.01997 --- Test cost:0.04418
0.3-0.2 run 2 - epoch 390/700 --- Train cost:0.02058 --- Test cost:0.04511
0.3-0.2 run 2 - epoch 395/700 --- Train cost:0.02072 --- Test cost:0.04551
0.3-0.2 run 2 - epoch 400/700 --- Train cost:0.01957 --- Test cost:0.04624
0.3-0.2 run 2 - epoch 405/700 --- Train cost:0.01749 --- Test cost:0.04656
0.3-0.2 run 2 - epoch 410/700 --- Train cost:0.01553 --- Test cost:0.04521
0.3-0.2 run 2 - epoch 415/700 --- Train cost:0.01437 --- Test cost:0.04518
0.3-0.2 run 2 - epoch 420/700 --- Train cost:0.01342 --- Test cost:0.04605
0.3-0.2 run 2 - epoch 425/700 --- Train cost:0.01295 --- Test cost:0.04637
0.3-0.2 run 2 - epoch 430/700 --- Train cost:0.01348 --- Test cost:0.04360
0.3-0.2 run 2 - epoch 435/700 --- Train cost:0.01388 --- Test cost:0.03718
0.3-0.2 run 2 - epoch 440/700 --- Train cost:0.01462 --- Test cost:0.03346
0.3-0.2 run 2 - epoch 445/700 --- Train cost:0.01739 --- Test cost:0.03054
0.3-0.2 run 2 - epoch 450/700 --- Train cost:0.01838 --- Test cost:0.03134
0.3-0.2 run 2 - epoch 455/700 --- Train cost:0.02008 --- Test cost:0.03287
0.3-0.2 run 2 - epoch 460/700 --- Train cost:0.02103 --- Test cost:0.03486
0.3-0.2 run 2 - epoch 465/700 --- Train cost:0.02161 --- Test cost:0.03653
0.3-0.2 run 2 - epoch 470/700 --- Train cost:0.02231 --- Test cost:0.03592
0.3-0.2 run 2 - epoch 475/700 --- Train cost:0.02068 --- Test cost:0.03673
0.3-0.2 run 2 - epoch 480/700 --- Train cost:0.01813 --- Test cost:0.03697
0.3-0.2 run 2 - epoch 485/700 --- Train cost:0.01654 --- Test cost:0.03794
0.3-0.2 run 2 - epoch 490/700 --- Train cost:0.01731 --- Test cost:0.03931
0.3-0.2 run 2 - epoch 495/700 --- Train cost:0.01767 --- Test cost:0.03878
0.3-0.2 run 2 - epoch 500/700 --- Train cost:0.01837 --- Test cost:0.04062
0.3-0.2 run 2 - epoch 505/700 --- Train cost:0.02030 --- Test cost:0.04385
0.3-0.2 run 2 - epoch 510/700 --- Train cost:0.02215 --- Test cost:0.04440
0.3-0.2 run 2 - epoch 515/700 --- Train cost:0.02235 --- Test cost:0.04182
0.3-0.2 run 2 - epoch 520/700 --- Train cost:0.02186 --- Test cost:0.03839
0.3-0.2 run 2 - epoch 525/700 --- Train cost:0.02151 --- Test cost:0.03492
0.3-0.2 run 2 - epoch 530/700 --- Train cost:0.02122 --- Test cost:0.03194
0.3-0.2 run 2 - epoch 535/700 --- Train cost:0.02084 --- Test cost:0.03025
0.3-0.2 run 2 - epoch 540/700 --- Train cost:0.02012 --- Test cost:0.02985
0.3-0.2 run 2 - epoch 545/700 --- Train cost:0.02005 --- Test cost:0.03087
0.3-0.2 run 2 - epoch 550/700 --- Train cost:0.01955 --- Test cost:0.03097
0.3-0.2 run 2 - epoch 555/700 --- Train cost:0.01863 --- Test cost:0.03180
0.3-0.2 run 2 - epoch 560/700 --- Train cost:0.01808 --- Test cost:0.03208
0.3-0.2 run 2 - epoch 565/700 --- Train cost:0.01784 --- Test cost:0.03127
0.3-0.2 run 2 - epoch 570/700 --- Train cost:0.01751 --- Test cost:0.02968
0.3-0.2 run 2 - epoch 575/700 --- Train cost:0.01743 --- Test cost:0.02819
0.3-0.2 run 2 - epoch 580/700 --- Train cost:0.01767 --- Test cost:0.02914
0.3-0.2 run 2 - epoch 585/700 --- Train cost:0.01847 --- Test cost:0.03020
0.3-0.2 run 2 - epoch 590/700 --- Train cost:0.01937 --- Test cost:0.03271
0.3-0.2 run 2 - epoch 595/700 --- Train cost:0.01993 --- Test cost:0.03561
0.3-0.2 run 2 - epoch 600/700 --- Train cost:0.02050 --- Test cost:0.03822
0.3-0.2 run 2 - epoch 605/700 --- Train cost:0.02168 --- Test cost:0.03987
0.3-0.2 run 2 - epoch 610/700 --- Train cost:0.02587 --- Test cost:0.03998
0.3-0.2 run 2 - epoch 615/700 --- Train cost:0.02779 --- Test cost:0.04133
0.3-0.2 run 2 - epoch 620/700 --- Train cost:0.02519 --- Test cost:0.04356
0.3-0.2 run 2 - epoch 625/700 --- Train cost:0.02274 --- Test cost:0.04118
0.3-0.2 run 2 - epoch 630/700 --- Train cost:0.02200 --- Test cost:0.04114
0.3-0.2 run 2 - epoch 635/700 --- Train cost:0.02261 --- Test cost:0.04003
0.3-0.2 run 2 - epoch 640/700 --- Train cost:0.02230 --- Test cost:0.03769
0.3-0.2 run 2 - epoch 645/700 --- Train cost:0.02251 --- Test cost:0.03571
0.3-0.2 run 2 - epoch 650/700 --- Train cost:0.02192 --- Test cost:0.03764
0.3-0.2 run 2 - epoch 655/700 --- Train cost:0.02261 --- Test cost:0.03894
0.3-0.2 run 2 - epoch 660/700 --- Train cost:0.02199 --- Test cost:0.03986
0.3-0.2 run 2 - epoch 665/700 --- Train cost:0.02149 --- Test cost:0.04118
0.3-0.2 run 2 - epoch 670/700 --- Train cost:0.02190 --- Test cost:0.04345
0.3-0.2 run 2 - epoch 675/700 --- Train cost:0.02264 --- Test cost:0.04714
0.3-0.2 run 2 - epoch 680/700 --- Train cost:0.02259 --- Test cost:0.05051
0.3-0.2 run 2 - epoch 685/700 --- Train cost:0.02147 --- Test cost:0.05481
0.3-0.2 run 2 - epoch 690/700 --- Train cost:0.01968 --- Test cost:0.05929
0.3-0.2 run 2 - epoch 695/700 --- Train cost:0.01821 --- Test cost:0.06038
0.7-0.7 run 0 - epoch 0/700 --- Train cost:0.48694 --- Test cost:0.18159
0.7-0.7 run 0 - epoch 5/700 --- Train cost:0.45133 --- Test cost:0.18601
0.7-0.7 run 0 - epoch 10/700 --- Train cost:0.44266 --- Test cost:0.18347
0.7-0.7 run 0 - epoch 15/700 --- Train cost:0.43543 --- Test cost:0.18362
0.7-0.7 run 0 - epoch 20/700 --- Train cost:0.42925 --- Test cost:0.18564
0.7-0.7 run 0 - epoch 25/700 --- Train cost:0.42731 --- Test cost:0.19032
0.7-0.7 run 0 - epoch 30/700 --- Train cost:0.42100 --- Test cost:0.19420
0.7-0.7 run 0 - epoch 35/700 --- Train cost:0.41877 --- Test cost:0.19420
0.7-0.7 run 0 - epoch 40/700 --- Train cost:0.41817 --- Test cost:0.20158
0.7-0.7 run 0 - epoch 45/700 --- Train cost:0.42655 --- Test cost:0.21943
0.7-0.7 run 0 - epoch 50/700 --- Train cost:0.43367 --- Test cost:0.23445
0.7-0.7 run 0 - epoch 55/700 --- Train cost:0.44196 --- Test cost:0.24297
0.7-0.7 run 0 - epoch 60/700 --- Train cost:0.44581 --- Test cost:0.24633
0.7-0.7 run 0 - epoch 65/700 --- Train cost:0.44184 --- Test cost:0.24406
0.7-0.7 run 0 - epoch 70/700 --- Train cost:0.44265 --- Test cost:0.24380
0.7-0.7 run 0 - epoch 75/700 --- Train cost:0.44070 --- Test cost:0.24793
0.7-0.7 run 0 - epoch 80/700 --- Train cost:0.44367 --- Test cost:0.25030
0.7-0.7 run 0 - epoch 85/700 --- Train cost:0.45098 --- Test cost:0.25671
0.7-0.7 run 0 - epoch 90/700 --- Train cost:0.46509 --- Test cost:0.27224
0.7-0.7 run 0 - epoch 95/700 --- Train cost:0.46767 --- Test cost:0.27762
0.7-0.7 run 0 - epoch 100/700 --- Train cost:0.46680 --- Test cost:0.27358
0.7-0.7 run 0 - epoch 105/700 --- Train cost:0.46435 --- Test cost:0.26891
0.7-0.7 run 0 - epoch 110/700 --- Train cost:0.46345 --- Test cost:0.27216
0.7-0.7 run 0 - epoch 115/700 --- Train cost:0.46815 --- Test cost:0.27692
0.7-0.7 run 0 - epoch 120/700 --- Train cost:0.46472 --- Test cost:0.26916
0.7-0.7 run 0 - epoch 125/700 --- Train cost:0.45106 --- Test cost:0.26434
0.7-0.7 run 0 - epoch 130/700 --- Train cost:0.44020 --- Test cost:0.26362
0.7-0.7 run 0 - epoch 135/700 --- Train cost:0.43706 --- Test cost:0.26030
0.7-0.7 run 0 - epoch 140/700 --- Train cost:0.43919 --- Test cost:0.26062
0.7-0.7 run 0 - epoch 145/700 --- Train cost:0.43811 --- Test cost:0.25650
0.7-0.7 run 0 - epoch 150/700 --- Train cost:0.43700 --- Test cost:0.26488
0.7-0.7 run 0 - epoch 155/700 --- Train cost:0.43901 --- Test cost:0.27257
0.7-0.7 run 0 - epoch 160/700 --- Train cost:0.44622 --- Test cost:0.28039
0.7-0.7 run 0 - epoch 165/700 --- Train cost:0.45288 --- Test cost:0.29006
0.7-0.7 run 0 - epoch 170/700 --- Train cost:0.45550 --- Test cost:0.29721
0.7-0.7 run 0 - epoch 175/700 --- Train cost:0.46679 --- Test cost:0.30513
0.7-0.7 run 0 - epoch 180/700 --- Train cost:0.47493 --- Test cost:0.31189
0.7-0.7 run 0 - epoch 185/700 --- Train cost:0.47523 --- Test cost:0.31563
0.7-0.7 run 0 - epoch 190/700 --- Train cost:0.48710 --- Test cost:0.32458
0.7-0.7 run 0 - epoch 195/700 --- Train cost:0.49693 --- Test cost:0.33346
0.7-0.7 run 0 - epoch 200/700 --- Train cost:0.49587 --- Test cost:0.32419
0.7-0.7 run 0 - epoch 205/700 --- Train cost:0.49408 --- Test cost:0.31387
0.7-0.7 run 0 - epoch 210/700 --- Train cost:0.49738 --- Test cost:0.31817
0.7-0.7 run 0 - epoch 215/700 --- Train cost:0.48917 --- Test cost:0.31511
0.7-0.7 run 0 - epoch 220/700 --- Train cost:0.47885 --- Test cost:0.30822
0.7-0.7 run 0 - epoch 225/700 --- Train cost:0.46831 --- Test cost:0.30017
0.7-0.7 run 0 - epoch 230/700 --- Train cost:0.46747 --- Test cost:0.29509
0.7-0.7 run 0 - epoch 235/700 --- Train cost:0.46623 --- Test cost:0.28736
0.7-0.7 run 0 - epoch 240/700 --- Train cost:0.46360 --- Test cost:0.27791
0.7-0.7 run 0 - epoch 245/700 --- Train cost:0.46114 --- Test cost:0.26730
0.7-0.7 run 0 - epoch 250/700 --- Train cost:0.45611 --- Test cost:0.25630
0.7-0.7 run 0 - epoch 255/700 --- Train cost:0.45037 --- Test cost:0.25168
0.7-0.7 run 0 - epoch 260/700 --- Train cost:0.44469 --- Test cost:0.25399
0.7-0.7 run 0 - epoch 265/700 --- Train cost:0.44792 --- Test cost:0.26350
0.7-0.7 run 0 - epoch 270/700 --- Train cost:0.44430 --- Test cost:0.26503
0.7-0.7 run 0 - epoch 275/700 --- Train cost:0.44347 --- Test cost:0.26522
0.7-0.7 run 0 - epoch 280/700 --- Train cost:0.44104 --- Test cost:0.25622
0.7-0.7 run 0 - epoch 285/700 --- Train cost:0.42769 --- Test cost:0.24000
0.7-0.7 run 0 - epoch 290/700 --- Train cost:0.40354 --- Test cost:0.22049
0.7-0.7 run 0 - epoch 295/700 --- Train cost:0.38719 --- Test cost:0.20794
0.7-0.7 run 0 - epoch 300/700 --- Train cost:0.37759 --- Test cost:0.20069
0.7-0.7 run 0 - epoch 305/700 --- Train cost:0.37247 --- Test cost:0.19699
0.7-0.7 run 0 - epoch 310/700 --- Train cost:0.37298 --- Test cost:0.19319
0.7-0.7 run 0 - epoch 315/700 --- Train cost:0.37427 --- Test cost:0.19019
0.7-0.7 run 0 - epoch 320/700 --- Train cost:0.38292 --- Test cost:0.19009
0.7-0.7 run 0 - epoch 325/700 --- Train cost:0.37765 --- Test cost:0.18081
0.7-0.7 run 0 - epoch 330/700 --- Train cost:0.37783 --- Test cost:0.17595
0.7-0.7 run 0 - epoch 335/700 --- Train cost:0.37735 --- Test cost:0.17224
0.7-0.7 run 0 - epoch 340/700 --- Train cost:0.37579 --- Test cost:0.16966
0.7-0.7 run 0 - epoch 345/700 --- Train cost:0.37607 --- Test cost:0.16701
0.7-0.7 run 0 - epoch 350/700 --- Train cost:0.37827 --- Test cost:0.16981
0.7-0.7 run 0 - epoch 355/700 --- Train cost:0.38056 --- Test cost:0.17700
0.7-0.7 run 0 - epoch 360/700 --- Train cost:0.38604 --- Test cost:0.18457
0.7-0.7 run 0 - epoch 365/700 --- Train cost:0.39772 --- Test cost:0.19238
0.7-0.7 run 0 - epoch 370/700 --- Train cost:0.40266 --- Test cost:0.20014
0.7-0.7 run 0 - epoch 375/700 --- Train cost:0.40168 --- Test cost:0.20156
0.7-0.7 run 0 - epoch 380/700 --- Train cost:0.41055 --- Test cost:0.20561
0.7-0.7 run 0 - epoch 385/700 --- Train cost:0.40819 --- Test cost:0.20230
0.7-0.7 run 0 - epoch 390/700 --- Train cost:0.40825 --- Test cost:0.20408
0.7-0.7 run 0 - epoch 395/700 --- Train cost:0.40333 --- Test cost:0.20006
0.7-0.7 run 0 - epoch 400/700 --- Train cost:0.39877 --- Test cost:0.20025
0.7-0.7 run 0 - epoch 405/700 --- Train cost:0.40695 --- Test cost:0.20493
0.7-0.7 run 0 - epoch 410/700 --- Train cost:0.41314 --- Test cost:0.20398
0.7-0.7 run 0 - epoch 415/700 --- Train cost:0.40823 --- Test cost:0.19830
0.7-0.7 run 0 - epoch 420/700 --- Train cost:0.39964 --- Test cost:0.18839
0.7-0.7 run 0 - epoch 425/700 --- Train cost:0.39551 --- Test cost:0.17568
0.7-0.7 run 0 - epoch 430/700 --- Train cost:0.39482 --- Test cost:0.17051
0.7-0.7 run 0 - epoch 435/700 --- Train cost:0.39493 --- Test cost:0.17337
0.7-0.7 run 0 - epoch 440/700 --- Train cost:0.38720 --- Test cost:0.16616
0.7-0.7 run 0 - epoch 445/700 --- Train cost:0.37722 --- Test cost:0.15667
0.7-0.7 run 0 - epoch 450/700 --- Train cost:0.37811 --- Test cost:0.15306
0.7-0.7 run 0 - epoch 455/700 --- Train cost:0.38048 --- Test cost:0.15148
0.7-0.7 run 0 - epoch 460/700 --- Train cost:0.38578 --- Test cost:0.15344
0.7-0.7 run 0 - epoch 465/700 --- Train cost:0.38161 --- Test cost:0.15030
0.7-0.7 run 0 - epoch 470/700 --- Train cost:0.37868 --- Test cost:0.14834
0.7-0.7 run 0 - epoch 475/700 --- Train cost:0.38603 --- Test cost:0.14733
0.7-0.7 run 0 - epoch 480/700 --- Train cost:0.38871 --- Test cost:0.14412
0.7-0.7 run 0 - epoch 485/700 --- Train cost:0.39302 --- Test cost:0.14307
0.7-0.7 run 0 - epoch 490/700 --- Train cost:0.39034 --- Test cost:0.14161
0.7-0.7 run 0 - epoch 495/700 --- Train cost:0.38373 --- Test cost:0.13809
0.7-0.7 run 0 - epoch 500/700 --- Train cost:0.38406 --- Test cost:0.13817
0.7-0.7 run 0 - epoch 505/700 --- Train cost:0.38498 --- Test cost:0.14435
0.7-0.7 run 0 - epoch 510/700 --- Train cost:0.37525 --- Test cost:0.14556
0.7-0.7 run 0 - epoch 515/700 --- Train cost:0.36514 --- Test cost:0.14936
0.7-0.7 run 0 - epoch 520/700 --- Train cost:0.35913 --- Test cost:0.15554
0.7-0.7 run 0 - epoch 525/700 --- Train cost:0.36207 --- Test cost:0.17047
0.7-0.7 run 0 - epoch 530/700 --- Train cost:0.36535 --- Test cost:0.17816
0.7-0.7 run 0 - epoch 535/700 --- Train cost:0.36539 --- Test cost:0.18307
0.7-0.7 run 0 - epoch 540/700 --- Train cost:0.36174 --- Test cost:0.18684
0.7-0.7 run 0 - epoch 545/700 --- Train cost:0.36346 --- Test cost:0.19253
0.7-0.7 run 0 - epoch 550/700 --- Train cost:0.36878 --- Test cost:0.19725
0.7-0.7 run 0 - epoch 555/700 --- Train cost:0.37222 --- Test cost:0.20353
0.7-0.7 run 0 - epoch 560/700 --- Train cost:0.37081 --- Test cost:0.20474
0.7-0.7 run 0 - epoch 565/700 --- Train cost:0.35958 --- Test cost:0.20530
0.7-0.7 run 0 - epoch 570/700 --- Train cost:0.34750 --- Test cost:0.20396
0.7-0.7 run 0 - epoch 575/700 --- Train cost:0.34858 --- Test cost:0.20859
0.7-0.7 run 0 - epoch 580/700 --- Train cost:0.35691 --- Test cost:0.20612
0.7-0.7 run 0 - epoch 585/700 --- Train cost:0.36982 --- Test cost:0.20662
0.7-0.7 run 0 - epoch 590/700 --- Train cost:0.38588 --- Test cost:0.21320
0.7-0.7 run 0 - epoch 595/700 --- Train cost:0.39234 --- Test cost:0.20825
0.7-0.7 run 0 - epoch 600/700 --- Train cost:0.39357 --- Test cost:0.20547
0.7-0.7 run 0 - epoch 605/700 --- Train cost:0.39525 --- Test cost:0.19908
0.7-0.7 run 0 - epoch 610/700 --- Train cost:0.40385 --- Test cost:0.19959
0.7-0.7 run 0 - epoch 615/700 --- Train cost:0.41621 --- Test cost:0.20243
0.7-0.7 run 0 - epoch 620/700 --- Train cost:0.43105 --- Test cost:0.21817
0.7-0.7 run 0 - epoch 625/700 --- Train cost:0.44099 --- Test cost:0.22861
0.7-0.7 run 0 - epoch 630/700 --- Train cost:0.45350 --- Test cost:0.23422
0.7-0.7 run 0 - epoch 635/700 --- Train cost:0.45594 --- Test cost:0.23467
0.7-0.7 run 0 - epoch 640/700 --- Train cost:0.45833 --- Test cost:0.23053
0.7-0.7 run 0 - epoch 645/700 --- Train cost:0.45528 --- Test cost:0.22126
0.7-0.7 run 0 - epoch 650/700 --- Train cost:0.45358 --- Test cost:0.21023
0.7-0.7 run 0 - epoch 655/700 --- Train cost:0.44874 --- Test cost:0.19694
0.7-0.7 run 0 - epoch 660/700 --- Train cost:0.44231 --- Test cost:0.18700
0.7-0.7 run 0 - epoch 665/700 --- Train cost:0.43445 --- Test cost:0.17822
0.7-0.7 run 0 - epoch 670/700 --- Train cost:0.42189 --- Test cost:0.17231
0.7-0.7 run 0 - epoch 675/700 --- Train cost:0.40009 --- Test cost:0.15920
0.7-0.7 run 0 - epoch 680/700 --- Train cost:0.39000 --- Test cost:0.15155
0.7-0.7 run 0 - epoch 685/700 --- Train cost:0.39403 --- Test cost:0.14920
0.7-0.7 run 0 - epoch 690/700 --- Train cost:0.39859 --- Test cost:0.14477
0.7-0.7 run 0 - epoch 695/700 --- Train cost:0.39514 --- Test cost:0.13717
0.7-0.7 run 1 - epoch 0/700 --- Train cost:0.29358 --- Test cost:0.10004
0.7-0.7 run 1 - epoch 5/700 --- Train cost:0.29562 --- Test cost:0.08927
0.7-0.7 run 1 - epoch 10/700 --- Train cost:0.29830 --- Test cost:0.09813
0.7-0.7 run 1 - epoch 15/700 --- Train cost:0.30584 --- Test cost:0.11427
0.7-0.7 run 1 - epoch 20/700 --- Train cost:0.30768 --- Test cost:0.12313
0.7-0.7 run 1 - epoch 25/700 --- Train cost:0.30571 --- Test cost:0.13305
0.7-0.7 run 1 - epoch 30/700 --- Train cost:0.30763 --- Test cost:0.14353
0.7-0.7 run 1 - epoch 35/700 --- Train cost:0.31223 --- Test cost:0.15247
0.7-0.7 run 1 - epoch 40/700 --- Train cost:0.31444 --- Test cost:0.16273
0.7-0.7 run 1 - epoch 45/700 --- Train cost:0.31952 --- Test cost:0.17322
0.7-0.7 run 1 - epoch 50/700 --- Train cost:0.31981 --- Test cost:0.18617
0.7-0.7 run 1 - epoch 55/700 --- Train cost:0.31634 --- Test cost:0.19106
0.7-0.7 run 1 - epoch 60/700 --- Train cost:0.31258 --- Test cost:0.19226
0.7-0.7 run 1 - epoch 65/700 --- Train cost:0.30884 --- Test cost:0.19185
0.7-0.7 run 1 - epoch 70/700 --- Train cost:0.30403 --- Test cost:0.19113
0.7-0.7 run 1 - epoch 75/700 --- Train cost:0.29350 --- Test cost:0.18588
0.7-0.7 run 1 - epoch 80/700 --- Train cost:0.28498 --- Test cost:0.17491
0.7-0.7 run 1 - epoch 85/700 --- Train cost:0.27136 --- Test cost:0.16793
0.7-0.7 run 1 - epoch 90/700 --- Train cost:0.26303 --- Test cost:0.15911
0.7-0.7 run 1 - epoch 95/700 --- Train cost:0.25530 --- Test cost:0.15196
0.7-0.7 run 1 - epoch 100/700 --- Train cost:0.25767 --- Test cost:0.15182
0.7-0.7 run 1 - epoch 105/700 --- Train cost:0.26665 --- Test cost:0.15105
0.7-0.7 run 1 - epoch 110/700 --- Train cost:0.27480 --- Test cost:0.14792
0.7-0.7 run 1 - epoch 115/700 --- Train cost:0.28630 --- Test cost:0.14845
0.7-0.7 run 1 - epoch 120/700 --- Train cost:0.30122 --- Test cost:0.14967
0.7-0.7 run 1 - epoch 125/700 --- Train cost:0.31297 --- Test cost:0.15004
0.7-0.7 run 1 - epoch 130/700 --- Train cost:0.32264 --- Test cost:0.15090
0.7-0.7 run 1 - epoch 135/700 --- Train cost:0.32352 --- Test cost:0.14529
0.7-0.7 run 1 - epoch 140/700 --- Train cost:0.32180 --- Test cost:0.14048
0.7-0.7 run 1 - epoch 145/700 --- Train cost:0.32766 --- Test cost:0.13811
0.7-0.7 run 1 - epoch 150/700 --- Train cost:0.34203 --- Test cost:0.14134
0.7-0.7 run 1 - epoch 155/700 --- Train cost:0.35293 --- Test cost:0.14463
0.7-0.7 run 1 - epoch 160/700 --- Train cost:0.35675 --- Test cost:0.14175
0.7-0.7 run 1 - epoch 165/700 --- Train cost:0.35355 --- Test cost:0.13170
0.7-0.7 run 1 - epoch 170/700 --- Train cost:0.35530 --- Test cost:0.12645
0.7-0.7 run 1 - epoch 175/700 --- Train cost:0.36155 --- Test cost:0.12634
0.7-0.7 run 1 - epoch 180/700 --- Train cost:0.36238 --- Test cost:0.12274
0.7-0.7 run 1 - epoch 185/700 --- Train cost:0.35822 --- Test cost:0.12516
0.7-0.7 run 1 - epoch 190/700 --- Train cost:0.35456 --- Test cost:0.12251
0.7-0.7 run 1 - epoch 195/700 --- Train cost:0.35061 --- Test cost:0.12055
0.7-0.7 run 1 - epoch 200/700 --- Train cost:0.34387 --- Test cost:0.11710
0.7-0.7 run 1 - epoch 205/700 --- Train cost:0.33607 --- Test cost:0.11503
0.7-0.7 run 1 - epoch 210/700 --- Train cost:0.32664 --- Test cost:0.11193
0.7-0.7 run 1 - epoch 215/700 --- Train cost:0.31188 --- Test cost:0.10780
0.7-0.7 run 1 - epoch 220/700 --- Train cost:0.30542 --- Test cost:0.10449
0.7-0.7 run 1 - epoch 225/700 --- Train cost:0.30320 --- Test cost:0.10244
0.7-0.7 run 1 - epoch 230/700 --- Train cost:0.30621 --- Test cost:0.10352
0.7-0.7 run 1 - epoch 235/700 --- Train cost:0.31252 --- Test cost:0.10736
0.7-0.7 run 1 - epoch 240/700 --- Train cost:0.31187 --- Test cost:0.11024
0.7-0.7 run 1 - epoch 245/700 --- Train cost:0.31513 --- Test cost:0.11634
0.7-0.7 run 1 - epoch 250/700 --- Train cost:0.31237 --- Test cost:0.11715
0.7-0.7 run 1 - epoch 255/700 --- Train cost:0.31365 --- Test cost:0.11557
0.7-0.7 run 1 - epoch 260/700 --- Train cost:0.31508 --- Test cost:0.11481
0.7-0.7 run 1 - epoch 265/700 --- Train cost:0.31632 --- Test cost:0.11552
0.7-0.7 run 1 - epoch 270/700 --- Train cost:0.31537 --- Test cost:0.11706
0.7-0.7 run 1 - epoch 275/700 --- Train cost:0.30798 --- Test cost:0.12300
0.7-0.7 run 1 - epoch 280/700 --- Train cost:0.29972 --- Test cost:0.12489
0.7-0.7 run 1 - epoch 285/700 --- Train cost:0.29876 --- Test cost:0.12912
0.7-0.7 run 1 - epoch 290/700 --- Train cost:0.30667 --- Test cost:0.13576
0.7-0.7 run 1 - epoch 295/700 --- Train cost:0.31459 --- Test cost:0.14385
0.7-0.7 run 1 - epoch 300/700 --- Train cost:0.32012 --- Test cost:0.14676
0.7-0.7 run 1 - epoch 305/700 --- Train cost:0.31801 --- Test cost:0.14874
0.7-0.7 run 1 - epoch 310/700 --- Train cost:0.31168 --- Test cost:0.15247
0.7-0.7 run 1 - epoch 315/700 --- Train cost:0.31095 --- Test cost:0.15273
0.7-0.7 run 1 - epoch 320/700 --- Train cost:0.30766 --- Test cost:0.14921
0.7-0.7 run 1 - epoch 325/700 --- Train cost:0.31012 --- Test cost:0.15022
0.7-0.7 run 1 - epoch 330/700 --- Train cost:0.31308 --- Test cost:0.15089
0.7-0.7 run 1 - epoch 335/700 --- Train cost:0.31083 --- Test cost:0.14378
0.7-0.7 run 1 - epoch 340/700 --- Train cost:0.31134 --- Test cost:0.13989
0.7-0.7 run 1 - epoch 345/700 --- Train cost:0.31198 --- Test cost:0.13972
0.7-0.7 run 1 - epoch 350/700 --- Train cost:0.32131 --- Test cost:0.14785
0.7-0.7 run 1 - epoch 355/700 --- Train cost:0.33550 --- Test cost:0.16138
0.7-0.7 run 1 - epoch 360/700 --- Train cost:0.33708 --- Test cost:0.16262
0.7-0.7 run 1 - epoch 365/700 --- Train cost:0.33248 --- Test cost:0.15744
0.7-0.7 run 1 - epoch 370/700 --- Train cost:0.33540 --- Test cost:0.15838
0.7-0.7 run 1 - epoch 375/700 --- Train cost:0.33821 --- Test cost:0.16209
0.7-0.7 run 1 - epoch 380/700 --- Train cost:0.33374 --- Test cost:0.16230
0.7-0.7 run 1 - epoch 385/700 --- Train cost:0.32498 --- Test cost:0.15911
0.7-0.7 run 1 - epoch 390/700 --- Train cost:0.32137 --- Test cost:0.15766
0.7-0.7 run 1 - epoch 395/700 --- Train cost:0.31979 --- Test cost:0.16168
0.7-0.7 run 1 - epoch 400/700 --- Train cost:0.31886 --- Test cost:0.17175
0.7-0.7 run 1 - epoch 405/700 --- Train cost:0.31401 --- Test cost:0.17555
0.7-0.7 run 1 - epoch 410/700 --- Train cost:0.31555 --- Test cost:0.17807
0.7-0.7 run 1 - epoch 415/700 --- Train cost:0.31647 --- Test cost:0.18183
0.7-0.7 run 1 - epoch 420/700 --- Train cost:0.31898 --- Test cost:0.18739
0.7-0.7 run 1 - epoch 425/700 --- Train cost:0.31567 --- Test cost:0.19097
0.7-0.7 run 1 - epoch 430/700 --- Train cost:0.31468 --- Test cost:0.19269
0.7-0.7 run 1 - epoch 435/700 --- Train cost:0.30736 --- Test cost:0.19150
0.7-0.7 run 1 - epoch 440/700 --- Train cost:0.29837 --- Test cost:0.19498
0.7-0.7 run 1 - epoch 445/700 --- Train cost:0.28992 --- Test cost:0.20124
0.7-0.7 run 1 - epoch 450/700 --- Train cost:0.28765 --- Test cost:0.21018
0.7-0.7 run 1 - epoch 455/700 --- Train cost:0.28457 --- Test cost:0.22184
0.7-0.7 run 1 - epoch 460/700 --- Train cost:0.28292 --- Test cost:0.23305
0.7-0.7 run 1 - epoch 465/700 --- Train cost:0.27106 --- Test cost:0.22972
0.7-0.7 run 1 - epoch 470/700 --- Train cost:0.26564 --- Test cost:0.22419
0.7-0.7 run 1 - epoch 475/700 --- Train cost:0.26370 --- Test cost:0.21683
0.7-0.7 run 1 - epoch 480/700 --- Train cost:0.25979 --- Test cost:0.20745
0.7-0.7 run 1 - epoch 485/700 --- Train cost:0.25938 --- Test cost:0.20060
0.7-0.7 run 1 - epoch 490/700 --- Train cost:0.25192 --- Test cost:0.18966
0.7-0.7 run 1 - epoch 495/700 --- Train cost:0.24650 --- Test cost:0.17946
0.7-0.7 run 1 - epoch 500/700 --- Train cost:0.24864 --- Test cost:0.17283
0.7-0.7 run 1 - epoch 505/700 --- Train cost:0.25101 --- Test cost:0.16612
0.7-0.7 run 1 - epoch 510/700 --- Train cost:0.25201 --- Test cost:0.15781
0.7-0.7 run 1 - epoch 515/700 --- Train cost:0.26089 --- Test cost:0.15945
0.7-0.7 run 1 - epoch 520/700 --- Train cost:0.27329 --- Test cost:0.15753
0.7-0.7 run 1 - epoch 525/700 --- Train cost:0.27786 --- Test cost:0.14996
0.7-0.7 run 1 - epoch 530/700 --- Train cost:0.28370 --- Test cost:0.14662
0.7-0.7 run 1 - epoch 535/700 --- Train cost:0.28911 --- Test cost:0.14295
0.7-0.7 run 1 - epoch 540/700 --- Train cost:0.28622 --- Test cost:0.14291
0.7-0.7 run 1 - epoch 545/700 --- Train cost:0.28351 --- Test cost:0.13921
0.7-0.7 run 1 - epoch 550/700 --- Train cost:0.27121 --- Test cost:0.13668
0.7-0.7 run 1 - epoch 555/700 --- Train cost:0.24713 --- Test cost:0.12633
0.7-0.7 run 1 - epoch 560/700 --- Train cost:0.24302 --- Test cost:0.12539
0.7-0.7 run 1 - epoch 565/700 --- Train cost:0.24177 --- Test cost:0.12594
0.7-0.7 run 1 - epoch 570/700 --- Train cost:0.24071 --- Test cost:0.12040
0.7-0.7 run 1 - epoch 575/700 --- Train cost:0.24747 --- Test cost:0.11985
0.7-0.7 run 1 - epoch 580/700 --- Train cost:0.24997 --- Test cost:0.11531
0.7-0.7 run 1 - epoch 585/700 --- Train cost:0.25019 --- Test cost:0.11012
0.7-0.7 run 1 - epoch 590/700 --- Train cost:0.24599 --- Test cost:0.09890
0.7-0.7 run 1 - epoch 595/700 --- Train cost:0.24409 --- Test cost:0.08938
0.7-0.7 run 1 - epoch 600/700 --- Train cost:0.24055 --- Test cost:0.08287
0.7-0.7 run 1 - epoch 605/700 --- Train cost:0.24055 --- Test cost:0.08124
0.7-0.7 run 1 - epoch 610/700 --- Train cost:0.23607 --- Test cost:0.08007
0.7-0.7 run 1 - epoch 615/700 --- Train cost:0.23199 --- Test cost:0.07462
0.7-0.7 run 1 - epoch 620/700 --- Train cost:0.23038 --- Test cost:0.07189
0.7-0.7 run 1 - epoch 625/700 --- Train cost:0.23302 --- Test cost:0.07066
0.7-0.7 run 1 - epoch 630/700 --- Train cost:0.23930 --- Test cost:0.06843
0.7-0.7 run 1 - epoch 635/700 --- Train cost:0.24611 --- Test cost:0.06503
0.7-0.7 run 1 - epoch 640/700 --- Train cost:0.24974 --- Test cost:0.06417
0.7-0.7 run 1 - epoch 645/700 --- Train cost:0.25392 --- Test cost:0.06473
0.7-0.7 run 1 - epoch 650/700 --- Train cost:0.25927 --- Test cost:0.06840
0.7-0.7 run 1 - epoch 655/700 --- Train cost:0.26451 --- Test cost:0.07206
0.7-0.7 run 1 - epoch 660/700 --- Train cost:0.27431 --- Test cost:0.07613
0.7-0.7 run 1 - epoch 665/700 --- Train cost:0.28359 --- Test cost:0.07886
0.7-0.7 run 1 - epoch 670/700 --- Train cost:0.28954 --- Test cost:0.08042
0.7-0.7 run 1 - epoch 675/700 --- Train cost:0.29998 --- Test cost:0.08122
0.7-0.7 run 1 - epoch 680/700 --- Train cost:0.30792 --- Test cost:0.08440
0.7-0.7 run 1 - epoch 685/700 --- Train cost:0.31818 --- Test cost:0.08598
0.7-0.7 run 1 - epoch 690/700 --- Train cost:0.32664 --- Test cost:0.08584
0.7-0.7 run 1 - epoch 695/700 --- Train cost:0.33422 --- Test cost:0.08522
0.7-0.7 run 2 - epoch 0/700 --- Train cost:0.37401 --- Test cost:0.16714
0.7-0.7 run 2 - epoch 5/700 --- Train cost:0.38369 --- Test cost:0.15754
0.7-0.7 run 2 - epoch 10/700 --- Train cost:0.39699 --- Test cost:0.16756
0.7-0.7 run 2 - epoch 15/700 --- Train cost:0.40373 --- Test cost:0.17435
0.7-0.7 run 2 - epoch 20/700 --- Train cost:0.40979 --- Test cost:0.18412
0.7-0.7 run 2 - epoch 25/700 --- Train cost:0.41868 --- Test cost:0.19635
0.7-0.7 run 2 - epoch 30/700 --- Train cost:0.42532 --- Test cost:0.20321
0.7-0.7 run 2 - epoch 35/700 --- Train cost:0.43590 --- Test cost:0.21194
0.7-0.7 run 2 - epoch 40/700 --- Train cost:0.44526 --- Test cost:0.21365
0.7-0.7 run 2 - epoch 45/700 --- Train cost:0.45004 --- Test cost:0.21677
0.7-0.7 run 2 - epoch 50/700 --- Train cost:0.45354 --- Test cost:0.22390
0.7-0.7 run 2 - epoch 55/700 --- Train cost:0.45082 --- Test cost:0.22317
0.7-0.7 run 2 - epoch 60/700 --- Train cost:0.45462 --- Test cost:0.22701
0.7-0.7 run 2 - epoch 65/700 --- Train cost:0.46895 --- Test cost:0.23407
0.7-0.7 run 2 - epoch 70/700 --- Train cost:0.47450 --- Test cost:0.23077
0.7-0.7 run 2 - epoch 75/700 --- Train cost:0.48380 --- Test cost:0.23090
0.7-0.7 run 2 - epoch 80/700 --- Train cost:0.49063 --- Test cost:0.24205
0.7-0.7 run 2 - epoch 85/700 --- Train cost:0.49676 --- Test cost:0.24845
0.7-0.7 run 2 - epoch 90/700 --- Train cost:0.49630 --- Test cost:0.24431
0.7-0.7 run 2 - epoch 95/700 --- Train cost:0.50739 --- Test cost:0.24749
0.7-0.7 run 2 - epoch 100/700 --- Train cost:0.51176 --- Test cost:0.24729
0.7-0.7 run 2 - epoch 105/700 --- Train cost:0.50988 --- Test cost:0.23966
0.7-0.7 run 2 - epoch 110/700 --- Train cost:0.50731 --- Test cost:0.22839
0.7-0.7 run 2 - epoch 115/700 --- Train cost:0.49640 --- Test cost:0.21617
0.7-0.7 run 2 - epoch 120/700 --- Train cost:0.50322 --- Test cost:0.21768
0.7-0.7 run 2 - epoch 125/700 --- Train cost:0.51728 --- Test cost:0.22410
0.7-0.7 run 2 - epoch 130/700 --- Train cost:0.51751 --- Test cost:0.22351
0.7-0.7 run 2 - epoch 135/700 --- Train cost:0.51789 --- Test cost:0.22254
0.7-0.7 run 2 - epoch 140/700 --- Train cost:0.50712 --- Test cost:0.20596
0.7-0.7 run 2 - epoch 145/700 --- Train cost:0.50923 --- Test cost:0.20111
0.7-0.7 run 2 - epoch 150/700 --- Train cost:0.51981 --- Test cost:0.19884
0.7-0.7 run 2 - epoch 155/700 --- Train cost:0.52774 --- Test cost:0.20481
0.7-0.7 run 2 - epoch 160/700 --- Train cost:0.52958 --- Test cost:0.21109
0.7-0.7 run 2 - epoch 165/700 --- Train cost:0.53263 --- Test cost:0.21732
0.7-0.7 run 2 - epoch 170/700 --- Train cost:0.53219 --- Test cost:0.21988
0.7-0.7 run 2 - epoch 175/700 --- Train cost:0.53100 --- Test cost:0.22886
0.7-0.7 run 2 - epoch 180/700 --- Train cost:0.52766 --- Test cost:0.23419
0.7-0.7 run 2 - epoch 185/700 --- Train cost:0.52484 --- Test cost:0.23345
0.7-0.7 run 2 - epoch 190/700 --- Train cost:0.52674 --- Test cost:0.23846
0.7-0.7 run 2 - epoch 195/700 --- Train cost:0.52637 --- Test cost:0.23948
0.7-0.7 run 2 - epoch 200/700 --- Train cost:0.51790 --- Test cost:0.23756
0.7-0.7 run 2 - epoch 205/700 --- Train cost:0.51252 --- Test cost:0.23939
0.7-0.7 run 2 - epoch 210/700 --- Train cost:0.50808 --- Test cost:0.24008
0.7-0.7 run 2 - epoch 215/700 --- Train cost:0.49692 --- Test cost:0.23191
0.7-0.7 run 2 - epoch 220/700 --- Train cost:0.47958 --- Test cost:0.22474
0.7-0.7 run 2 - epoch 225/700 --- Train cost:0.47624 --- Test cost:0.22279
0.7-0.7 run 2 - epoch 230/700 --- Train cost:0.46946 --- Test cost:0.21351
0.7-0.7 run 2 - epoch 235/700 --- Train cost:0.46608 --- Test cost:0.20953
0.7-0.7 run 2 - epoch 240/700 --- Train cost:0.47163 --- Test cost:0.20564
0.7-0.7 run 2 - epoch 245/700 --- Train cost:0.47409 --- Test cost:0.20331
0.7-0.7 run 2 - epoch 250/700 --- Train cost:0.47398 --- Test cost:0.20205
0.7-0.7 run 2 - epoch 255/700 --- Train cost:0.48132 --- Test cost:0.20843
0.7-0.7 run 2 - epoch 260/700 --- Train cost:0.48416 --- Test cost:0.21112
0.7-0.7 run 2 - epoch 265/700 --- Train cost:0.48746 --- Test cost:0.20545
0.7-0.7 run 2 - epoch 270/700 --- Train cost:0.49397 --- Test cost:0.20514
0.7-0.7 run 2 - epoch 275/700 --- Train cost:0.50312 --- Test cost:0.19975
0.7-0.7 run 2 - epoch 280/700 --- Train cost:0.50932 --- Test cost:0.19193
0.7-0.7 run 2 - epoch 285/700 --- Train cost:0.51182 --- Test cost:0.18140
0.7-0.7 run 2 - epoch 290/700 --- Train cost:0.51096 --- Test cost:0.17216
0.7-0.7 run 2 - epoch 295/700 --- Train cost:0.51677 --- Test cost:0.16731
0.7-0.7 run 2 - epoch 300/700 --- Train cost:0.52956 --- Test cost:0.17133
0.7-0.7 run 2 - epoch 305/700 --- Train cost:0.54421 --- Test cost:0.17103
0.7-0.7 run 2 - epoch 310/700 --- Train cost:0.54871 --- Test cost:0.17076
0.7-0.7 run 2 - epoch 315/700 --- Train cost:0.55535 --- Test cost:0.16884
0.7-0.7 run 2 - epoch 320/700 --- Train cost:0.56483 --- Test cost:0.17044
0.7-0.7 run 2 - epoch 325/700 --- Train cost:0.56122 --- Test cost:0.17665
0.7-0.7 run 2 - epoch 330/700 --- Train cost:0.55180 --- Test cost:0.18125
0.7-0.7 run 2 - epoch 335/700 --- Train cost:0.55243 --- Test cost:0.19562
0.7-0.7 run 2 - epoch 340/700 --- Train cost:0.55877 --- Test cost:0.21321
0.7-0.7 run 2 - epoch 345/700 --- Train cost:0.56604 --- Test cost:0.23505
0.7-0.7 run 2 - epoch 350/700 --- Train cost:0.56562 --- Test cost:0.24703
0.7-0.7 run 2 - epoch 355/700 --- Train cost:0.56474 --- Test cost:0.24705
0.7-0.7 run 2 - epoch 360/700 --- Train cost:0.56264 --- Test cost:0.23709
0.7-0.7 run 2 - epoch 365/700 --- Train cost:0.56610 --- Test cost:0.23458
0.7-0.7 run 2 - epoch 370/700 --- Train cost:0.57724 --- Test cost:0.23848
0.7-0.7 run 2 - epoch 375/700 --- Train cost:0.58381 --- Test cost:0.24055
0.7-0.7 run 2 - epoch 380/700 --- Train cost:0.58681 --- Test cost:0.22896
0.7-0.7 run 2 - epoch 385/700 --- Train cost:0.57849 --- Test cost:0.21829
0.7-0.7 run 2 - epoch 390/700 --- Train cost:0.57526 --- Test cost:0.21713
0.7-0.7 run 2 - epoch 395/700 --- Train cost:0.57066 --- Test cost:0.21360
0.7-0.7 run 2 - epoch 400/700 --- Train cost:0.55831 --- Test cost:0.20889
0.7-0.7 run 2 - epoch 405/700 --- Train cost:0.54510 --- Test cost:0.20701
0.7-0.7 run 2 - epoch 410/700 --- Train cost:0.54627 --- Test cost:0.20956
0.7-0.7 run 2 - epoch 415/700 --- Train cost:0.55227 --- Test cost:0.21839
0.7-0.7 run 2 - epoch 420/700 --- Train cost:0.55974 --- Test cost:0.22540
0.7-0.7 run 2 - epoch 425/700 --- Train cost:0.56738 --- Test cost:0.23546
0.7-0.7 run 2 - epoch 430/700 --- Train cost:0.57224 --- Test cost:0.23986
0.7-0.7 run 2 - epoch 435/700 --- Train cost:0.57561 --- Test cost:0.25008
0.7-0.7 run 2 - epoch 440/700 --- Train cost:0.57805 --- Test cost:0.25512
0.7-0.7 run 2 - epoch 445/700 --- Train cost:0.56517 --- Test cost:0.25398
0.7-0.7 run 2 - epoch 450/700 --- Train cost:0.54523 --- Test cost:0.25012
0.7-0.7 run 2 - epoch 455/700 --- Train cost:0.52495 --- Test cost:0.24045
0.7-0.7 run 2 - epoch 460/700 --- Train cost:0.51111 --- Test cost:0.23039
0.7-0.7 run 2 - epoch 465/700 --- Train cost:0.49880 --- Test cost:0.21973
0.7-0.7 run 2 - epoch 470/700 --- Train cost:0.48413 --- Test cost:0.20893
0.7-0.7 run 2 - epoch 475/700 --- Train cost:0.47716 --- Test cost:0.19987
0.7-0.7 run 2 - epoch 480/700 --- Train cost:0.48024 --- Test cost:0.19714
0.7-0.7 run 2 - epoch 485/700 --- Train cost:0.47504 --- Test cost:0.19677
0.7-0.7 run 2 - epoch 490/700 --- Train cost:0.46867 --- Test cost:0.19144
0.7-0.7 run 2 - epoch 495/700 --- Train cost:0.46943 --- Test cost:0.19224
0.7-0.7 run 2 - epoch 500/700 --- Train cost:0.46463 --- Test cost:0.19479
0.7-0.7 run 2 - epoch 505/700 --- Train cost:0.45287 --- Test cost:0.18943
0.7-0.7 run 2 - epoch 510/700 --- Train cost:0.44250 --- Test cost:0.18984
0.7-0.7 run 2 - epoch 515/700 --- Train cost:0.43649 --- Test cost:0.19257
0.7-0.7 run 2 - epoch 520/700 --- Train cost:0.43326 --- Test cost:0.19549
0.7-0.7 run 2 - epoch 525/700 --- Train cost:0.43221 --- Test cost:0.20513
0.7-0.7 run 2 - epoch 530/700 --- Train cost:0.43072 --- Test cost:0.20765
0.7-0.7 run 2 - epoch 535/700 --- Train cost:0.42460 --- Test cost:0.20938
0.7-0.7 run 2 - epoch 540/700 --- Train cost:0.41998 --- Test cost:0.21191
0.7-0.7 run 2 - epoch 545/700 --- Train cost:0.41134 --- Test cost:0.21242
0.7-0.7 run 2 - epoch 550/700 --- Train cost:0.40963 --- Test cost:0.21842
0.7-0.7 run 2 - epoch 555/700 --- Train cost:0.40604 --- Test cost:0.22251
0.7-0.7 run 2 - epoch 560/700 --- Train cost:0.39802 --- Test cost:0.21908
0.7-0.7 run 2 - epoch 565/700 --- Train cost:0.39163 --- Test cost:0.21332
0.7-0.7 run 2 - epoch 570/700 --- Train cost:0.39213 --- Test cost:0.21169
0.7-0.7 run 2 - epoch 575/700 --- Train cost:0.39527 --- Test cost:0.21466
0.7-0.7 run 2 - epoch 580/700 --- Train cost:0.39338 --- Test cost:0.21445
0.7-0.7 run 2 - epoch 585/700 --- Train cost:0.38505 --- Test cost:0.20274
0.7-0.7 run 2 - epoch 590/700 --- Train cost:0.37672 --- Test cost:0.19304
0.7-0.7 run 2 - epoch 595/700 --- Train cost:0.37163 --- Test cost:0.19102
0.7-0.7 run 2 - epoch 600/700 --- Train cost:0.36788 --- Test cost:0.19128
0.7-0.7 run 2 - epoch 605/700 --- Train cost:0.36246 --- Test cost:0.18895
0.7-0.7 run 2 - epoch 610/700 --- Train cost:0.35591 --- Test cost:0.18291
0.7-0.7 run 2 - epoch 615/700 --- Train cost:0.35000 --- Test cost:0.17632
0.7-0.7 run 2 - epoch 620/700 --- Train cost:0.34414 --- Test cost:0.16742
0.7-0.7 run 2 - epoch 625/700 --- Train cost:0.34351 --- Test cost:0.15873
0.7-0.7 run 2 - epoch 630/700 --- Train cost:0.34201 --- Test cost:0.15686
0.7-0.7 run 2 - epoch 635/700 --- Train cost:0.34434 --- Test cost:0.15672
0.7-0.7 run 2 - epoch 640/700 --- Train cost:0.34492 --- Test cost:0.15906
0.7-0.7 run 2 - epoch 645/700 --- Train cost:0.35059 --- Test cost:0.16424
0.7-0.7 run 2 - epoch 650/700 --- Train cost:0.35887 --- Test cost:0.16778
0.7-0.7 run 2 - epoch 655/700 --- Train cost:0.37010 --- Test cost:0.16903
0.7-0.7 run 2 - epoch 660/700 --- Train cost:0.37748 --- Test cost:0.16429
0.7-0.7 run 2 - epoch 665/700 --- Train cost:0.38475 --- Test cost:0.16418
0.7-0.7 run 2 - epoch 670/700 --- Train cost:0.38857 --- Test cost:0.16255
0.7-0.7 run 2 - epoch 675/700 --- Train cost:0.38648 --- Test cost:0.15830
0.7-0.7 run 2 - epoch 680/700 --- Train cost:0.38868 --- Test cost:0.16170
0.7-0.7 run 2 - epoch 685/700 --- Train cost:0.39062 --- Test cost:0.16472
0.7-0.7 run 2 - epoch 690/700 --- Train cost:0.38889 --- Test cost:0.16444
0.7-0.7 run 2 - epoch 695/700 --- Train cost:0.38060 --- Test cost:0.16110

Performance evaluation

Let’s compare the difference in performance with a plot:

fig, axs = plt.subplots(1, 2, figsize=(12, 4))
plt.subplots_adjust(wspace=0.05)
axs[0].set_title("MSE train")
for k, v in train_history.items():
    train_losses = np.array(v)
    mean_train_history = np.mean(train_losses, axis=0)
    std_train_history = np.std(train_losses, axis=0,)

    mean_train_history = mean_train_history.reshape((epochs,))
    std_train_history = std_train_history.reshape((epochs,))

    # shadow standard deviation
    axs[0].fill_between(
        range(epochs),
        mean_train_history - std_train_history,
        mean_train_history + std_train_history,
        alpha=0.2,
    )
    # average trend
    axs[0].plot(range(epochs), mean_train_history, label=f"{k}")  # Avg Loss

axs[1].set_title("MSE test")
for k, v in test_history.items():
    test_losses = np.array(v)
    mean_test_history = np.mean(test_losses, axis=0)
    std_test_history = np.std(test_losses, axis=0,)

    mean_test_history = mean_test_history.reshape((epochs,))
    std_test_history = std_test_history.reshape((epochs,))

    # shadow standard deviation
    axs[1].fill_between(
        range(epochs),
        mean_test_history - std_test_history,
        mean_test_history + std_test_history,
        alpha=0.2,
    )
    # averange trend
    axs[1].plot(range(epochs), mean_test_history, label=f"{k}")  # Avg Loss

axs[0].legend(loc="upper center", bbox_to_anchor=(1.01, 1.25), ncol=4, fancybox=True, shadow=True)

for ax in axs.flat:
    ax.set_xlabel("Epochs")
    ax.set_ylabel("MSE")
    ax.set_yscale("log")
    ax.set_ylim([1e-3, 0.6])
    ax.label_outer()

plt.subplots_adjust(bottom=0.3)

plt.show()

On the left you can see that without dropout there is a deep minimization of the training loss, moderate values of dropout converge, whereas high drop probabilities impede any learning. On the right, we can see the difference in generalization during the optimization process. Standard training without dropout initially reaches a low value of generalization error, but as the model starts to learn the noise in the training data (overfitting), the generalization error grows back. Oppositely, moderate values of dropout enable generalization errors comparable to the respective training ones. As the learning is not successful for elevated drop probabilities, the generalization error is huge. It is interesting to notice that the “not-learning” error is very close to the final error of the QNN trained without dropout.

Hence, one can conclude that low values of dropout greatly improve the generalization performance of the model and remove overfitting, even if the randomness of the technique inevitably makes the training a little noisy. On the other hand, high drop probabilities only hinder the training process.

Validation

To validate the technique we can also check how the model predicts in the whole \([-1,1]\) range with and without quantum dropout.

X, X_test, y, y_test = make_sin_dataset(dataset_size=20, test_size=0.25)

# spanning the whole range
x_ax = jnp.linspace(-1, 1, 100).reshape(100, 1)

# selecting which run we want to plot
run = 1

fig, ax = plt.subplots()
styles = ["dashed", "-.", "solid", "-."]
for i, k in enumerate(train_history.keys()):
    if k[0] == 0.3:
        alpha = 1
    else:
        alpha = 0.5
    # predicting and rescaling
    yp = scaler.inverse_transform(qnn(x_ax, opt_params[k][run], keep_rot).reshape(-1, 1))
    plt.plot([[i] for i in np.linspace(-1, 1, 100)], yp, label=k, alpha=alpha, linestyle=styles[i])

plt.scatter(X, y, label="Training", zorder=10)
plt.scatter(X_test, y_test, label="Test", zorder=10)

ylabel = r"$y = \sin(\pi\cdot x) + \epsilon$"
plt.xlabel("x", fontsize="medium")
plt.ylabel(ylabel, fontsize="medium")
plt.legend()
ax.xaxis.set_major_locator(ticker.MultipleLocator(0.5))
ax.yaxis.set_major_locator(ticker.MultipleLocator(0.5))

plt.show()

The model without dropout overfits the noisy data by trying to exactly predict each of them, whereas dropout actually mitigates overfitting and makes the approximation of the underlying sinusoidal function way smoother.

Conclusion

In this demo, we explained the basic idea behind quantum dropout and how to avoid overfitting by randomly “dropping” some rotation gates of a QNN during the training phase. We invite you to check out the paper 1 for more dropout techniques and additional analysis. Try it yourself and develop new dropout strategies.

References

1(1,2): Scala, F., Ceschini, A., Panella, M., & Gerace, D. (2023). A General Approach to Dropout in Quantum Neural Networks. Adv. Quantum Technol., 2300220.
2: Hinton, G., Srivastava, N., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2012). Improving neural networks by preventing co-adaptation of feature detectors. arXiv:1207.0580..
3: Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research, 15(56):1929−1958..
4: Kiani,B. T., Lloyd, S., & Maity, R. (2020). Learning Unitaries by Gradient Descent. arXiv: 2001.11897..
5: Larocca, M., Ju, N., García-Martín, D., Coles, P. J., & Cerezo, M. (2023). Theory of overparametrization in quantum neural networks. Nat. Comp. Science, 3, 542–551..

About the author

Francesco Scala

Francesco Scala is a Ph.D. student in QML at the University of Pavia, Italy. His main research deals with both theoretical and numerical aspects of QML algorithms and Quantum Neural Networks, focusing in particular on overparametrization and regulari...

Total running time of the script: (6 minutes 47.790 seconds)

Share demo

Ask a question on the forum

About

Software & Documentation

Resources

Topics

Community & Support

Dropout in Quantum Neural Networks

What is overfitting and dropout?

Why dropout for Quantum Neural Networks?

Quantum dropout of rotations in a sine regression

The circuit

Dropping rotations

Noisy sinusoidal function

Optimization

Training the model

Performance evaluation

Validation

Conclusion

References

About the author

Related Demos