- Demos/
- Quantum Machine Learning/
Dropout in Quantum Neural Networks
Dropout in Quantum Neural Networks
Published: March 11, 2024. Last updated: November 05, 2024.
Are you struggling with overfitting while training Quantum Neural Networks (QNNs)?
In this demo, we show how to exploit the quantum version of the dropout technique to avoid the problem of overfitting in overparametrized QNNs. What follows is based on the paper “A General Approach to Dropout in Quantum Neural Networks” by F. Scala, et al. 1.
What is overfitting and dropout?
Neural Networks (NNs) usually require highly flexible models with lots of trainable parameters in order to learn a certain underlying function (or data distribution). However, being able to learn with low in-sample error is not enough; generalization — the ability to provide good predictions on previously unseen data — is also desirable.
Highly expressive models may suffer from overfitting, which means that they are trained too well on the training data, and as a result perform poorly on new, unseen data. This happens because the model has learned the noise in the training data, rather than the underlying pattern that is generalizable to new data.
Dropout is a common technique for classical Deep Neural Networks (DNNs) preventing computational units from becoming too specialized and reducing the risk of overfitting 2, 3. It consists of randomly removing neurons or connections only during training to block the flow of information. Once the model is trained, the DNN is employed in its original form.
Why dropout for Quantum Neural Networks?
Recently, it has been shown that the use of overparametrized QNN models changes the optimization landscape by removing lots of local minima 4, 5. On the one hand, this increased number of parameters leads to faster and easier training, but on the other hand, it may drive the model to overfit the data. This is also strictly related to the repeated encoding of classical data to achieve nonlinearity in the computation. This is why, inspired from classical DNNs, one can think of applying some sort of dropout to QNNs. This would correspond to randomly dropping some (groups of) parameterized gates during training to achieve better generalization.
Quantum dropout of rotations in a sine regression
In this demo we will exploit quantum dropout to avoid overfitting during the regression of noisy data originally coming from the sinusoidal function. In particular, we will randomly “drop” rotations during the training phase. In practice, this will correspond to temporarily setting parameters to a value of 0.
Let’s start by importing Pennylane and numpy and fixing the random seed for reproducibility:
import numpy as np
import pennylane as qml
seed = 12345
np.random.seed(seed=seed)
The circuit
Now we define the embedding of classical data and the variational ansatz that will then be combined
to construct our QNN. Dropout will happen inside the variational ansatz. Obtaining dropout with standard
Pennylane would be quite straightforward by means of some “if statements”, but the training procedure
will take ages. Here we will leverage JAX in order to speed up the training process with
Just In Time (JIT) compilation. The drawback is that the definition of the variational ansatz becomes a
little elaborated, since JAX has its own language for conditional statements. For this purpose we
define two functions true_cond and false_cond to work with jax.lax.cond`, which is the JAX
conditional statement. See this demo
for additional insights on how to optimize QNNs with JAX.
Practically speaking, rotation dropout will be performed by passing a list to the ansatz.
The single qubit rotations are applied depending on the values stored in this list:
if the value is negative the rotation is dropped (rotation dropout), otherwise it is applied.
How to produce this list will be explained later in this demo (see the make_dropout function).
import jax # require for Just In Time (JIT) compilation
import jax.numpy as jnp
jax.config.update("jax_platform_name", "cpu")
jax.config.update("jax_enable_x64", True)
def embedding(x, wires):
# Encodes the datum multiple times in the register,
# employing also nonlinear functions
assert len(x) == 1 # check feature is 1-D
for i in wires:
qml.RY(jnp.arcsin(x), wires=i)
for i in wires:
qml.RZ(jnp.arccos(x ** 2), wires=i)
def true_cond(angle):
# necessary for using an if statement within jitted function
# exploiting jax.lax.cond
# if this function is assessed the rotation is dropped
return 0.0
def false_cond(angle):
# necessary for using an if statement within jitted function
# exploiting jax.lax.cond
# if this function is assessed the rotation is kept
return angle
def var_ansatz(
theta, wires, rotations=[qml.RX, qml.RZ, qml.RX], entangler=qml.CNOT, keep_rotation=None
):
"""Single layer of the variational ansatz for our QNN.
We have a single qubit rotation per each qubit (wire) followed by
a linear chain of entangling gates (entangler). This structure is repeated per each rotation in `rotations`
(defining `inner_layers`).
The single qubit rotations are applied depending on the values stored in `keep_rotation`:
if the value is negative the rotation is dropped (rotation dropout), otherwise it is applied.
Params:
- theta: variational angles that will undergo optimization
- wires: list of qubits (wires)
- rotations: list of rotation kind per each `inner_layer`
- entangler: entangling gate
- keep_rotation: list of lists. There is one list per each `inner_layer`.
In each list there are indexes of the rotations that we want to apply.
Some of these values may be substituted by -1 value
which means that the rotation gate wont be applied (dropout).
"""
# the length of `rotations` defines the number of inner layers
N = len(wires)
assert len(theta) == 3 * N
wires = list(wires)
counter = 0
# keep_rotations contains a list per each inner_layer
for rots in keep_rotation:
# we cicle over the elements of the lists inside keep_rotation
for qb, keep_or_drop in enumerate(rots):
rot = rotations[counter] # each inner layer can have a different rotation
angle = theta[counter * N + qb]
# conditional statement implementing dropout
# if `keep_or_drop` is negative the rotation is dropped
angle_drop = jax.lax.cond(keep_or_drop < 0, true_cond, false_cond, angle)
rot(angle_drop, wires=wires[qb])
for qb in wires[:-1]:
entangler(wires=[wires[qb], wires[qb + 1]])
counter += 1
And then we define the hyperparameters of our QNN, namely the number of qubits,
the number of sublayers in the variational ansatz (inner_layers) and the resulting
number of parameters per layer:
n_qubits = 5
inner_layers = 3
params_per_layer = n_qubits * inner_layers
Now we actually build the QNN:
def create_circuit(n_qubits, layers):
device = qml.device("default.qubit", wires=n_qubits)
@qml.qnode(device)
def circuit(x, theta, keep_rot):
# print(x)
# print(theta)
for i in range(layers):
embedding(x, wires=range(n_qubits))
keep_rotation = keep_rot[i]
var_ansatz(
theta[i * params_per_layer : (i + 1) * params_per_layer],
wires=range(n_qubits),
entangler=qml.CNOT,
keep_rotation=keep_rotation,
)
return qml.expval(qml.PauliZ(wires=0)) # we measure only the first qubit
return circuit
Let’s have a look at a single layer of our QNN:
import matplotlib.pyplot as plt
plt.style.use("pennylane.drawer.plot") # set pennylane theme, which is nice to see
# create the circuit with given number of qubits and layers
layers = 1
circ = create_circuit(n_qubits, layers=layers)
# for the moment let's keep all the rotations in all sublayers
keep_all_rot = [
[list(range((n_qubits))) for j in range(1, inner_layers + 1)],
]
# we count the parameters
numbered_params = np.array(range(params_per_layer * layers), dtype=float)
# we encode a single coordinate
single_sample = np.array([0])
qml.draw_mpl(circ, decimals=2,)(single_sample, numbered_params, keep_all_rot)
plt.show()
We now build the model that we will employ for the regression task.
Since we want to have an overparametrized QNN, we will add 10 layers and we exploit
JAX to speed the training up:
layers = 10
qnn_tmp = create_circuit(n_qubits, layers)
qnn_tmp = jax.jit(qnn_tmp)
qnn_batched = jax.vmap(
qnn_tmp, (0, None, None)
) # we want to vmap on 0-axis of the first circuit param
# in this way we process in parallel all the inputs
# We jit for faster execution
qnn = jax.jit(qnn_batched)
Dropping rotations
As anticipated, we need to set some random parameters to 0 at each optimization step. Given a layer
dropout rate \(p_L\) (this will be called layer_drop_rate) and the gate dropout rate \(p_G\)
(this will be called rot_drop_rate), the probability \(p\) that a
gate is dropped in a layer can be calculated with the conditioned probability law:
\[p=p(A\cap B)=p(A|B)p(B)=p_Gp_L\]
where \(B\) represents the selection of a specific layer and \(A\) the selection of a specific gate within the chosen layer.
In the following cell we define a function that produces the list of the indices of rotation gates that
are kept. For gates which are dropped, the value -1 is assigned to the corresponding index. The structure of the list
is nested; we have one list per inner_layer inside one list per each layer, all contained in another list.
This function will be called at each iteration.
def make_dropout(key):
drop_layers = []
for lay in range(layers):
# each layer has prob p_L=layer_drop_rate of being dropped
# according to that for every layer we sample
# if we have to appy dropout in it or not
out = jax.random.choice(
key, jnp.array(range(2)), p=jnp.array([1 - layer_drop_rate, layer_drop_rate])
)
key = jax.random.split(key)[0] # update the random key
if out == 1: # if it has to be dropped
drop_layers.append(lay)
keep_rot = []
# we make list of indexes corresponding to the rotations gates
# that are kept in the computation during a single train step
for i in range(layers):
# each list is divded in layers and then in "inner layers"
# this is strictly related to the QNN architecture that we use
keep_rot_layer = [list(range((n_qubits))) for j in range(1, inner_layers + 1)]
if i in drop_layers: # if dropout has to be applied in this layer
keep_rot_layer = [] # list of indexes for a single layer
inner_keep_r = [] # list of indexes for a single inner layer
for param in range(params_per_layer):
# each rotation within the layer has prob p=rot_drop_rate of being dropped
# according to that for every parameter (rotation) we sample
# if we have to drop it or not
out = jax.random.choice(
key, jnp.array(range(2)), p=jnp.array([1 - rot_drop_rate, rot_drop_rate])
)
key = jax.random.split(key)[0] # update the random key
if out == 0: # if we have to keep it
inner_keep_r.append(param % n_qubits) # % is required because we work
# inner layer by inner layer
else: # if the rotation has to be dropped
inner_keep_r.append(-1) # we assign the value -1
if param % n_qubits == n_qubits - 1: # if it's the last qubit of the register
# append the inner layer list
keep_rot_layer.append(inner_keep_r)
# and reset it
inner_keep_r = []
keep_rot.append(keep_rot_layer)
return jnp.array(keep_rot)
We can check the output of the make_dropout function:
# setting the drop probability
layer_drop_rate, rot_drop_rate = 0.5, 0.3 # 15% probability of dropping a gate
# JAX random key
key = jax.random.PRNGKey(12345)
# create the list of indexes,
# -1 implies we are dropping a gate
keep_rot = make_dropout(key)
# let's just print the list for first layer
print(keep_rot[0])
[[ 0 1 2 3 4]
[ 0 1 -1 3 4]
[ 0 1 2 -1 4]]
Noisy sinusoidal function
To test the effectiveness of the dropout technique, we will use a prototypical dataset with which it is very easy to overfit: the sinusoidal function. We produce some points according to the \(\sin\) function and then we add some white Gaussian noise (noise that follows a normal distribution) \(\epsilon.\) The noise is essential to obtain overfitting; when our model is extremely expressive, it is capable of exactly fit each point and some parameters become hyper-specialized in recognizing the noisy features. This makes predictions on new unseen data difficult, since the overfitting model did not learn the true underlying data distribution. The dropout technique will help in avoiding co-adaptation and hyper-specialization, effectively reducing overfitting.
from sklearn.model_selection import train_test_split
def make_sin_dataset(dataset_size=100, test_size=0.4, noise_value=0.4, plot=False):
"""1D regression problem y=sin(x*\pi)"""
# x-axis
x_ax = np.linspace(-1, 1, dataset_size)
y = [[np.sin(x * np.pi)] for x in x_ax]
np.random.seed(123)
# noise vector
noise = np.array([np.random.normal(0, 0.5, 1) for i in y]) * noise_value
X = np.array(x_ax)
y = np.array(y + noise) # apply noise
# split the dataset
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=test_size, random_state=40, shuffle=True
)
X_train = X_train.reshape(-1, 1)
X_test = X_test.reshape(-1, 1)
y_train = y_train.reshape(-1, 1)
y_test = y_test.reshape(-1, 1)
return X_train, X_test, y_train, y_test
from matplotlib import ticker
X, X_test, y, y_test = make_sin_dataset(dataset_size=20, test_size=0.25)
fig, ax = plt.subplots()
plt.plot(X, y, "o", label="Training")
plt.plot(X_test, y_test, "o", label="Test")
plt.plot(
np.linspace(-1, 1, 100),
[np.sin(x * np.pi) for x in np.linspace(-1, 1, 100)],
linestyle="dotted",
label=r"$\sin(x)$",
)
plt.ylabel(r"$y = \sin(\pi\cdot x) + \epsilon$")
plt.xlabel(r"$x$")
ax.xaxis.set_major_locator(ticker.MultipleLocator(0.5))
ax.yaxis.set_major_locator(ticker.MultipleLocator(0.5))
plt.legend()
plt.show()
Since our circuit is only able to provide outputs in the range \([-1,1],\) we rescale all the noisy data within this range. To do this we leverage the MinMaxScaler from sklearn. It is common practice to fit the scaler only from training data and then apply it also to the test. The reason behind this is that in general one only has knowledge about the training dataset. (If the training dataset is not exhaustively representative of the underlying distribution, this preprocessing may lead to some outliers in the test set to be scaled out of the desired range.)
from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler(feature_range=(-1, 1))
y = scaler.fit_transform(y)
y_test = scaler.transform(y_test)
# reshaping for computation
y = y.reshape(-1,)
y_test = y_test.reshape(-1,)
Optimization
At this point we have to set the hyperparameters of the optimization, namely the number of epochs, the learning rate, and the optimizer:
import optax # optimization using jax
epochs = 700
optimizer = optax.adam(learning_rate=0.01)
We define the cost function as the Mean Square Error:
@jax.jit
def calculate_mse_cost(X, y, theta, keep_rot):
yp = qnn(X, theta, keep_rot)
# depending on your version of Pennylane you may require the following line
#####
yp = jnp.array(yp).T
#####
cost = jnp.mean((yp - y) ** 2)
return cost
# Optimization update step
@jax.jit
def optimizer_update(opt_state, params, x, y, keep_rot):
loss, grads = jax.value_and_grad(lambda theta: calculate_mse_cost(x, y, theta, keep_rot))(
params
)
updates, opt_state = optimizer.update(grads, opt_state)
params = optax.apply_updates(params, updates)
return params, opt_state, loss
Training the model
And now we can try to train the model. We execute different runs of the training to understand the
average behaviour of quantum dropout. To see the effect of dropout we can set different values of
layer_drop_rate and rot_drop_rate:
n_run = 3
drop_rates = [(0.0, 0.0), (0.3, 0.2), (0.7, 0.7)]
train_history = {}
test_history = {}
opt_params = {}
for layer_drop_rate, rot_drop_rate in drop_rates:
# initialization of some lists to store data
costs_per_comb = []
test_costs_per_comb = []
opt_params_per_comb = []
# we execute multiple runs in order to see the average behaviour
for tmp_seed in range(seed, seed + n_run):
key = jax.random.PRNGKey(tmp_seed)
assert len(X.shape) == 2 # X must be a matrix
assert len(y.shape) == 1 # y must be an array
assert X.shape[0] == y.shape[0] # compatibility check
# parameters initialization with gaussian ditribution
initial_params = jax.random.normal(key, shape=(layers * params_per_layer,))
# update the random key
key = jax.random.split(key)[0]
params = jnp.copy(initial_params)
# optimizer initialization
opt_state = optimizer.init(initial_params)
# lists for saving single run training and test cost trend
costs = []
test_costs = []
for epoch in range(epochs):
# generate the list for dropout
keep_rot = make_dropout(key)
# update the random key
key = jax.random.split(key)[0]
# optimization step
params, opt_state, cost = optimizer_update(opt_state, params, X, y, keep_rot)
############## performance evaluation #############
# inference is done with the original model
# with all the gates
keep_rot = jnp.array(
[
[list(range((n_qubits))) for j in range(1, inner_layers + 1)]
for i in range(layers)
]
)
# inference on train set
cost = calculate_mse_cost(X, y, params, keep_rot)
costs.append(cost)
# inference on test set
test_cost = calculate_mse_cost(X_test, y_test, params, keep_rot)
test_costs.append(test_cost)
# we print updates every 5 iterations
if epoch % 5 == 0:
print(
f"{layer_drop_rate:.1f}-{rot_drop_rate:.1f}",
f"run {tmp_seed-seed} - epoch {epoch}/{epochs}",
f"--- Train cost:{cost:.5f}",
f"--- Test cost:{test_cost:.5f}",
end="\r",
)
costs_per_comb.append(costs)
test_costs_per_comb.append(test_costs)
opt_params_per_comb.append(params)
print()
costs_per_comb = np.array(costs_per_comb)
test_costs_per_comb = np.array(test_costs_per_comb)
opt_params_per_comb = np.array(opt_params_per_comb)
train_history[(layer_drop_rate, rot_drop_rate)] = costs_per_comb
test_history[(layer_drop_rate, rot_drop_rate)] = test_costs_per_comb
opt_params[(layer_drop_rate, rot_drop_rate)] = opt_params_per_comb
0.0-0.0 run 0 - epoch 0/700 --- Train cost:0.41948 --- Test cost:0.15604
0.0-0.0 run 0 - epoch 5/700 --- Train cost:0.17625 --- Test cost:0.06937
0.0-0.0 run 0 - epoch 10/700 --- Train cost:0.08036 --- Test cost:0.03687
0.0-0.0 run 0 - epoch 15/700 --- Train cost:0.04314 --- Test cost:0.02814
0.0-0.0 run 0 - epoch 20/700 --- Train cost:0.02818 --- Test cost:0.02979
0.0-0.0 run 0 - epoch 25/700 --- Train cost:0.02180 --- Test cost:0.03449
0.0-0.0 run 0 - epoch 30/700 --- Train cost:0.01806 --- Test cost:0.03888
0.0-0.0 run 0 - epoch 35/700 --- Train cost:0.01544 --- Test cost:0.04073
0.0-0.0 run 0 - epoch 40/700 --- Train cost:0.01343 --- Test cost:0.04056
0.0-0.0 run 0 - epoch 45/700 --- Train cost:0.01194 --- Test cost:0.04051
0.0-0.0 run 0 - epoch 50/700 --- Train cost:0.01096 --- Test cost:0.04126
0.0-0.0 run 0 - epoch 55/700 --- Train cost:0.01028 --- Test cost:0.04203
0.0-0.0 run 0 - epoch 60/700 --- Train cost:0.00970 --- Test cost:0.04225
0.0-0.0 run 0 - epoch 65/700 --- Train cost:0.00920 --- Test cost:0.04211
0.0-0.0 run 0 - epoch 70/700 --- Train cost:0.00876 --- Test cost:0.04196
0.0-0.0 run 0 - epoch 75/700 --- Train cost:0.00839 --- Test cost:0.04192
0.0-0.0 run 0 - epoch 80/700 --- Train cost:0.00804 --- Test cost:0.04194
0.0-0.0 run 0 - epoch 85/700 --- Train cost:0.00773 --- Test cost:0.04201
0.0-0.0 run 0 - epoch 90/700 --- Train cost:0.00744 --- Test cost:0.04207
0.0-0.0 run 0 - epoch 95/700 --- Train cost:0.00717 --- Test cost:0.04200
0.0-0.0 run 0 - epoch 100/700 --- Train cost:0.00691 --- Test cost:0.04184
0.0-0.0 run 0 - epoch 105/700 --- Train cost:0.00667 --- Test cost:0.04179
0.0-0.0 run 0 - epoch 110/700 --- Train cost:0.00644 --- Test cost:0.04194
0.0-0.0 run 0 - epoch 115/700 --- Train cost:0.00621 --- Test cost:0.04219
0.0-0.0 run 0 - epoch 120/700 --- Train cost:0.00600 --- Test cost:0.04245
0.0-0.0 run 0 - epoch 125/700 --- Train cost:0.00580 --- Test cost:0.04272
0.0-0.0 run 0 - epoch 130/700 --- Train cost:0.00560 --- Test cost:0.04305
0.0-0.0 run 0 - epoch 135/700 --- Train cost:0.00541 --- Test cost:0.04343
0.0-0.0 run 0 - epoch 140/700 --- Train cost:0.00522 --- Test cost:0.04384
0.0-0.0 run 0 - epoch 145/700 --- Train cost:0.00505 --- Test cost:0.04427
0.0-0.0 run 0 - epoch 150/700 --- Train cost:0.00487 --- Test cost:0.04471
0.0-0.0 run 0 - epoch 155/700 --- Train cost:0.00471 --- Test cost:0.04515
0.0-0.0 run 0 - epoch 160/700 --- Train cost:0.00454 --- Test cost:0.04563
0.0-0.0 run 0 - epoch 165/700 --- Train cost:0.00438 --- Test cost:0.04613
0.0-0.0 run 0 - epoch 170/700 --- Train cost:0.00423 --- Test cost:0.04665
0.0-0.0 run 0 - epoch 175/700 --- Train cost:0.00408 --- Test cost:0.04718
0.0-0.0 run 0 - epoch 180/700 --- Train cost:0.00394 --- Test cost:0.04774
0.0-0.0 run 0 - epoch 185/700 --- Train cost:0.00379 --- Test cost:0.04831
0.0-0.0 run 0 - epoch 190/700 --- Train cost:0.00366 --- Test cost:0.04890
0.0-0.0 run 0 - epoch 195/700 --- Train cost:0.00352 --- Test cost:0.04949
0.0-0.0 run 0 - epoch 200/700 --- Train cost:0.00339 --- Test cost:0.05009
0.0-0.0 run 0 - epoch 205/700 --- Train cost:0.00327 --- Test cost:0.05070
0.0-0.0 run 0 - epoch 210/700 --- Train cost:0.00315 --- Test cost:0.05131
0.0-0.0 run 0 - epoch 215/700 --- Train cost:0.00303 --- Test cost:0.05192
0.0-0.0 run 0 - epoch 220/700 --- Train cost:0.00291 --- Test cost:0.05253
0.0-0.0 run 0 - epoch 225/700 --- Train cost:0.00280 --- Test cost:0.05313
0.0-0.0 run 0 - epoch 230/700 --- Train cost:0.00270 --- Test cost:0.05372
0.0-0.0 run 0 - epoch 235/700 --- Train cost:0.00260 --- Test cost:0.05429
0.0-0.0 run 0 - epoch 240/700 --- Train cost:0.00250 --- Test cost:0.05486
0.0-0.0 run 0 - epoch 245/700 --- Train cost:0.00241 --- Test cost:0.05541
0.0-0.0 run 0 - epoch 250/700 --- Train cost:0.00232 --- Test cost:0.05594
0.0-0.0 run 0 - epoch 255/700 --- Train cost:0.00223 --- Test cost:0.05645
0.0-0.0 run 0 - epoch 260/700 --- Train cost:0.00215 --- Test cost:0.05695
0.0-0.0 run 0 - epoch 265/700 --- Train cost:0.00207 --- Test cost:0.05743
0.0-0.0 run 0 - epoch 270/700 --- Train cost:0.00199 --- Test cost:0.05788
0.0-0.0 run 0 - epoch 275/700 --- Train cost:0.00192 --- Test cost:0.05832
0.0-0.0 run 0 - epoch 280/700 --- Train cost:0.00185 --- Test cost:0.05874
0.0-0.0 run 0 - epoch 285/700 --- Train cost:0.00179 --- Test cost:0.05915
0.0-0.0 run 0 - epoch 290/700 --- Train cost:0.00173 --- Test cost:0.05954
0.0-0.0 run 0 - epoch 295/700 --- Train cost:0.00167 --- Test cost:0.05993
0.0-0.0 run 0 - epoch 300/700 --- Train cost:0.00161 --- Test cost:0.06030
0.0-0.0 run 0 - epoch 305/700 --- Train cost:0.00155 --- Test cost:0.06067
0.0-0.0 run 0 - epoch 310/700 --- Train cost:0.00150 --- Test cost:0.06103
0.0-0.0 run 0 - epoch 315/700 --- Train cost:0.00145 --- Test cost:0.06140
0.0-0.0 run 0 - epoch 320/700 --- Train cost:0.00140 --- Test cost:0.06176
0.0-0.0 run 0 - epoch 325/700 --- Train cost:0.00135 --- Test cost:0.06213
0.0-0.0 run 0 - epoch 330/700 --- Train cost:0.00130 --- Test cost:0.06250
0.0-0.0 run 0 - epoch 335/700 --- Train cost:0.00125 --- Test cost:0.06288
0.0-0.0 run 0 - epoch 340/700 --- Train cost:0.00121 --- Test cost:0.06327
0.0-0.0 run 0 - epoch 345/700 --- Train cost:0.00117 --- Test cost:0.06366
0.0-0.0 run 0 - epoch 350/700 --- Train cost:0.00112 --- Test cost:0.06406
0.0-0.0 run 0 - epoch 355/700 --- Train cost:0.00108 --- Test cost:0.06447
0.0-0.0 run 0 - epoch 360/700 --- Train cost:0.00104 --- Test cost:0.06489
0.0-0.0 run 0 - epoch 365/700 --- Train cost:0.00100 --- Test cost:0.06532
0.0-0.0 run 0 - epoch 370/700 --- Train cost:0.00096 --- Test cost:0.06575
0.0-0.0 run 0 - epoch 375/700 --- Train cost:0.00093 --- Test cost:0.06619
0.0-0.0 run 0 - epoch 380/700 --- Train cost:0.00089 --- Test cost:0.06664
0.0-0.0 run 0 - epoch 385/700 --- Train cost:0.00086 --- Test cost:0.06709
0.0-0.0 run 0 - epoch 390/700 --- Train cost:0.00083 --- Test cost:0.06755
0.0-0.0 run 0 - epoch 395/700 --- Train cost:0.00080 --- Test cost:0.06801
0.0-0.0 run 0 - epoch 400/700 --- Train cost:0.00077 --- Test cost:0.06847
0.0-0.0 run 0 - epoch 405/700 --- Train cost:0.00074 --- Test cost:0.06894
0.0-0.0 run 0 - epoch 410/700 --- Train cost:0.00071 --- Test cost:0.06940
0.0-0.0 run 0 - epoch 415/700 --- Train cost:0.00069 --- Test cost:0.06986
0.0-0.0 run 0 - epoch 420/700 --- Train cost:0.00066 --- Test cost:0.07032
0.0-0.0 run 0 - epoch 425/700 --- Train cost:0.00064 --- Test cost:0.07078
0.0-0.0 run 0 - epoch 430/700 --- Train cost:0.00062 --- Test cost:0.07123
0.0-0.0 run 0 - epoch 435/700 --- Train cost:0.00060 --- Test cost:0.07167
0.0-0.0 run 0 - epoch 440/700 --- Train cost:0.00058 --- Test cost:0.07211
0.0-0.0 run 0 - epoch 445/700 --- Train cost:0.00056 --- Test cost:0.07254
0.0-0.0 run 0 - epoch 450/700 --- Train cost:0.00054 --- Test cost:0.07296
0.0-0.0 run 0 - epoch 455/700 --- Train cost:0.00053 --- Test cost:0.07338
0.0-0.0 run 0 - epoch 460/700 --- Train cost:0.00051 --- Test cost:0.07378
0.0-0.0 run 0 - epoch 465/700 --- Train cost:0.00050 --- Test cost:0.07418
0.0-0.0 run 0 - epoch 470/700 --- Train cost:0.00048 --- Test cost:0.07456
0.0-0.0 run 0 - epoch 475/700 --- Train cost:0.00047 --- Test cost:0.07494
0.0-0.0 run 0 - epoch 480/700 --- Train cost:0.00045 --- Test cost:0.07531
0.0-0.0 run 0 - epoch 485/700 --- Train cost:0.00044 --- Test cost:0.07567
0.0-0.0 run 0 - epoch 490/700 --- Train cost:0.00043 --- Test cost:0.07602
0.0-0.0 run 0 - epoch 495/700 --- Train cost:0.00042 --- Test cost:0.07636
0.0-0.0 run 0 - epoch 500/700 --- Train cost:0.00041 --- Test cost:0.07669
0.0-0.0 run 0 - epoch 505/700 --- Train cost:0.00040 --- Test cost:0.07701
0.0-0.0 run 0 - epoch 510/700 --- Train cost:0.00039 --- Test cost:0.07733
0.0-0.0 run 0 - epoch 515/700 --- Train cost:0.00038 --- Test cost:0.07763
0.0-0.0 run 0 - epoch 520/700 --- Train cost:0.00037 --- Test cost:0.07793
0.0-0.0 run 0 - epoch 525/700 --- Train cost:0.00036 --- Test cost:0.07822
0.0-0.0 run 0 - epoch 530/700 --- Train cost:0.00035 --- Test cost:0.07851
0.0-0.0 run 0 - epoch 535/700 --- Train cost:0.00034 --- Test cost:0.07879
0.0-0.0 run 0 - epoch 540/700 --- Train cost:0.00033 --- Test cost:0.07906
0.0-0.0 run 0 - epoch 545/700 --- Train cost:0.00032 --- Test cost:0.07932
0.0-0.0 run 0 - epoch 550/700 --- Train cost:0.00032 --- Test cost:0.07958
0.0-0.0 run 0 - epoch 555/700 --- Train cost:0.00031 --- Test cost:0.07983
0.0-0.0 run 0 - epoch 560/700 --- Train cost:0.00030 --- Test cost:0.08008
0.0-0.0 run 0 - epoch 565/700 --- Train cost:0.00029 --- Test cost:0.08032
0.0-0.0 run 0 - epoch 570/700 --- Train cost:0.00029 --- Test cost:0.08056
0.0-0.0 run 0 - epoch 575/700 --- Train cost:0.00028 --- Test cost:0.08079
0.0-0.0 run 0 - epoch 580/700 --- Train cost:0.00027 --- Test cost:0.08102
0.0-0.0 run 0 - epoch 585/700 --- Train cost:0.00027 --- Test cost:0.08124
0.0-0.0 run 0 - epoch 590/700 --- Train cost:0.00026 --- Test cost:0.08146
0.0-0.0 run 0 - epoch 595/700 --- Train cost:0.00026 --- Test cost:0.08167
0.0-0.0 run 0 - epoch 600/700 --- Train cost:0.00025 --- Test cost:0.08188
0.0-0.0 run 0 - epoch 605/700 --- Train cost:0.00024 --- Test cost:0.08209
0.0-0.0 run 0 - epoch 610/700 --- Train cost:0.00024 --- Test cost:0.08229
0.0-0.0 run 0 - epoch 615/700 --- Train cost:0.00023 --- Test cost:0.08249
0.0-0.0 run 0 - epoch 620/700 --- Train cost:0.00023 --- Test cost:0.08269
0.0-0.0 run 0 - epoch 625/700 --- Train cost:0.00022 --- Test cost:0.08288
0.0-0.0 run 0 - epoch 630/700 --- Train cost:0.00022 --- Test cost:0.08307
0.0-0.0 run 0 - epoch 635/700 --- Train cost:0.00021 --- Test cost:0.08326
0.0-0.0 run 0 - epoch 640/700 --- Train cost:0.00021 --- Test cost:0.08344
0.0-0.0 run 0 - epoch 645/700 --- Train cost:0.00020 --- Test cost:0.08363
0.0-0.0 run 0 - epoch 650/700 --- Train cost:0.00020 --- Test cost:0.08380
0.0-0.0 run 0 - epoch 655/700 --- Train cost:0.00019 --- Test cost:0.08398
0.0-0.0 run 0 - epoch 660/700 --- Train cost:0.00019 --- Test cost:0.08415
0.0-0.0 run 0 - epoch 665/700 --- Train cost:0.00019 --- Test cost:0.08432
0.0-0.0 run 0 - epoch 670/700 --- Train cost:0.00018 --- Test cost:0.08449
0.0-0.0 run 0 - epoch 675/700 --- Train cost:0.00018 --- Test cost:0.08466
0.0-0.0 run 0 - epoch 680/700 --- Train cost:0.00017 --- Test cost:0.08482
0.0-0.0 run 0 - epoch 685/700 --- Train cost:0.00017 --- Test cost:0.08498
0.0-0.0 run 0 - epoch 690/700 --- Train cost:0.00017 --- Test cost:0.08514
0.0-0.0 run 0 - epoch 695/700 --- Train cost:0.00016 --- Test cost:0.08529
0.0-0.0 run 1 - epoch 0/700 --- Train cost:0.24656 --- Test cost:0.08702
0.0-0.0 run 1 - epoch 5/700 --- Train cost:0.08913 --- Test cost:0.04158
0.0-0.0 run 1 - epoch 10/700 --- Train cost:0.04125 --- Test cost:0.03336
0.0-0.0 run 1 - epoch 15/700 --- Train cost:0.02747 --- Test cost:0.03105
0.0-0.0 run 1 - epoch 20/700 --- Train cost:0.02192 --- Test cost:0.02927
0.0-0.0 run 1 - epoch 25/700 --- Train cost:0.01856 --- Test cost:0.02839
0.0-0.0 run 1 - epoch 30/700 --- Train cost:0.01619 --- Test cost:0.02840
0.0-0.0 run 1 - epoch 35/700 --- Train cost:0.01447 --- Test cost:0.02881
0.0-0.0 run 1 - epoch 40/700 --- Train cost:0.01311 --- Test cost:0.02902
0.0-0.0 run 1 - epoch 45/700 --- Train cost:0.01186 --- Test cost:0.02896
0.0-0.0 run 1 - epoch 50/700 --- Train cost:0.01078 --- Test cost:0.02889
0.0-0.0 run 1 - epoch 55/700 --- Train cost:0.00986 --- Test cost:0.02908
0.0-0.0 run 1 - epoch 60/700 --- Train cost:0.00904 --- Test cost:0.02942
0.0-0.0 run 1 - epoch 65/700 --- Train cost:0.00832 --- Test cost:0.02980
0.0-0.0 run 1 - epoch 70/700 --- Train cost:0.00768 --- Test cost:0.03032
0.0-0.0 run 1 - epoch 75/700 --- Train cost:0.00712 --- Test cost:0.03113
0.0-0.0 run 1 - epoch 80/700 --- Train cost:0.00661 --- Test cost:0.03226
0.0-0.0 run 1 - epoch 85/700 --- Train cost:0.00615 --- Test cost:0.03358
0.0-0.0 run 1 - epoch 90/700 --- Train cost:0.00573 --- Test cost:0.03490
0.0-0.0 run 1 - epoch 95/700 --- Train cost:0.00535 --- Test cost:0.03619
0.0-0.0 run 1 - epoch 100/700 --- Train cost:0.00501 --- Test cost:0.03755
0.0-0.0 run 1 - epoch 105/700 --- Train cost:0.00469 --- Test cost:0.03904
0.0-0.0 run 1 - epoch 110/700 --- Train cost:0.00440 --- Test cost:0.04061
0.0-0.0 run 1 - epoch 115/700 --- Train cost:0.00413 --- Test cost:0.04219
0.0-0.0 run 1 - epoch 120/700 --- Train cost:0.00388 --- Test cost:0.04374
0.0-0.0 run 1 - epoch 125/700 --- Train cost:0.00365 --- Test cost:0.04523
0.0-0.0 run 1 - epoch 130/700 --- Train cost:0.00343 --- Test cost:0.04665
0.0-0.0 run 1 - epoch 135/700 --- Train cost:0.00323 --- Test cost:0.04802
0.0-0.0 run 1 - epoch 140/700 --- Train cost:0.00303 --- Test cost:0.04937
0.0-0.0 run 1 - epoch 145/700 --- Train cost:0.00285 --- Test cost:0.05070
0.0-0.0 run 1 - epoch 150/700 --- Train cost:0.00268 --- Test cost:0.05199
0.0-0.0 run 1 - epoch 155/700 --- Train cost:0.00252 --- Test cost:0.05326
0.0-0.0 run 1 - epoch 160/700 --- Train cost:0.00237 --- Test cost:0.05451
0.0-0.0 run 1 - epoch 165/700 --- Train cost:0.00223 --- Test cost:0.05574
0.0-0.0 run 1 - epoch 170/700 --- Train cost:0.00210 --- Test cost:0.05694
0.0-0.0 run 1 - epoch 175/700 --- Train cost:0.00198 --- Test cost:0.05813
0.0-0.0 run 1 - epoch 180/700 --- Train cost:0.00186 --- Test cost:0.05929
0.0-0.0 run 1 - epoch 185/700 --- Train cost:0.00175 --- Test cost:0.06042
0.0-0.0 run 1 - epoch 190/700 --- Train cost:0.00166 --- Test cost:0.06153
0.0-0.0 run 1 - epoch 195/700 --- Train cost:0.00156 --- Test cost:0.06260
0.0-0.0 run 1 - epoch 200/700 --- Train cost:0.00148 --- Test cost:0.06364
0.0-0.0 run 1 - epoch 205/700 --- Train cost:0.00140 --- Test cost:0.06466
0.0-0.0 run 1 - epoch 210/700 --- Train cost:0.00133 --- Test cost:0.06564
0.0-0.0 run 1 - epoch 215/700 --- Train cost:0.00126 --- Test cost:0.06659
0.0-0.0 run 1 - epoch 220/700 --- Train cost:0.00119 --- Test cost:0.06751
0.0-0.0 run 1 - epoch 225/700 --- Train cost:0.00113 --- Test cost:0.06840
0.0-0.0 run 1 - epoch 230/700 --- Train cost:0.00108 --- Test cost:0.06926
0.0-0.0 run 1 - epoch 235/700 --- Train cost:0.00103 --- Test cost:0.07010
0.0-0.0 run 1 - epoch 240/700 --- Train cost:0.00098 --- Test cost:0.07091
0.0-0.0 run 1 - epoch 245/700 --- Train cost:0.00093 --- Test cost:0.07169
0.0-0.0 run 1 - epoch 250/700 --- Train cost:0.00089 --- Test cost:0.07244
0.0-0.0 run 1 - epoch 255/700 --- Train cost:0.00085 --- Test cost:0.07317
0.0-0.0 run 1 - epoch 260/700 --- Train cost:0.00081 --- Test cost:0.07388
0.0-0.0 run 1 - epoch 265/700 --- Train cost:0.00078 --- Test cost:0.07456
0.0-0.0 run 1 - epoch 270/700 --- Train cost:0.00075 --- Test cost:0.07522
0.0-0.0 run 1 - epoch 275/700 --- Train cost:0.00072 --- Test cost:0.07586
0.0-0.0 run 1 - epoch 280/700 --- Train cost:0.00069 --- Test cost:0.07647
0.0-0.0 run 1 - epoch 285/700 --- Train cost:0.00066 --- Test cost:0.07707
0.0-0.0 run 1 - epoch 290/700 --- Train cost:0.00063 --- Test cost:0.07764
0.0-0.0 run 1 - epoch 295/700 --- Train cost:0.00061 --- Test cost:0.07819
0.0-0.0 run 1 - epoch 300/700 --- Train cost:0.00058 --- Test cost:0.07873
0.0-0.0 run 1 - epoch 305/700 --- Train cost:0.00056 --- Test cost:0.07924
0.0-0.0 run 1 - epoch 310/700 --- Train cost:0.00054 --- Test cost:0.07974
0.0-0.0 run 1 - epoch 315/700 --- Train cost:0.00052 --- Test cost:0.08022
0.0-0.0 run 1 - epoch 320/700 --- Train cost:0.00050 --- Test cost:0.08068
0.0-0.0 run 1 - epoch 325/700 --- Train cost:0.00048 --- Test cost:0.08113
0.0-0.0 run 1 - epoch 330/700 --- Train cost:0.00046 --- Test cost:0.08156
0.0-0.0 run 1 - epoch 335/700 --- Train cost:0.00045 --- Test cost:0.08198
0.0-0.0 run 1 - epoch 340/700 --- Train cost:0.00043 --- Test cost:0.08238
0.0-0.0 run 1 - epoch 345/700 --- Train cost:0.00042 --- Test cost:0.08277
0.0-0.0 run 1 - epoch 350/700 --- Train cost:0.00040 --- Test cost:0.08314
0.0-0.0 run 1 - epoch 355/700 --- Train cost:0.00039 --- Test cost:0.08351
0.0-0.0 run 1 - epoch 360/700 --- Train cost:0.00038 --- Test cost:0.08386
0.0-0.0 run 1 - epoch 365/700 --- Train cost:0.00036 --- Test cost:0.08420
0.0-0.0 run 1 - epoch 370/700 --- Train cost:0.00035 --- Test cost:0.08453
0.0-0.0 run 1 - epoch 375/700 --- Train cost:0.00034 --- Test cost:0.08485
0.0-0.0 run 1 - epoch 380/700 --- Train cost:0.00033 --- Test cost:0.08515
0.0-0.0 run 1 - epoch 385/700 --- Train cost:0.00032 --- Test cost:0.08545
0.0-0.0 run 1 - epoch 390/700 --- Train cost:0.00031 --- Test cost:0.08574
0.0-0.0 run 1 - epoch 395/700 --- Train cost:0.00030 --- Test cost:0.08602
0.0-0.0 run 1 - epoch 400/700 --- Train cost:0.00029 --- Test cost:0.08629
0.0-0.0 run 1 - epoch 405/700 --- Train cost:0.00028 --- Test cost:0.08656
0.0-0.0 run 1 - epoch 410/700 --- Train cost:0.00028 --- Test cost:0.08681
0.0-0.0 run 1 - epoch 415/700 --- Train cost:0.00027 --- Test cost:0.08706
0.0-0.0 run 1 - epoch 420/700 --- Train cost:0.00026 --- Test cost:0.08730
0.0-0.0 run 1 - epoch 425/700 --- Train cost:0.00025 --- Test cost:0.08754
0.0-0.0 run 1 - epoch 430/700 --- Train cost:0.00025 --- Test cost:0.08777
0.0-0.0 run 1 - epoch 435/700 --- Train cost:0.00024 --- Test cost:0.08799
0.0-0.0 run 1 - epoch 440/700 --- Train cost:0.00023 --- Test cost:0.08821
0.0-0.0 run 1 - epoch 445/700 --- Train cost:0.00023 --- Test cost:0.08842
0.0-0.0 run 1 - epoch 450/700 --- Train cost:0.00022 --- Test cost:0.08863
0.0-0.0 run 1 - epoch 455/700 --- Train cost:0.00022 --- Test cost:0.08883
0.0-0.0 run 1 - epoch 460/700 --- Train cost:0.00021 --- Test cost:0.08902
0.0-0.0 run 1 - epoch 465/700 --- Train cost:0.00020 --- Test cost:0.08922
0.0-0.0 run 1 - epoch 470/700 --- Train cost:0.00020 --- Test cost:0.08940
0.0-0.0 run 1 - epoch 475/700 --- Train cost:0.00020 --- Test cost:0.08959
0.0-0.0 run 1 - epoch 480/700 --- Train cost:0.00019 --- Test cost:0.08977
0.0-0.0 run 1 - epoch 485/700 --- Train cost:0.00019 --- Test cost:0.08995
0.0-0.0 run 1 - epoch 490/700 --- Train cost:0.00018 --- Test cost:0.09012
0.0-0.0 run 1 - epoch 495/700 --- Train cost:0.00018 --- Test cost:0.09029
0.0-0.0 run 1 - epoch 500/700 --- Train cost:0.00017 --- Test cost:0.09045
0.0-0.0 run 1 - epoch 505/700 --- Train cost:0.00017 --- Test cost:0.09062
0.0-0.0 run 1 - epoch 510/700 --- Train cost:0.00017 --- Test cost:0.09078
0.0-0.0 run 1 - epoch 515/700 --- Train cost:0.00016 --- Test cost:0.09094
0.0-0.0 run 1 - epoch 520/700 --- Train cost:0.00016 --- Test cost:0.09109
0.0-0.0 run 1 - epoch 525/700 --- Train cost:0.00015 --- Test cost:0.09124
0.0-0.0 run 1 - epoch 530/700 --- Train cost:0.00015 --- Test cost:0.09139
0.0-0.0 run 1 - epoch 535/700 --- Train cost:0.00015 --- Test cost:0.09154
0.0-0.0 run 1 - epoch 540/700 --- Train cost:0.00014 --- Test cost:0.09169
0.0-0.0 run 1 - epoch 545/700 --- Train cost:0.00014 --- Test cost:0.09183
0.0-0.0 run 1 - epoch 550/700 --- Train cost:0.00014 --- Test cost:0.09197
0.0-0.0 run 1 - epoch 555/700 --- Train cost:0.00014 --- Test cost:0.09211
0.0-0.0 run 1 - epoch 560/700 --- Train cost:0.00013 --- Test cost:0.09225
0.0-0.0 run 1 - epoch 565/700 --- Train cost:0.00013 --- Test cost:0.09239
0.0-0.0 run 1 - epoch 570/700 --- Train cost:0.00013 --- Test cost:0.09252
0.0-0.0 run 1 - epoch 575/700 --- Train cost:0.00012 --- Test cost:0.09265
0.0-0.0 run 1 - epoch 580/700 --- Train cost:0.00012 --- Test cost:0.09279
0.0-0.0 run 1 - epoch 585/700 --- Train cost:0.00012 --- Test cost:0.09292
0.0-0.0 run 1 - epoch 590/700 --- Train cost:0.00012 --- Test cost:0.09304
0.0-0.0 run 1 - epoch 595/700 --- Train cost:0.00011 --- Test cost:0.09317
0.0-0.0 run 1 - epoch 600/700 --- Train cost:0.00011 --- Test cost:0.09330
0.0-0.0 run 1 - epoch 605/700 --- Train cost:0.00011 --- Test cost:0.09342
0.0-0.0 run 1 - epoch 610/700 --- Train cost:0.00011 --- Test cost:0.09355
0.0-0.0 run 1 - epoch 615/700 --- Train cost:0.00011 --- Test cost:0.09367
0.0-0.0 run 1 - epoch 620/700 --- Train cost:0.00010 --- Test cost:0.09379
0.0-0.0 run 1 - epoch 625/700 --- Train cost:0.00010 --- Test cost:0.09391
0.0-0.0 run 1 - epoch 630/700 --- Train cost:0.00010 --- Test cost:0.09403
0.0-0.0 run 1 - epoch 635/700 --- Train cost:0.00010 --- Test cost:0.09415
0.0-0.0 run 1 - epoch 640/700 --- Train cost:0.00010 --- Test cost:0.09426
0.0-0.0 run 1 - epoch 645/700 --- Train cost:0.00009 --- Test cost:0.09438
0.0-0.0 run 1 - epoch 650/700 --- Train cost:0.00009 --- Test cost:0.09450
0.0-0.0 run 1 - epoch 655/700 --- Train cost:0.00009 --- Test cost:0.09461
0.0-0.0 run 1 - epoch 660/700 --- Train cost:0.00009 --- Test cost:0.09472
0.0-0.0 run 1 - epoch 665/700 --- Train cost:0.00009 --- Test cost:0.09484
0.0-0.0 run 1 - epoch 670/700 --- Train cost:0.00008 --- Test cost:0.09495
0.0-0.0 run 1 - epoch 675/700 --- Train cost:0.00008 --- Test cost:0.09506
0.0-0.0 run 1 - epoch 680/700 --- Train cost:0.00008 --- Test cost:0.09517
0.0-0.0 run 1 - epoch 685/700 --- Train cost:0.00008 --- Test cost:0.09528
0.0-0.0 run 1 - epoch 690/700 --- Train cost:0.00008 --- Test cost:0.09539
0.0-0.0 run 1 - epoch 695/700 --- Train cost:0.00008 --- Test cost:0.09550
0.0-0.0 run 2 - epoch 0/700 --- Train cost:0.32211 --- Test cost:0.14482
0.0-0.0 run 2 - epoch 5/700 --- Train cost:0.15515 --- Test cost:0.07197
0.0-0.0 run 2 - epoch 10/700 --- Train cost:0.07803 --- Test cost:0.04644
0.0-0.0 run 2 - epoch 15/700 --- Train cost:0.04212 --- Test cost:0.04713
0.0-0.0 run 2 - epoch 20/700 --- Train cost:0.02660 --- Test cost:0.06245
0.0-0.0 run 2 - epoch 25/700 --- Train cost:0.01997 --- Test cost:0.07354
0.0-0.0 run 2 - epoch 30/700 --- Train cost:0.01639 --- Test cost:0.07600
0.0-0.0 run 2 - epoch 35/700 --- Train cost:0.01419 --- Test cost:0.07577
0.0-0.0 run 2 - epoch 40/700 --- Train cost:0.01266 --- Test cost:0.07584
0.0-0.0 run 2 - epoch 45/700 --- Train cost:0.01150 --- Test cost:0.07567
0.0-0.0 run 2 - epoch 50/700 --- Train cost:0.01058 --- Test cost:0.07488
0.0-0.0 run 2 - epoch 55/700 --- Train cost:0.00986 --- Test cost:0.07427
0.0-0.0 run 2 - epoch 60/700 --- Train cost:0.00924 --- Test cost:0.07422
0.0-0.0 run 2 - epoch 65/700 --- Train cost:0.00872 --- Test cost:0.07441
0.0-0.0 run 2 - epoch 70/700 --- Train cost:0.00828 --- Test cost:0.07487
0.0-0.0 run 2 - epoch 75/700 --- Train cost:0.00789 --- Test cost:0.07585
0.0-0.0 run 2 - epoch 80/700 --- Train cost:0.00754 --- Test cost:0.07722
0.0-0.0 run 2 - epoch 85/700 --- Train cost:0.00722 --- Test cost:0.07862
0.0-0.0 run 2 - epoch 90/700 --- Train cost:0.00692 --- Test cost:0.07992
0.0-0.0 run 2 - epoch 95/700 --- Train cost:0.00664 --- Test cost:0.08116
0.0-0.0 run 2 - epoch 100/700 --- Train cost:0.00638 --- Test cost:0.08232
0.0-0.0 run 2 - epoch 105/700 --- Train cost:0.00613 --- Test cost:0.08338
0.0-0.0 run 2 - epoch 110/700 --- Train cost:0.00589 --- Test cost:0.08434
0.0-0.0 run 2 - epoch 115/700 --- Train cost:0.00566 --- Test cost:0.08524
0.0-0.0 run 2 - epoch 120/700 --- Train cost:0.00544 --- Test cost:0.08614
0.0-0.0 run 2 - epoch 125/700 --- Train cost:0.00522 --- Test cost:0.08703
0.0-0.0 run 2 - epoch 130/700 --- Train cost:0.00502 --- Test cost:0.08789
0.0-0.0 run 2 - epoch 135/700 --- Train cost:0.00482 --- Test cost:0.08876
0.0-0.0 run 2 - epoch 140/700 --- Train cost:0.00463 --- Test cost:0.08965
0.0-0.0 run 2 - epoch 145/700 --- Train cost:0.00445 --- Test cost:0.09055
0.0-0.0 run 2 - epoch 150/700 --- Train cost:0.00427 --- Test cost:0.09145
0.0-0.0 run 2 - epoch 155/700 --- Train cost:0.00410 --- Test cost:0.09238
0.0-0.0 run 2 - epoch 160/700 --- Train cost:0.00394 --- Test cost:0.09332
0.0-0.0 run 2 - epoch 165/700 --- Train cost:0.00378 --- Test cost:0.09428
0.0-0.0 run 2 - epoch 170/700 --- Train cost:0.00362 --- Test cost:0.09526
0.0-0.0 run 2 - epoch 175/700 --- Train cost:0.00347 --- Test cost:0.09625
0.0-0.0 run 2 - epoch 180/700 --- Train cost:0.00332 --- Test cost:0.09727
0.0-0.0 run 2 - epoch 185/700 --- Train cost:0.00318 --- Test cost:0.09832
0.0-0.0 run 2 - epoch 190/700 --- Train cost:0.00304 --- Test cost:0.09939
0.0-0.0 run 2 - epoch 195/700 --- Train cost:0.00290 --- Test cost:0.10047
0.0-0.0 run 2 - epoch 200/700 --- Train cost:0.00277 --- Test cost:0.10156
0.0-0.0 run 2 - epoch 205/700 --- Train cost:0.00264 --- Test cost:0.10266
0.0-0.0 run 2 - epoch 210/700 --- Train cost:0.00252 --- Test cost:0.10375
0.0-0.0 run 2 - epoch 215/700 --- Train cost:0.00240 --- Test cost:0.10484
0.0-0.0 run 2 - epoch 220/700 --- Train cost:0.00229 --- Test cost:0.10591
0.0-0.0 run 2 - epoch 225/700 --- Train cost:0.00218 --- Test cost:0.10694
0.0-0.0 run 2 - epoch 230/700 --- Train cost:0.00208 --- Test cost:0.10795
0.0-0.0 run 2 - epoch 235/700 --- Train cost:0.00199 --- Test cost:0.10891
0.0-0.0 run 2 - epoch 240/700 --- Train cost:0.00189 --- Test cost:0.10982
0.0-0.0 run 2 - epoch 245/700 --- Train cost:0.00181 --- Test cost:0.11068
0.0-0.0 run 2 - epoch 250/700 --- Train cost:0.00173 --- Test cost:0.11148
0.0-0.0 run 2 - epoch 255/700 --- Train cost:0.00165 --- Test cost:0.11223
0.0-0.0 run 2 - epoch 260/700 --- Train cost:0.00158 --- Test cost:0.11292
0.0-0.0 run 2 - epoch 265/700 --- Train cost:0.00151 --- Test cost:0.11356
0.0-0.0 run 2 - epoch 270/700 --- Train cost:0.00145 --- Test cost:0.11414
0.0-0.0 run 2 - epoch 275/700 --- Train cost:0.00139 --- Test cost:0.11468
0.0-0.0 run 2 - epoch 280/700 --- Train cost:0.00134 --- Test cost:0.11516
0.0-0.0 run 2 - epoch 285/700 --- Train cost:0.00129 --- Test cost:0.11561
0.0-0.0 run 2 - epoch 290/700 --- Train cost:0.00124 --- Test cost:0.11602
0.0-0.0 run 2 - epoch 295/700 --- Train cost:0.00119 --- Test cost:0.11639
0.0-0.0 run 2 - epoch 300/700 --- Train cost:0.00115 --- Test cost:0.11673
0.0-0.0 run 2 - epoch 305/700 --- Train cost:0.00111 --- Test cost:0.11705
0.0-0.0 run 2 - epoch 310/700 --- Train cost:0.00107 --- Test cost:0.11734
0.0-0.0 run 2 - epoch 315/700 --- Train cost:0.00104 --- Test cost:0.11760
0.0-0.0 run 2 - epoch 320/700 --- Train cost:0.00100 --- Test cost:0.11785
0.0-0.0 run 2 - epoch 325/700 --- Train cost:0.00097 --- Test cost:0.11808
0.0-0.0 run 2 - epoch 330/700 --- Train cost:0.00094 --- Test cost:0.11830
0.0-0.0 run 2 - epoch 335/700 --- Train cost:0.00091 --- Test cost:0.11850
0.0-0.0 run 2 - epoch 340/700 --- Train cost:0.00088 --- Test cost:0.11869
0.0-0.0 run 2 - epoch 345/700 --- Train cost:0.00085 --- Test cost:0.11887
0.0-0.0 run 2 - epoch 350/700 --- Train cost:0.00082 --- Test cost:0.11904
0.0-0.0 run 2 - epoch 355/700 --- Train cost:0.00080 --- Test cost:0.11920
0.0-0.0 run 2 - epoch 360/700 --- Train cost:0.00077 --- Test cost:0.11936
0.0-0.0 run 2 - epoch 365/700 --- Train cost:0.00075 --- Test cost:0.11951
0.0-0.0 run 2 - epoch 370/700 --- Train cost:0.00073 --- Test cost:0.11966
0.0-0.0 run 2 - epoch 375/700 --- Train cost:0.00070 --- Test cost:0.11980
0.0-0.0 run 2 - epoch 380/700 --- Train cost:0.00068 --- Test cost:0.11994
0.0-0.0 run 2 - epoch 385/700 --- Train cost:0.00066 --- Test cost:0.12008
0.0-0.0 run 2 - epoch 390/700 --- Train cost:0.00064 --- Test cost:0.12021
0.0-0.0 run 2 - epoch 395/700 --- Train cost:0.00062 --- Test cost:0.12035
0.0-0.0 run 2 - epoch 400/700 --- Train cost:0.00060 --- Test cost:0.12048
0.0-0.0 run 2 - epoch 405/700 --- Train cost:0.00058 --- Test cost:0.12061
0.0-0.0 run 2 - epoch 410/700 --- Train cost:0.00056 --- Test cost:0.12074
0.0-0.0 run 2 - epoch 415/700 --- Train cost:0.00054 --- Test cost:0.12087
0.0-0.0 run 2 - epoch 420/700 --- Train cost:0.00053 --- Test cost:0.12100
0.0-0.0 run 2 - epoch 425/700 --- Train cost:0.00051 --- Test cost:0.12113
0.0-0.0 run 2 - epoch 430/700 --- Train cost:0.00049 --- Test cost:0.12126
0.0-0.0 run 2 - epoch 435/700 --- Train cost:0.00048 --- Test cost:0.12139
0.0-0.0 run 2 - epoch 440/700 --- Train cost:0.00046 --- Test cost:0.12152
0.0-0.0 run 2 - epoch 445/700 --- Train cost:0.00045 --- Test cost:0.12165
0.0-0.0 run 2 - epoch 450/700 --- Train cost:0.00043 --- Test cost:0.12179
0.0-0.0 run 2 - epoch 455/700 --- Train cost:0.00042 --- Test cost:0.12192
0.0-0.0 run 2 - epoch 460/700 --- Train cost:0.00040 --- Test cost:0.12206
0.0-0.0 run 2 - epoch 465/700 --- Train cost:0.00039 --- Test cost:0.12219
0.0-0.0 run 2 - epoch 470/700 --- Train cost:0.00038 --- Test cost:0.12233
0.0-0.0 run 2 - epoch 475/700 --- Train cost:0.00036 --- Test cost:0.12247
0.0-0.0 run 2 - epoch 480/700 --- Train cost:0.00035 --- Test cost:0.12261
0.0-0.0 run 2 - epoch 485/700 --- Train cost:0.00034 --- Test cost:0.12275
0.0-0.0 run 2 - epoch 490/700 --- Train cost:0.00033 --- Test cost:0.12289
0.0-0.0 run 2 - epoch 495/700 --- Train cost:0.00032 --- Test cost:0.12303
0.0-0.0 run 2 - epoch 500/700 --- Train cost:0.00031 --- Test cost:0.12317
0.0-0.0 run 2 - epoch 505/700 --- Train cost:0.00030 --- Test cost:0.12331
0.0-0.0 run 2 - epoch 510/700 --- Train cost:0.00028 --- Test cost:0.12345
0.0-0.0 run 2 - epoch 515/700 --- Train cost:0.00028 --- Test cost:0.12360
0.0-0.0 run 2 - epoch 520/700 --- Train cost:0.00027 --- Test cost:0.12374
0.0-0.0 run 2 - epoch 525/700 --- Train cost:0.00026 --- Test cost:0.12388
0.0-0.0 run 2 - epoch 530/700 --- Train cost:0.00025 --- Test cost:0.12402
0.0-0.0 run 2 - epoch 535/700 --- Train cost:0.00024 --- Test cost:0.12417
0.0-0.0 run 2 - epoch 540/700 --- Train cost:0.00023 --- Test cost:0.12431
0.0-0.0 run 2 - epoch 545/700 --- Train cost:0.00022 --- Test cost:0.12445
0.0-0.0 run 2 - epoch 550/700 --- Train cost:0.00022 --- Test cost:0.12459
0.0-0.0 run 2 - epoch 555/700 --- Train cost:0.00021 --- Test cost:0.12473
0.0-0.0 run 2 - epoch 560/700 --- Train cost:0.00020 --- Test cost:0.12486
0.0-0.0 run 2 - epoch 565/700 --- Train cost:0.00020 --- Test cost:0.12500
0.0-0.0 run 2 - epoch 570/700 --- Train cost:0.00019 --- Test cost:0.12514
0.0-0.0 run 2 - epoch 575/700 --- Train cost:0.00018 --- Test cost:0.12527
0.0-0.0 run 2 - epoch 580/700 --- Train cost:0.00018 --- Test cost:0.12540
0.0-0.0 run 2 - epoch 585/700 --- Train cost:0.00017 --- Test cost:0.12553
0.0-0.0 run 2 - epoch 590/700 --- Train cost:0.00017 --- Test cost:0.12566
0.0-0.0 run 2 - epoch 595/700 --- Train cost:0.00016 --- Test cost:0.12578
0.0-0.0 run 2 - epoch 600/700 --- Train cost:0.00016 --- Test cost:0.12591
0.0-0.0 run 2 - epoch 605/700 --- Train cost:0.00015 --- Test cost:0.12603
0.0-0.0 run 2 - epoch 610/700 --- Train cost:0.00015 --- Test cost:0.12615
0.0-0.0 run 2 - epoch 615/700 --- Train cost:0.00014 --- Test cost:0.12626
0.0-0.0 run 2 - epoch 620/700 --- Train cost:0.00014 --- Test cost:0.12638
0.0-0.0 run 2 - epoch 625/700 --- Train cost:0.00013 --- Test cost:0.12649
0.0-0.0 run 2 - epoch 630/700 --- Train cost:0.00013 --- Test cost:0.12660
0.0-0.0 run 2 - epoch 635/700 --- Train cost:0.00013 --- Test cost:0.12671
0.0-0.0 run 2 - epoch 640/700 --- Train cost:0.00012 --- Test cost:0.12681
0.0-0.0 run 2 - epoch 645/700 --- Train cost:0.00012 --- Test cost:0.12691
0.0-0.0 run 2 - epoch 650/700 --- Train cost:0.00012 --- Test cost:0.12701
0.0-0.0 run 2 - epoch 655/700 --- Train cost:0.00011 --- Test cost:0.12710
0.0-0.0 run 2 - epoch 660/700 --- Train cost:0.00011 --- Test cost:0.12720
0.0-0.0 run 2 - epoch 665/700 --- Train cost:0.00011 --- Test cost:0.12729
0.0-0.0 run 2 - epoch 670/700 --- Train cost:0.00011 --- Test cost:0.12737
0.0-0.0 run 2 - epoch 675/700 --- Train cost:0.00010 --- Test cost:0.12746
0.0-0.0 run 2 - epoch 680/700 --- Train cost:0.00010 --- Test cost:0.12754
0.0-0.0 run 2 - epoch 685/700 --- Train cost:0.00010 --- Test cost:0.12762
0.0-0.0 run 2 - epoch 690/700 --- Train cost:0.00010 --- Test cost:0.12769
0.0-0.0 run 2 - epoch 695/700 --- Train cost:0.00009 --- Test cost:0.12777
0.3-0.2 run 0 - epoch 0/700 --- Train cost:0.47606 --- Test cost:0.17619
0.3-0.2 run 0 - epoch 5/700 --- Train cost:0.40491 --- Test cost:0.15495
0.3-0.2 run 0 - epoch 10/700 --- Train cost:0.36241 --- Test cost:0.13692
0.3-0.2 run 0 - epoch 15/700 --- Train cost:0.31689 --- Test cost:0.12580
0.3-0.2 run 0 - epoch 20/700 --- Train cost:0.27189 --- Test cost:0.11358
0.3-0.2 run 0 - epoch 25/700 --- Train cost:0.24435 --- Test cost:0.11085
0.3-0.2 run 0 - epoch 30/700 --- Train cost:0.23198 --- Test cost:0.11171
0.3-0.2 run 0 - epoch 35/700 --- Train cost:0.20060 --- Test cost:0.10482
0.3-0.2 run 0 - epoch 40/700 --- Train cost:0.15768 --- Test cost:0.09010
0.3-0.2 run 0 - epoch 45/700 --- Train cost:0.11749 --- Test cost:0.07223
0.3-0.2 run 0 - epoch 50/700 --- Train cost:0.08208 --- Test cost:0.05499
0.3-0.2 run 0 - epoch 55/700 --- Train cost:0.06563 --- Test cost:0.04828
0.3-0.2 run 0 - epoch 60/700 --- Train cost:0.05824 --- Test cost:0.04427
0.3-0.2 run 0 - epoch 65/700 --- Train cost:0.05473 --- Test cost:0.04176
0.3-0.2 run 0 - epoch 70/700 --- Train cost:0.05463 --- Test cost:0.04203
0.3-0.2 run 0 - epoch 75/700 --- Train cost:0.05668 --- Test cost:0.04220
0.3-0.2 run 0 - epoch 80/700 --- Train cost:0.05745 --- Test cost:0.04136
0.3-0.2 run 0 - epoch 85/700 --- Train cost:0.05734 --- Test cost:0.04099
0.3-0.2 run 0 - epoch 90/700 --- Train cost:0.05854 --- Test cost:0.04100
0.3-0.2 run 0 - epoch 95/700 --- Train cost:0.06390 --- Test cost:0.04238
0.3-0.2 run 0 - epoch 100/700 --- Train cost:0.06861 --- Test cost:0.04367
0.3-0.2 run 0 - epoch 105/700 --- Train cost:0.06949 --- Test cost:0.04351
0.3-0.2 run 0 - epoch 110/700 --- Train cost:0.06846 --- Test cost:0.04309
0.3-0.2 run 0 - epoch 115/700 --- Train cost:0.06571 --- Test cost:0.04127
0.3-0.2 run 0 - epoch 120/700 --- Train cost:0.06714 --- Test cost:0.04178
0.3-0.2 run 0 - epoch 125/700 --- Train cost:0.06910 --- Test cost:0.04450
0.3-0.2 run 0 - epoch 130/700 --- Train cost:0.06950 --- Test cost:0.04580
0.3-0.2 run 0 - epoch 135/700 --- Train cost:0.07205 --- Test cost:0.04737
0.3-0.2 run 0 - epoch 140/700 --- Train cost:0.07827 --- Test cost:0.05209
0.3-0.2 run 0 - epoch 145/700 --- Train cost:0.08319 --- Test cost:0.05399
0.3-0.2 run 0 - epoch 150/700 --- Train cost:0.08677 --- Test cost:0.05451
0.3-0.2 run 0 - epoch 155/700 --- Train cost:0.08372 --- Test cost:0.05102
0.3-0.2 run 0 - epoch 160/700 --- Train cost:0.07929 --- Test cost:0.04767
0.3-0.2 run 0 - epoch 165/700 --- Train cost:0.07403 --- Test cost:0.04636
0.3-0.2 run 0 - epoch 170/700 --- Train cost:0.06885 --- Test cost:0.04365
0.3-0.2 run 0 - epoch 175/700 --- Train cost:0.06322 --- Test cost:0.04118
0.3-0.2 run 0 - epoch 180/700 --- Train cost:0.05602 --- Test cost:0.03614
0.3-0.2 run 0 - epoch 185/700 --- Train cost:0.04753 --- Test cost:0.03105
0.3-0.2 run 0 - epoch 190/700 --- Train cost:0.04127 --- Test cost:0.02818
0.3-0.2 run 0 - epoch 195/700 --- Train cost:0.03718 --- Test cost:0.02850
0.3-0.2 run 0 - epoch 200/700 --- Train cost:0.03546 --- Test cost:0.03204
0.3-0.2 run 0 - epoch 205/700 --- Train cost:0.03469 --- Test cost:0.03423
0.3-0.2 run 0 - epoch 210/700 --- Train cost:0.03399 --- Test cost:0.03465
0.3-0.2 run 0 - epoch 215/700 --- Train cost:0.03328 --- Test cost:0.03534
0.3-0.2 run 0 - epoch 220/700 --- Train cost:0.03072 --- Test cost:0.03649
0.3-0.2 run 0 - epoch 225/700 --- Train cost:0.02834 --- Test cost:0.03625
0.3-0.2 run 0 - epoch 230/700 --- Train cost:0.02672 --- Test cost:0.03533
0.3-0.2 run 0 - epoch 235/700 --- Train cost:0.02428 --- Test cost:0.03428
0.3-0.2 run 0 - epoch 240/700 --- Train cost:0.02221 --- Test cost:0.03201
0.3-0.2 run 0 - epoch 245/700 --- Train cost:0.02076 --- Test cost:0.02863
0.3-0.2 run 0 - epoch 250/700 --- Train cost:0.01993 --- Test cost:0.02707
0.3-0.2 run 0 - epoch 255/700 --- Train cost:0.01938 --- Test cost:0.02727
0.3-0.2 run 0 - epoch 260/700 --- Train cost:0.01960 --- Test cost:0.02682
0.3-0.2 run 0 - epoch 265/700 --- Train cost:0.01931 --- Test cost:0.02692
0.3-0.2 run 0 - epoch 270/700 --- Train cost:0.01890 --- Test cost:0.02738
0.3-0.2 run 0 - epoch 275/700 --- Train cost:0.01884 --- Test cost:0.02744
0.3-0.2 run 0 - epoch 280/700 --- Train cost:0.01797 --- Test cost:0.02864
0.3-0.2 run 0 - epoch 285/700 --- Train cost:0.01717 --- Test cost:0.02974
0.3-0.2 run 0 - epoch 290/700 --- Train cost:0.01682 --- Test cost:0.03020
0.3-0.2 run 0 - epoch 295/700 --- Train cost:0.01683 --- Test cost:0.02911
0.3-0.2 run 0 - epoch 300/700 --- Train cost:0.01660 --- Test cost:0.02998
0.3-0.2 run 0 - epoch 305/700 --- Train cost:0.01617 --- Test cost:0.03147
0.3-0.2 run 0 - epoch 310/700 --- Train cost:0.01655 --- Test cost:0.02997
0.3-0.2 run 0 - epoch 315/700 --- Train cost:0.01699 --- Test cost:0.02838
0.3-0.2 run 0 - epoch 320/700 --- Train cost:0.01758 --- Test cost:0.02794
0.3-0.2 run 0 - epoch 325/700 --- Train cost:0.01870 --- Test cost:0.02761
0.3-0.2 run 0 - epoch 330/700 --- Train cost:0.01911 --- Test cost:0.02595
0.3-0.2 run 0 - epoch 335/700 --- Train cost:0.02007 --- Test cost:0.02274
0.3-0.2 run 0 - epoch 340/700 --- Train cost:0.02188 --- Test cost:0.02069
0.3-0.2 run 0 - epoch 345/700 --- Train cost:0.02271 --- Test cost:0.01895
0.3-0.2 run 0 - epoch 350/700 --- Train cost:0.02297 --- Test cost:0.01911
0.3-0.2 run 0 - epoch 355/700 --- Train cost:0.02420 --- Test cost:0.01979
0.3-0.2 run 0 - epoch 360/700 --- Train cost:0.02649 --- Test cost:0.01930
0.3-0.2 run 0 - epoch 365/700 --- Train cost:0.02836 --- Test cost:0.02010
0.3-0.2 run 0 - epoch 370/700 --- Train cost:0.02728 --- Test cost:0.02165
0.3-0.2 run 0 - epoch 375/700 --- Train cost:0.02535 --- Test cost:0.02276
0.3-0.2 run 0 - epoch 380/700 --- Train cost:0.02468 --- Test cost:0.02406
0.3-0.2 run 0 - epoch 385/700 --- Train cost:0.02509 --- Test cost:0.02421
0.3-0.2 run 0 - epoch 390/700 --- Train cost:0.02393 --- Test cost:0.02529
0.3-0.2 run 0 - epoch 395/700 --- Train cost:0.02267 --- Test cost:0.02655
0.3-0.2 run 0 - epoch 400/700 --- Train cost:0.02049 --- Test cost:0.02755
0.3-0.2 run 0 - epoch 405/700 --- Train cost:0.01956 --- Test cost:0.02625
0.3-0.2 run 0 - epoch 410/700 --- Train cost:0.01878 --- Test cost:0.02549
0.3-0.2 run 0 - epoch 415/700 --- Train cost:0.01875 --- Test cost:0.02229
0.3-0.2 run 0 - epoch 420/700 --- Train cost:0.01866 --- Test cost:0.02048
0.3-0.2 run 0 - epoch 425/700 --- Train cost:0.01779 --- Test cost:0.01958
0.3-0.2 run 0 - epoch 430/700 --- Train cost:0.01675 --- Test cost:0.02042
0.3-0.2 run 0 - epoch 435/700 --- Train cost:0.01615 --- Test cost:0.02255
0.3-0.2 run 0 - epoch 440/700 --- Train cost:0.01546 --- Test cost:0.02485
0.3-0.2 run 0 - epoch 445/700 --- Train cost:0.01523 --- Test cost:0.02668
0.3-0.2 run 0 - epoch 450/700 --- Train cost:0.01568 --- Test cost:0.02932
0.3-0.2 run 0 - epoch 455/700 --- Train cost:0.01658 --- Test cost:0.03298
0.3-0.2 run 0 - epoch 460/700 --- Train cost:0.01744 --- Test cost:0.03585
0.3-0.2 run 0 - epoch 465/700 --- Train cost:0.01803 --- Test cost:0.03597
0.3-0.2 run 0 - epoch 470/700 --- Train cost:0.01955 --- Test cost:0.03434
0.3-0.2 run 0 - epoch 475/700 --- Train cost:0.02067 --- Test cost:0.03210
0.3-0.2 run 0 - epoch 480/700 --- Train cost:0.02037 --- Test cost:0.03026
0.3-0.2 run 0 - epoch 485/700 --- Train cost:0.01895 --- Test cost:0.02938
0.3-0.2 run 0 - epoch 490/700 --- Train cost:0.01774 --- Test cost:0.02958
0.3-0.2 run 0 - epoch 495/700 --- Train cost:0.01723 --- Test cost:0.03170
0.3-0.2 run 0 - epoch 500/700 --- Train cost:0.01692 --- Test cost:0.03297
0.3-0.2 run 0 - epoch 505/700 --- Train cost:0.01671 --- Test cost:0.03376
0.3-0.2 run 0 - epoch 510/700 --- Train cost:0.01683 --- Test cost:0.03293
0.3-0.2 run 0 - epoch 515/700 --- Train cost:0.01770 --- Test cost:0.03231
0.3-0.2 run 0 - epoch 520/700 --- Train cost:0.01787 --- Test cost:0.03141
0.3-0.2 run 0 - epoch 525/700 --- Train cost:0.01720 --- Test cost:0.03014
0.3-0.2 run 0 - epoch 530/700 --- Train cost:0.01619 --- Test cost:0.03057
0.3-0.2 run 0 - epoch 535/700 --- Train cost:0.01601 --- Test cost:0.03083
0.3-0.2 run 0 - epoch 540/700 --- Train cost:0.01612 --- Test cost:0.03070
0.3-0.2 run 0 - epoch 545/700 --- Train cost:0.01574 --- Test cost:0.03290
0.3-0.2 run 0 - epoch 550/700 --- Train cost:0.01625 --- Test cost:0.03325
0.3-0.2 run 0 - epoch 555/700 --- Train cost:0.01698 --- Test cost:0.03418
0.3-0.2 run 0 - epoch 560/700 --- Train cost:0.01668 --- Test cost:0.03583
0.3-0.2 run 0 - epoch 565/700 --- Train cost:0.01682 --- Test cost:0.03725
0.3-0.2 run 0 - epoch 570/700 --- Train cost:0.01708 --- Test cost:0.03564
0.3-0.2 run 0 - epoch 575/700 --- Train cost:0.01755 --- Test cost:0.03299
0.3-0.2 run 0 - epoch 580/700 --- Train cost:0.01787 --- Test cost:0.03198
0.3-0.2 run 0 - epoch 585/700 --- Train cost:0.01694 --- Test cost:0.03235
0.3-0.2 run 0 - epoch 590/700 --- Train cost:0.01641 --- Test cost:0.03314
0.3-0.2 run 0 - epoch 595/700 --- Train cost:0.01679 --- Test cost:0.03300
0.3-0.2 run 0 - epoch 600/700 --- Train cost:0.01819 --- Test cost:0.03303
0.3-0.2 run 0 - epoch 605/700 --- Train cost:0.01947 --- Test cost:0.03337
0.3-0.2 run 0 - epoch 610/700 --- Train cost:0.01828 --- Test cost:0.03462
0.3-0.2 run 0 - epoch 615/700 --- Train cost:0.01640 --- Test cost:0.03760
0.3-0.2 run 0 - epoch 620/700 --- Train cost:0.01539 --- Test cost:0.03958
0.3-0.2 run 0 - epoch 625/700 --- Train cost:0.01417 --- Test cost:0.04136
0.3-0.2 run 0 - epoch 630/700 --- Train cost:0.01333 --- Test cost:0.04329
0.3-0.2 run 0 - epoch 635/700 --- Train cost:0.01219 --- Test cost:0.04554
0.3-0.2 run 0 - epoch 640/700 --- Train cost:0.01101 --- Test cost:0.04733
0.3-0.2 run 0 - epoch 645/700 --- Train cost:0.01051 --- Test cost:0.04716
0.3-0.2 run 0 - epoch 650/700 --- Train cost:0.01033 --- Test cost:0.04391
0.3-0.2 run 0 - epoch 655/700 --- Train cost:0.01065 --- Test cost:0.04019
0.3-0.2 run 0 - epoch 660/700 --- Train cost:0.01158 --- Test cost:0.03914
0.3-0.2 run 0 - epoch 665/700 --- Train cost:0.01218 --- Test cost:0.04120
0.3-0.2 run 0 - epoch 670/700 --- Train cost:0.01252 --- Test cost:0.04520
0.3-0.2 run 0 - epoch 675/700 --- Train cost:0.01322 --- Test cost:0.04921
0.3-0.2 run 0 - epoch 680/700 --- Train cost:0.01317 --- Test cost:0.05070
0.3-0.2 run 0 - epoch 685/700 --- Train cost:0.01265 --- Test cost:0.05084
0.3-0.2 run 0 - epoch 690/700 --- Train cost:0.01196 --- Test cost:0.05020
0.3-0.2 run 0 - epoch 695/700 --- Train cost:0.01165 --- Test cost:0.04836
0.3-0.2 run 1 - epoch 0/700 --- Train cost:0.29753 --- Test cost:0.10869
0.3-0.2 run 1 - epoch 5/700 --- Train cost:0.26330 --- Test cost:0.08804
0.3-0.2 run 1 - epoch 10/700 --- Train cost:0.21336 --- Test cost:0.07231
0.3-0.2 run 1 - epoch 15/700 --- Train cost:0.17060 --- Test cost:0.05948
0.3-0.2 run 1 - epoch 20/700 --- Train cost:0.12846 --- Test cost:0.05353
0.3-0.2 run 1 - epoch 25/700 --- Train cost:0.09453 --- Test cost:0.05322
0.3-0.2 run 1 - epoch 30/700 --- Train cost:0.07272 --- Test cost:0.05897
0.3-0.2 run 1 - epoch 35/700 --- Train cost:0.06275 --- Test cost:0.06653
0.3-0.2 run 1 - epoch 40/700 --- Train cost:0.05683 --- Test cost:0.07305
0.3-0.2 run 1 - epoch 45/700 --- Train cost:0.04927 --- Test cost:0.07256
0.3-0.2 run 1 - epoch 50/700 --- Train cost:0.03885 --- Test cost:0.06714
0.3-0.2 run 1 - epoch 55/700 --- Train cost:0.03106 --- Test cost:0.06172
0.3-0.2 run 1 - epoch 60/700 --- Train cost:0.02770 --- Test cost:0.05626
0.3-0.2 run 1 - epoch 65/700 --- Train cost:0.02680 --- Test cost:0.05032
0.3-0.2 run 1 - epoch 70/700 --- Train cost:0.02594 --- Test cost:0.04432
0.3-0.2 run 1 - epoch 75/700 --- Train cost:0.02597 --- Test cost:0.04153
0.3-0.2 run 1 - epoch 80/700 --- Train cost:0.02653 --- Test cost:0.04166
0.3-0.2 run 1 - epoch 85/700 --- Train cost:0.02755 --- Test cost:0.04234
0.3-0.2 run 1 - epoch 90/700 --- Train cost:0.02812 --- Test cost:0.04202
0.3-0.2 run 1 - epoch 95/700 --- Train cost:0.02861 --- Test cost:0.04093
0.3-0.2 run 1 - epoch 100/700 --- Train cost:0.02843 --- Test cost:0.04032
0.3-0.2 run 1 - epoch 105/700 --- Train cost:0.02706 --- Test cost:0.04262
0.3-0.2 run 1 - epoch 110/700 --- Train cost:0.02685 --- Test cost:0.04394
0.3-0.2 run 1 - epoch 115/700 --- Train cost:0.02726 --- Test cost:0.04233
0.3-0.2 run 1 - epoch 120/700 --- Train cost:0.02671 --- Test cost:0.04016
0.3-0.2 run 1 - epoch 125/700 --- Train cost:0.02577 --- Test cost:0.04100
0.3-0.2 run 1 - epoch 130/700 --- Train cost:0.02352 --- Test cost:0.04145
0.3-0.2 run 1 - epoch 135/700 --- Train cost:0.02132 --- Test cost:0.04209
0.3-0.2 run 1 - epoch 140/700 --- Train cost:0.01952 --- Test cost:0.04329
0.3-0.2 run 1 - epoch 145/700 --- Train cost:0.01841 --- Test cost:0.04427
0.3-0.2 run 1 - epoch 150/700 --- Train cost:0.01800 --- Test cost:0.04643
0.3-0.2 run 1 - epoch 155/700 --- Train cost:0.01809 --- Test cost:0.04260
0.3-0.2 run 1 - epoch 160/700 --- Train cost:0.01875 --- Test cost:0.03847
0.3-0.2 run 1 - epoch 165/700 --- Train cost:0.01914 --- Test cost:0.03827
0.3-0.2 run 1 - epoch 170/700 --- Train cost:0.01847 --- Test cost:0.03662
0.3-0.2 run 1 - epoch 175/700 --- Train cost:0.01780 --- Test cost:0.03229
0.3-0.2 run 1 - epoch 180/700 --- Train cost:0.01785 --- Test cost:0.02940
0.3-0.2 run 1 - epoch 185/700 --- Train cost:0.01835 --- Test cost:0.02859
0.3-0.2 run 1 - epoch 190/700 --- Train cost:0.01878 --- Test cost:0.02969
0.3-0.2 run 1 - epoch 195/700 --- Train cost:0.02021 --- Test cost:0.02947
0.3-0.2 run 1 - epoch 200/700 --- Train cost:0.02121 --- Test cost:0.02781
0.3-0.2 run 1 - epoch 205/700 --- Train cost:0.02254 --- Test cost:0.02726
0.3-0.2 run 1 - epoch 210/700 --- Train cost:0.02484 --- Test cost:0.02858
0.3-0.2 run 1 - epoch 215/700 --- Train cost:0.02534 --- Test cost:0.02795
0.3-0.2 run 1 - epoch 220/700 --- Train cost:0.02506 --- Test cost:0.02560
0.3-0.2 run 1 - epoch 225/700 --- Train cost:0.02613 --- Test cost:0.02338
0.3-0.2 run 1 - epoch 230/700 --- Train cost:0.02696 --- Test cost:0.02419
0.3-0.2 run 1 - epoch 235/700 --- Train cost:0.02761 --- Test cost:0.02587
0.3-0.2 run 1 - epoch 240/700 --- Train cost:0.02758 --- Test cost:0.02627
0.3-0.2 run 1 - epoch 245/700 --- Train cost:0.02569 --- Test cost:0.02712
0.3-0.2 run 1 - epoch 250/700 --- Train cost:0.02528 --- Test cost:0.02652
0.3-0.2 run 1 - epoch 255/700 --- Train cost:0.02580 --- Test cost:0.02681
0.3-0.2 run 1 - epoch 260/700 --- Train cost:0.02652 --- Test cost:0.02926
0.3-0.2 run 1 - epoch 265/700 --- Train cost:0.02866 --- Test cost:0.03026
0.3-0.2 run 1 - epoch 270/700 --- Train cost:0.03103 --- Test cost:0.02837
0.3-0.2 run 1 - epoch 275/700 --- Train cost:0.03371 --- Test cost:0.02630
0.3-0.2 run 1 - epoch 280/700 --- Train cost:0.03535 --- Test cost:0.02468
0.3-0.2 run 1 - epoch 285/700 --- Train cost:0.03386 --- Test cost:0.02466
0.3-0.2 run 1 - epoch 290/700 --- Train cost:0.02953 --- Test cost:0.02733
0.3-0.2 run 1 - epoch 295/700 --- Train cost:0.02535 --- Test cost:0.03228
0.3-0.2 run 1 - epoch 300/700 --- Train cost:0.02308 --- Test cost:0.03782
0.3-0.2 run 1 - epoch 305/700 --- Train cost:0.02196 --- Test cost:0.04193
0.3-0.2 run 1 - epoch 310/700 --- Train cost:0.02277 --- Test cost:0.03882
0.3-0.2 run 1 - epoch 315/700 --- Train cost:0.02445 --- Test cost:0.03420
0.3-0.2 run 1 - epoch 320/700 --- Train cost:0.02630 --- Test cost:0.03248
0.3-0.2 run 1 - epoch 325/700 --- Train cost:0.02795 --- Test cost:0.03070
0.3-0.2 run 1 - epoch 330/700 --- Train cost:0.02930 --- Test cost:0.03133
0.3-0.2 run 1 - epoch 335/700 --- Train cost:0.02918 --- Test cost:0.03339
0.3-0.2 run 1 - epoch 340/700 --- Train cost:0.02873 --- Test cost:0.03402
0.3-0.2 run 1 - epoch 345/700 --- Train cost:0.02699 --- Test cost:0.03452
0.3-0.2 run 1 - epoch 350/700 --- Train cost:0.02491 --- Test cost:0.03589
0.3-0.2 run 1 - epoch 355/700 --- Train cost:0.02182 --- Test cost:0.04081
0.3-0.2 run 1 - epoch 360/700 --- Train cost:0.01841 --- Test cost:0.04855
0.3-0.2 run 1 - epoch 365/700 --- Train cost:0.01629 --- Test cost:0.05739
0.3-0.2 run 1 - epoch 370/700 --- Train cost:0.01607 --- Test cost:0.05921
0.3-0.2 run 1 - epoch 375/700 --- Train cost:0.01552 --- Test cost:0.05969
0.3-0.2 run 1 - epoch 380/700 --- Train cost:0.01467 --- Test cost:0.06039
0.3-0.2 run 1 - epoch 385/700 --- Train cost:0.01440 --- Test cost:0.05962
0.3-0.2 run 1 - epoch 390/700 --- Train cost:0.01477 --- Test cost:0.05665
0.3-0.2 run 1 - epoch 395/700 --- Train cost:0.01496 --- Test cost:0.05206
0.3-0.2 run 1 - epoch 400/700 --- Train cost:0.01515 --- Test cost:0.04723
0.3-0.2 run 1 - epoch 405/700 --- Train cost:0.01550 --- Test cost:0.04108
0.3-0.2 run 1 - epoch 410/700 --- Train cost:0.01594 --- Test cost:0.03604
0.3-0.2 run 1 - epoch 415/700 --- Train cost:0.01745 --- Test cost:0.03178
0.3-0.2 run 1 - epoch 420/700 --- Train cost:0.01907 --- Test cost:0.03074
0.3-0.2 run 1 - epoch 425/700 --- Train cost:0.02072 --- Test cost:0.03100
0.3-0.2 run 1 - epoch 430/700 --- Train cost:0.02258 --- Test cost:0.02925
0.3-0.2 run 1 - epoch 435/700 --- Train cost:0.02556 --- Test cost:0.02682
0.3-0.2 run 1 - epoch 440/700 --- Train cost:0.02780 --- Test cost:0.02608
0.3-0.2 run 1 - epoch 445/700 --- Train cost:0.02890 --- Test cost:0.02605
0.3-0.2 run 1 - epoch 450/700 --- Train cost:0.02778 --- Test cost:0.02578
0.3-0.2 run 1 - epoch 455/700 --- Train cost:0.02589 --- Test cost:0.02465
0.3-0.2 run 1 - epoch 460/700 --- Train cost:0.02313 --- Test cost:0.02453
0.3-0.2 run 1 - epoch 465/700 --- Train cost:0.02230 --- Test cost:0.02470
0.3-0.2 run 1 - epoch 470/700 --- Train cost:0.02221 --- Test cost:0.02607
0.3-0.2 run 1 - epoch 475/700 --- Train cost:0.02244 --- Test cost:0.02747
0.3-0.2 run 1 - epoch 480/700 --- Train cost:0.02321 --- Test cost:0.02905
0.3-0.2 run 1 - epoch 485/700 --- Train cost:0.02427 --- Test cost:0.03002
0.3-0.2 run 1 - epoch 490/700 --- Train cost:0.02487 --- Test cost:0.03213
0.3-0.2 run 1 - epoch 495/700 --- Train cost:0.02497 --- Test cost:0.03094
0.3-0.2 run 1 - epoch 500/700 --- Train cost:0.02365 --- Test cost:0.03063
0.3-0.2 run 1 - epoch 505/700 --- Train cost:0.02153 --- Test cost:0.02974
0.3-0.2 run 1 - epoch 510/700 --- Train cost:0.02013 --- Test cost:0.02918
0.3-0.2 run 1 - epoch 515/700 --- Train cost:0.01988 --- Test cost:0.02849
0.3-0.2 run 1 - epoch 520/700 --- Train cost:0.02023 --- Test cost:0.02809
0.3-0.2 run 1 - epoch 525/700 --- Train cost:0.02155 --- Test cost:0.02827
0.3-0.2 run 1 - epoch 530/700 --- Train cost:0.02203 --- Test cost:0.02946
0.3-0.2 run 1 - epoch 535/700 --- Train cost:0.02275 --- Test cost:0.02949
0.3-0.2 run 1 - epoch 540/700 --- Train cost:0.02294 --- Test cost:0.02962
0.3-0.2 run 1 - epoch 545/700 --- Train cost:0.02214 --- Test cost:0.02953
0.3-0.2 run 1 - epoch 550/700 --- Train cost:0.01995 --- Test cost:0.03036
0.3-0.2 run 1 - epoch 555/700 --- Train cost:0.01818 --- Test cost:0.03044
0.3-0.2 run 1 - epoch 560/700 --- Train cost:0.01787 --- Test cost:0.02972
0.3-0.2 run 1 - epoch 565/700 --- Train cost:0.01949 --- Test cost:0.03088
0.3-0.2 run 1 - epoch 570/700 --- Train cost:0.01991 --- Test cost:0.03128
0.3-0.2 run 1 - epoch 575/700 --- Train cost:0.02000 --- Test cost:0.03229
0.3-0.2 run 1 - epoch 580/700 --- Train cost:0.02012 --- Test cost:0.03475
0.3-0.2 run 1 - epoch 585/700 --- Train cost:0.02029 --- Test cost:0.03853
0.3-0.2 run 1 - epoch 590/700 --- Train cost:0.02044 --- Test cost:0.04113
0.3-0.2 run 1 - epoch 595/700 --- Train cost:0.02109 --- Test cost:0.04275
0.3-0.2 run 1 - epoch 600/700 --- Train cost:0.02163 --- Test cost:0.04315
0.3-0.2 run 1 - epoch 605/700 --- Train cost:0.02199 --- Test cost:0.04196
0.3-0.2 run 1 - epoch 610/700 --- Train cost:0.02327 --- Test cost:0.03672
0.3-0.2 run 1 - epoch 615/700 --- Train cost:0.02422 --- Test cost:0.03375
0.3-0.2 run 1 - epoch 620/700 --- Train cost:0.02461 --- Test cost:0.03259
0.3-0.2 run 1 - epoch 625/700 --- Train cost:0.02398 --- Test cost:0.03313
0.3-0.2 run 1 - epoch 630/700 --- Train cost:0.02345 --- Test cost:0.03497
0.3-0.2 run 1 - epoch 635/700 --- Train cost:0.02421 --- Test cost:0.03511
0.3-0.2 run 1 - epoch 640/700 --- Train cost:0.02565 --- Test cost:0.03593
0.3-0.2 run 1 - epoch 645/700 --- Train cost:0.02613 --- Test cost:0.03628
0.3-0.2 run 1 - epoch 650/700 --- Train cost:0.02579 --- Test cost:0.03844
0.3-0.2 run 1 - epoch 655/700 --- Train cost:0.02616 --- Test cost:0.04036
0.3-0.2 run 1 - epoch 660/700 --- Train cost:0.02684 --- Test cost:0.03859
0.3-0.2 run 1 - epoch 665/700 --- Train cost:0.02739 --- Test cost:0.03761
0.3-0.2 run 1 - epoch 670/700 --- Train cost:0.02756 --- Test cost:0.03633
0.3-0.2 run 1 - epoch 675/700 --- Train cost:0.02639 --- Test cost:0.03506
0.3-0.2 run 1 - epoch 680/700 --- Train cost:0.02516 --- Test cost:0.03229
0.3-0.2 run 1 - epoch 685/700 --- Train cost:0.02443 --- Test cost:0.02950
0.3-0.2 run 1 - epoch 690/700 --- Train cost:0.02328 --- Test cost:0.02673
0.3-0.2 run 1 - epoch 695/700 --- Train cost:0.02258 --- Test cost:0.02543
0.3-0.2 run 2 - epoch 0/700 --- Train cost:0.34278 --- Test cost:0.15919
0.3-0.2 run 2 - epoch 5/700 --- Train cost:0.28132 --- Test cost:0.13218
0.3-0.2 run 2 - epoch 10/700 --- Train cost:0.23132 --- Test cost:0.11190
0.3-0.2 run 2 - epoch 15/700 --- Train cost:0.20663 --- Test cost:0.10601
0.3-0.2 run 2 - epoch 20/700 --- Train cost:0.17510 --- Test cost:0.09089
0.3-0.2 run 2 - epoch 25/700 --- Train cost:0.14803 --- Test cost:0.07539
0.3-0.2 run 2 - epoch 30/700 --- Train cost:0.13183 --- Test cost:0.06781
0.3-0.2 run 2 - epoch 35/700 --- Train cost:0.12741 --- Test cost:0.06498
0.3-0.2 run 2 - epoch 40/700 --- Train cost:0.13464 --- Test cost:0.07032
0.3-0.2 run 2 - epoch 45/700 --- Train cost:0.13521 --- Test cost:0.07230
0.3-0.2 run 2 - epoch 50/700 --- Train cost:0.11684 --- Test cost:0.06300
0.3-0.2 run 2 - epoch 55/700 --- Train cost:0.10356 --- Test cost:0.05661
0.3-0.2 run 2 - epoch 60/700 --- Train cost:0.10000 --- Test cost:0.05824
0.3-0.2 run 2 - epoch 65/700 --- Train cost:0.09885 --- Test cost:0.06258
0.3-0.2 run 2 - epoch 70/700 --- Train cost:0.09525 --- Test cost:0.06176
0.3-0.2 run 2 - epoch 75/700 --- Train cost:0.08257 --- Test cost:0.05414
0.3-0.2 run 2 - epoch 80/700 --- Train cost:0.06751 --- Test cost:0.04591
0.3-0.2 run 2 - epoch 85/700 --- Train cost:0.06228 --- Test cost:0.04132
0.3-0.2 run 2 - epoch 90/700 --- Train cost:0.06065 --- Test cost:0.04068
0.3-0.2 run 2 - epoch 95/700 --- Train cost:0.06269 --- Test cost:0.04089
0.3-0.2 run 2 - epoch 100/700 --- Train cost:0.06198 --- Test cost:0.03901
0.3-0.2 run 2 - epoch 105/700 --- Train cost:0.05928 --- Test cost:0.03481
0.3-0.2 run 2 - epoch 110/700 --- Train cost:0.05577 --- Test cost:0.03087
0.3-0.2 run 2 - epoch 115/700 --- Train cost:0.05254 --- Test cost:0.02750
0.3-0.2 run 2 - epoch 120/700 --- Train cost:0.04757 --- Test cost:0.02674
0.3-0.2 run 2 - epoch 125/700 --- Train cost:0.04103 --- Test cost:0.02500
0.3-0.2 run 2 - epoch 130/700 --- Train cost:0.03974 --- Test cost:0.02562
0.3-0.2 run 2 - epoch 135/700 --- Train cost:0.03892 --- Test cost:0.02729
0.3-0.2 run 2 - epoch 140/700 --- Train cost:0.03626 --- Test cost:0.02841
0.3-0.2 run 2 - epoch 145/700 --- Train cost:0.03162 --- Test cost:0.02885
0.3-0.2 run 2 - epoch 150/700 --- Train cost:0.02799 --- Test cost:0.02885
0.3-0.2 run 2 - epoch 155/700 --- Train cost:0.02635 --- Test cost:0.02905
0.3-0.2 run 2 - epoch 160/700 --- Train cost:0.02718 --- Test cost:0.02897
0.3-0.2 run 2 - epoch 165/700 --- Train cost:0.02886 --- Test cost:0.02912
0.3-0.2 run 2 - epoch 170/700 --- Train cost:0.02706 --- Test cost:0.03066
0.3-0.2 run 2 - epoch 175/700 --- Train cost:0.02518 --- Test cost:0.03333
0.3-0.2 run 2 - epoch 180/700 --- Train cost:0.02473 --- Test cost:0.03470
0.3-0.2 run 2 - epoch 185/700 --- Train cost:0.02495 --- Test cost:0.03578
0.3-0.2 run 2 - epoch 190/700 --- Train cost:0.02559 --- Test cost:0.03443
0.3-0.2 run 2 - epoch 195/700 --- Train cost:0.02744 --- Test cost:0.03276
0.3-0.2 run 2 - epoch 200/700 --- Train cost:0.02844 --- Test cost:0.03364
0.3-0.2 run 2 - epoch 205/700 --- Train cost:0.02869 --- Test cost:0.03450
0.3-0.2 run 2 - epoch 210/700 --- Train cost:0.02955 --- Test cost:0.03535
0.3-0.2 run 2 - epoch 215/700 --- Train cost:0.03238 --- Test cost:0.03724
0.3-0.2 run 2 - epoch 220/700 --- Train cost:0.03412 --- Test cost:0.03921
0.3-0.2 run 2 - epoch 225/700 --- Train cost:0.03353 --- Test cost:0.03855
0.3-0.2 run 2 - epoch 230/700 --- Train cost:0.03182 --- Test cost:0.03540
0.3-0.2 run 2 - epoch 235/700 --- Train cost:0.03060 --- Test cost:0.03439
0.3-0.2 run 2 - epoch 240/700 --- Train cost:0.03054 --- Test cost:0.03452
0.3-0.2 run 2 - epoch 245/700 --- Train cost:0.03235 --- Test cost:0.03604
0.3-0.2 run 2 - epoch 250/700 --- Train cost:0.03219 --- Test cost:0.03593
0.3-0.2 run 2 - epoch 255/700 --- Train cost:0.03195 --- Test cost:0.03635
0.3-0.2 run 2 - epoch 260/700 --- Train cost:0.03218 --- Test cost:0.03722
0.3-0.2 run 2 - epoch 265/700 --- Train cost:0.03181 --- Test cost:0.03760
0.3-0.2 run 2 - epoch 270/700 --- Train cost:0.03112 --- Test cost:0.03641
0.3-0.2 run 2 - epoch 275/700 --- Train cost:0.03101 --- Test cost:0.03436
0.3-0.2 run 2 - epoch 280/700 --- Train cost:0.03019 --- Test cost:0.03358
0.3-0.2 run 2 - epoch 285/700 --- Train cost:0.02781 --- Test cost:0.03371
0.3-0.2 run 2 - epoch 290/700 --- Train cost:0.02519 --- Test cost:0.03568
0.3-0.2 run 2 - epoch 295/700 --- Train cost:0.02357 --- Test cost:0.03853
0.3-0.2 run 2 - epoch 300/700 --- Train cost:0.02374 --- Test cost:0.03838
0.3-0.2 run 2 - epoch 305/700 --- Train cost:0.02319 --- Test cost:0.03819
0.3-0.2 run 2 - epoch 310/700 --- Train cost:0.02267 --- Test cost:0.03689
0.3-0.2 run 2 - epoch 315/700 --- Train cost:0.02285 --- Test cost:0.03611
0.3-0.2 run 2 - epoch 320/700 --- Train cost:0.02284 --- Test cost:0.03616
0.3-0.2 run 2 - epoch 325/700 --- Train cost:0.02083 --- Test cost:0.03639
0.3-0.2 run 2 - epoch 330/700 --- Train cost:0.01859 --- Test cost:0.03534
0.3-0.2 run 2 - epoch 335/700 --- Train cost:0.01835 --- Test cost:0.03335
0.3-0.2 run 2 - epoch 340/700 --- Train cost:0.01874 --- Test cost:0.03133
0.3-0.2 run 2 - epoch 345/700 --- Train cost:0.01900 --- Test cost:0.02970
0.3-0.2 run 2 - epoch 350/700 --- Train cost:0.01883 --- Test cost:0.02837
0.3-0.2 run 2 - epoch 355/700 --- Train cost:0.01890 --- Test cost:0.02819
0.3-0.2 run 2 - epoch 360/700 --- Train cost:0.01858 --- Test cost:0.02929
0.3-0.2 run 2 - epoch 365/700 --- Train cost:0.01810 --- Test cost:0.03308
0.3-0.2 run 2 - epoch 370/700 --- Train cost:0.01799 --- Test cost:0.03617
0.3-0.2 run 2 - epoch 375/700 --- Train cost:0.01923 --- Test cost:0.03711
0.3-0.2 run 2 - epoch 380/700 --- Train cost:0.01992 --- Test cost:0.04044
0.3-0.2 run 2 - epoch 385/700 --- Train cost:0.01997 --- Test cost:0.04418
0.3-0.2 run 2 - epoch 390/700 --- Train cost:0.02058 --- Test cost:0.04511
0.3-0.2 run 2 - epoch 395/700 --- Train cost:0.02072 --- Test cost:0.04551
0.3-0.2 run 2 - epoch 400/700 --- Train cost:0.01957 --- Test cost:0.04624
0.3-0.2 run 2 - epoch 405/700 --- Train cost:0.01749 --- Test cost:0.04656
0.3-0.2 run 2 - epoch 410/700 --- Train cost:0.01553 --- Test cost:0.04521
0.3-0.2 run 2 - epoch 415/700 --- Train cost:0.01437 --- Test cost:0.04518
0.3-0.2 run 2 - epoch 420/700 --- Train cost:0.01342 --- Test cost:0.04605
0.3-0.2 run 2 - epoch 425/700 --- Train cost:0.01295 --- Test cost:0.04637
0.3-0.2 run 2 - epoch 430/700 --- Train cost:0.01348 --- Test cost:0.04360
0.3-0.2 run 2 - epoch 435/700 --- Train cost:0.01388 --- Test cost:0.03718
0.3-0.2 run 2 - epoch 440/700 --- Train cost:0.01462 --- Test cost:0.03346
0.3-0.2 run 2 - epoch 445/700 --- Train cost:0.01739 --- Test cost:0.03054
0.3-0.2 run 2 - epoch 450/700 --- Train cost:0.01838 --- Test cost:0.03134
0.3-0.2 run 2 - epoch 455/700 --- Train cost:0.02008 --- Test cost:0.03287
0.3-0.2 run 2 - epoch 460/700 --- Train cost:0.02103 --- Test cost:0.03486
0.3-0.2 run 2 - epoch 465/700 --- Train cost:0.02161 --- Test cost:0.03653
0.3-0.2 run 2 - epoch 470/700 --- Train cost:0.02231 --- Test cost:0.03592
0.3-0.2 run 2 - epoch 475/700 --- Train cost:0.02068 --- Test cost:0.03673
0.3-0.2 run 2 - epoch 480/700 --- Train cost:0.01813 --- Test cost:0.03697
0.3-0.2 run 2 - epoch 485/700 --- Train cost:0.01654 --- Test cost:0.03794
0.3-0.2 run 2 - epoch 490/700 --- Train cost:0.01731 --- Test cost:0.03931
0.3-0.2 run 2 - epoch 495/700 --- Train cost:0.01767 --- Test cost:0.03878
0.3-0.2 run 2 - epoch 500/700 --- Train cost:0.01837 --- Test cost:0.04062
0.3-0.2 run 2 - epoch 505/700 --- Train cost:0.02030 --- Test cost:0.04385
0.3-0.2 run 2 - epoch 510/700 --- Train cost:0.02215 --- Test cost:0.04440
0.3-0.2 run 2 - epoch 515/700 --- Train cost:0.02235 --- Test cost:0.04182
0.3-0.2 run 2 - epoch 520/700 --- Train cost:0.02186 --- Test cost:0.03839
0.3-0.2 run 2 - epoch 525/700 --- Train cost:0.02151 --- Test cost:0.03492
0.3-0.2 run 2 - epoch 530/700 --- Train cost:0.02122 --- Test cost:0.03194
0.3-0.2 run 2 - epoch 535/700 --- Train cost:0.02084 --- Test cost:0.03025
0.3-0.2 run 2 - epoch 540/700 --- Train cost:0.02012 --- Test cost:0.02985
0.3-0.2 run 2 - epoch 545/700 --- Train cost:0.02005 --- Test cost:0.03087
0.3-0.2 run 2 - epoch 550/700 --- Train cost:0.01955 --- Test cost:0.03097
0.3-0.2 run 2 - epoch 555/700 --- Train cost:0.01863 --- Test cost:0.03180
0.3-0.2 run 2 - epoch 560/700 --- Train cost:0.01808 --- Test cost:0.03208
0.3-0.2 run 2 - epoch 565/700 --- Train cost:0.01784 --- Test cost:0.03127
0.3-0.2 run 2 - epoch 570/700 --- Train cost:0.01751 --- Test cost:0.02968
0.3-0.2 run 2 - epoch 575/700 --- Train cost:0.01743 --- Test cost:0.02819
0.3-0.2 run 2 - epoch 580/700 --- Train cost:0.01767 --- Test cost:0.02914
0.3-0.2 run 2 - epoch 585/700 --- Train cost:0.01847 --- Test cost:0.03020
0.3-0.2 run 2 - epoch 590/700 --- Train cost:0.01937 --- Test cost:0.03271
0.3-0.2 run 2 - epoch 595/700 --- Train cost:0.01993 --- Test cost:0.03561
0.3-0.2 run 2 - epoch 600/700 --- Train cost:0.02050 --- Test cost:0.03822
0.3-0.2 run 2 - epoch 605/700 --- Train cost:0.02168 --- Test cost:0.03987
0.3-0.2 run 2 - epoch 610/700 --- Train cost:0.02587 --- Test cost:0.03998
0.3-0.2 run 2 - epoch 615/700 --- Train cost:0.02779 --- Test cost:0.04133
0.3-0.2 run 2 - epoch 620/700 --- Train cost:0.02519 --- Test cost:0.04356
0.3-0.2 run 2 - epoch 625/700 --- Train cost:0.02274 --- Test cost:0.04118
0.3-0.2 run 2 - epoch 630/700 --- Train cost:0.02200 --- Test cost:0.04114
0.3-0.2 run 2 - epoch 635/700 --- Train cost:0.02261 --- Test cost:0.04003
0.3-0.2 run 2 - epoch 640/700 --- Train cost:0.02230 --- Test cost:0.03769
0.3-0.2 run 2 - epoch 645/700 --- Train cost:0.02251 --- Test cost:0.03571
0.3-0.2 run 2 - epoch 650/700 --- Train cost:0.02192 --- Test cost:0.03764
0.3-0.2 run 2 - epoch 655/700 --- Train cost:0.02261 --- Test cost:0.03894
0.3-0.2 run 2 - epoch 660/700 --- Train cost:0.02199 --- Test cost:0.03986
0.3-0.2 run 2 - epoch 665/700 --- Train cost:0.02149 --- Test cost:0.04118
0.3-0.2 run 2 - epoch 670/700 --- Train cost:0.02190 --- Test cost:0.04345
0.3-0.2 run 2 - epoch 675/700 --- Train cost:0.02264 --- Test cost:0.04714
0.3-0.2 run 2 - epoch 680/700 --- Train cost:0.02259 --- Test cost:0.05051
0.3-0.2 run 2 - epoch 685/700 --- Train cost:0.02147 --- Test cost:0.05481
0.3-0.2 run 2 - epoch 690/700 --- Train cost:0.01968 --- Test cost:0.05929
0.3-0.2 run 2 - epoch 695/700 --- Train cost:0.01821 --- Test cost:0.06038
0.7-0.7 run 0 - epoch 0/700 --- Train cost:0.48694 --- Test cost:0.18159
0.7-0.7 run 0 - epoch 5/700 --- Train cost:0.45133 --- Test cost:0.18601
0.7-0.7 run 0 - epoch 10/700 --- Train cost:0.44266 --- Test cost:0.18347
0.7-0.7 run 0 - epoch 15/700 --- Train cost:0.43543 --- Test cost:0.18362
0.7-0.7 run 0 - epoch 20/700 --- Train cost:0.42925 --- Test cost:0.18564
0.7-0.7 run 0 - epoch 25/700 --- Train cost:0.42731 --- Test cost:0.19032
0.7-0.7 run 0 - epoch 30/700 --- Train cost:0.42100 --- Test cost:0.19420
0.7-0.7 run 0 - epoch 35/700 --- Train cost:0.41877 --- Test cost:0.19420
0.7-0.7 run 0 - epoch 40/700 --- Train cost:0.41817 --- Test cost:0.20158
0.7-0.7 run 0 - epoch 45/700 --- Train cost:0.42655 --- Test cost:0.21943
0.7-0.7 run 0 - epoch 50/700 --- Train cost:0.43367 --- Test cost:0.23445
0.7-0.7 run 0 - epoch 55/700 --- Train cost:0.44196 --- Test cost:0.24297
0.7-0.7 run 0 - epoch 60/700 --- Train cost:0.44581 --- Test cost:0.24633
0.7-0.7 run 0 - epoch 65/700 --- Train cost:0.44184 --- Test cost:0.24406
0.7-0.7 run 0 - epoch 70/700 --- Train cost:0.44265 --- Test cost:0.24380
0.7-0.7 run 0 - epoch 75/700 --- Train cost:0.44070 --- Test cost:0.24793
0.7-0.7 run 0 - epoch 80/700 --- Train cost:0.44367 --- Test cost:0.25030
0.7-0.7 run 0 - epoch 85/700 --- Train cost:0.45098 --- Test cost:0.25671
0.7-0.7 run 0 - epoch 90/700 --- Train cost:0.46509 --- Test cost:0.27224
0.7-0.7 run 0 - epoch 95/700 --- Train cost:0.46767 --- Test cost:0.27762
0.7-0.7 run 0 - epoch 100/700 --- Train cost:0.46680 --- Test cost:0.27358
0.7-0.7 run 0 - epoch 105/700 --- Train cost:0.46435 --- Test cost:0.26891
0.7-0.7 run 0 - epoch 110/700 --- Train cost:0.46345 --- Test cost:0.27216
0.7-0.7 run 0 - epoch 115/700 --- Train cost:0.46815 --- Test cost:0.27692
0.7-0.7 run 0 - epoch 120/700 --- Train cost:0.46472 --- Test cost:0.26916
0.7-0.7 run 0 - epoch 125/700 --- Train cost:0.45106 --- Test cost:0.26434
0.7-0.7 run 0 - epoch 130/700 --- Train cost:0.44020 --- Test cost:0.26362
0.7-0.7 run 0 - epoch 135/700 --- Train cost:0.43706 --- Test cost:0.26030
0.7-0.7 run 0 - epoch 140/700 --- Train cost:0.43919 --- Test cost:0.26062
0.7-0.7 run 0 - epoch 145/700 --- Train cost:0.43811 --- Test cost:0.25650
0.7-0.7 run 0 - epoch 150/700 --- Train cost:0.43700 --- Test cost:0.26488
0.7-0.7 run 0 - epoch 155/700 --- Train cost:0.43901 --- Test cost:0.27257
0.7-0.7 run 0 - epoch 160/700 --- Train cost:0.44622 --- Test cost:0.28039
0.7-0.7 run 0 - epoch 165/700 --- Train cost:0.45288 --- Test cost:0.29006
0.7-0.7 run 0 - epoch 170/700 --- Train cost:0.45550 --- Test cost:0.29721
0.7-0.7 run 0 - epoch 175/700 --- Train cost:0.46679 --- Test cost:0.30513
0.7-0.7 run 0 - epoch 180/700 --- Train cost:0.47493 --- Test cost:0.31189
0.7-0.7 run 0 - epoch 185/700 --- Train cost:0.47523 --- Test cost:0.31563
0.7-0.7 run 0 - epoch 190/700 --- Train cost:0.48710 --- Test cost:0.32458
0.7-0.7 run 0 - epoch 195/700 --- Train cost:0.49693 --- Test cost:0.33346
0.7-0.7 run 0 - epoch 200/700 --- Train cost:0.49587 --- Test cost:0.32419
0.7-0.7 run 0 - epoch 205/700 --- Train cost:0.49408 --- Test cost:0.31387
0.7-0.7 run 0 - epoch 210/700 --- Train cost:0.49738 --- Test cost:0.31817
0.7-0.7 run 0 - epoch 215/700 --- Train cost:0.48917 --- Test cost:0.31511
0.7-0.7 run 0 - epoch 220/700 --- Train cost:0.47885 --- Test cost:0.30822
0.7-0.7 run 0 - epoch 225/700 --- Train cost:0.46831 --- Test cost:0.30017
0.7-0.7 run 0 - epoch 230/700 --- Train cost:0.46747 --- Test cost:0.29509
0.7-0.7 run 0 - epoch 235/700 --- Train cost:0.46623 --- Test cost:0.28736
0.7-0.7 run 0 - epoch 240/700 --- Train cost:0.46360 --- Test cost:0.27791
0.7-0.7 run 0 - epoch 245/700 --- Train cost:0.46114 --- Test cost:0.26730
0.7-0.7 run 0 - epoch 250/700 --- Train cost:0.45611 --- Test cost:0.25630
0.7-0.7 run 0 - epoch 255/700 --- Train cost:0.45037 --- Test cost:0.25168
0.7-0.7 run 0 - epoch 260/700 --- Train cost:0.44469 --- Test cost:0.25399
0.7-0.7 run 0 - epoch 265/700 --- Train cost:0.44792 --- Test cost:0.26350
0.7-0.7 run 0 - epoch 270/700 --- Train cost:0.44430 --- Test cost:0.26503
0.7-0.7 run 0 - epoch 275/700 --- Train cost:0.44347 --- Test cost:0.26522
0.7-0.7 run 0 - epoch 280/700 --- Train cost:0.44104 --- Test cost:0.25622
0.7-0.7 run 0 - epoch 285/700 --- Train cost:0.42769 --- Test cost:0.24000
0.7-0.7 run 0 - epoch 290/700 --- Train cost:0.40354 --- Test cost:0.22049
0.7-0.7 run 0 - epoch 295/700 --- Train cost:0.38719 --- Test cost:0.20794
0.7-0.7 run 0 - epoch 300/700 --- Train cost:0.37759 --- Test cost:0.20069
0.7-0.7 run 0 - epoch 305/700 --- Train cost:0.37247 --- Test cost:0.19699
0.7-0.7 run 0 - epoch 310/700 --- Train cost:0.37298 --- Test cost:0.19319
0.7-0.7 run 0 - epoch 315/700 --- Train cost:0.37427 --- Test cost:0.19019
0.7-0.7 run 0 - epoch 320/700 --- Train cost:0.38292 --- Test cost:0.19009
0.7-0.7 run 0 - epoch 325/700 --- Train cost:0.37765 --- Test cost:0.18081
0.7-0.7 run 0 - epoch 330/700 --- Train cost:0.37783 --- Test cost:0.17595
0.7-0.7 run 0 - epoch 335/700 --- Train cost:0.37735 --- Test cost:0.17224
0.7-0.7 run 0 - epoch 340/700 --- Train cost:0.37579 --- Test cost:0.16966
0.7-0.7 run 0 - epoch 345/700 --- Train cost:0.37607 --- Test cost:0.16701
0.7-0.7 run 0 - epoch 350/700 --- Train cost:0.37827 --- Test cost:0.16981
0.7-0.7 run 0 - epoch 355/700 --- Train cost:0.38056 --- Test cost:0.17700
0.7-0.7 run 0 - epoch 360/700 --- Train cost:0.38604 --- Test cost:0.18457
0.7-0.7 run 0 - epoch 365/700 --- Train cost:0.39772 --- Test cost:0.19238
0.7-0.7 run 0 - epoch 370/700 --- Train cost:0.40266 --- Test cost:0.20014
0.7-0.7 run 0 - epoch 375/700 --- Train cost:0.40168 --- Test cost:0.20156
0.7-0.7 run 0 - epoch 380/700 --- Train cost:0.41055 --- Test cost:0.20561
0.7-0.7 run 0 - epoch 385/700 --- Train cost:0.40819 --- Test cost:0.20230
0.7-0.7 run 0 - epoch 390/700 --- Train cost:0.40825 --- Test cost:0.20408
0.7-0.7 run 0 - epoch 395/700 --- Train cost:0.40333 --- Test cost:0.20006
0.7-0.7 run 0 - epoch 400/700 --- Train cost:0.39877 --- Test cost:0.20025
0.7-0.7 run 0 - epoch 405/700 --- Train cost:0.40695 --- Test cost:0.20493
0.7-0.7 run 0 - epoch 410/700 --- Train cost:0.41314 --- Test cost:0.20398
0.7-0.7 run 0 - epoch 415/700 --- Train cost:0.40823 --- Test cost:0.19830
0.7-0.7 run 0 - epoch 420/700 --- Train cost:0.39964 --- Test cost:0.18839
0.7-0.7 run 0 - epoch 425/700 --- Train cost:0.39551 --- Test cost:0.17568
0.7-0.7 run 0 - epoch 430/700 --- Train cost:0.39482 --- Test cost:0.17051
0.7-0.7 run 0 - epoch 435/700 --- Train cost:0.39493 --- Test cost:0.17337
0.7-0.7 run 0 - epoch 440/700 --- Train cost:0.38720 --- Test cost:0.16616
0.7-0.7 run 0 - epoch 445/700 --- Train cost:0.37722 --- Test cost:0.15667
0.7-0.7 run 0 - epoch 450/700 --- Train cost:0.37811 --- Test cost:0.15306
0.7-0.7 run 0 - epoch 455/700 --- Train cost:0.38048 --- Test cost:0.15148
0.7-0.7 run 0 - epoch 460/700 --- Train cost:0.38578 --- Test cost:0.15344
0.7-0.7 run 0 - epoch 465/700 --- Train cost:0.38161 --- Test cost:0.15030
0.7-0.7 run 0 - epoch 470/700 --- Train cost:0.37868 --- Test cost:0.14834
0.7-0.7 run 0 - epoch 475/700 --- Train cost:0.38603 --- Test cost:0.14733
0.7-0.7 run 0 - epoch 480/700 --- Train cost:0.38871 --- Test cost:0.14412
0.7-0.7 run 0 - epoch 485/700 --- Train cost:0.39302 --- Test cost:0.14307
0.7-0.7 run 0 - epoch 490/700 --- Train cost:0.39034 --- Test cost:0.14161
0.7-0.7 run 0 - epoch 495/700 --- Train cost:0.38373 --- Test cost:0.13809
0.7-0.7 run 0 - epoch 500/700 --- Train cost:0.38406 --- Test cost:0.13817
0.7-0.7 run 0 - epoch 505/700 --- Train cost:0.38498 --- Test cost:0.14435
0.7-0.7 run 0 - epoch 510/700 --- Train cost:0.37525 --- Test cost:0.14556
0.7-0.7 run 0 - epoch 515/700 --- Train cost:0.36514 --- Test cost:0.14936
0.7-0.7 run 0 - epoch 520/700 --- Train cost:0.35913 --- Test cost:0.15554
0.7-0.7 run 0 - epoch 525/700 --- Train cost:0.36207 --- Test cost:0.17047
0.7-0.7 run 0 - epoch 530/700 --- Train cost:0.36535 --- Test cost:0.17816
0.7-0.7 run 0 - epoch 535/700 --- Train cost:0.36539 --- Test cost:0.18307
0.7-0.7 run 0 - epoch 540/700 --- Train cost:0.36174 --- Test cost:0.18684
0.7-0.7 run 0 - epoch 545/700 --- Train cost:0.36346 --- Test cost:0.19253
0.7-0.7 run 0 - epoch 550/700 --- Train cost:0.36878 --- Test cost:0.19725
0.7-0.7 run 0 - epoch 555/700 --- Train cost:0.37222 --- Test cost:0.20353
0.7-0.7 run 0 - epoch 560/700 --- Train cost:0.37081 --- Test cost:0.20474
0.7-0.7 run 0 - epoch 565/700 --- Train cost:0.35958 --- Test cost:0.20530
0.7-0.7 run 0 - epoch 570/700 --- Train cost:0.34750 --- Test cost:0.20396
0.7-0.7 run 0 - epoch 575/700 --- Train cost:0.34858 --- Test cost:0.20859
0.7-0.7 run 0 - epoch 580/700 --- Train cost:0.35691 --- Test cost:0.20612
0.7-0.7 run 0 - epoch 585/700 --- Train cost:0.36982 --- Test cost:0.20662
0.7-0.7 run 0 - epoch 590/700 --- Train cost:0.38588 --- Test cost:0.21320
0.7-0.7 run 0 - epoch 595/700 --- Train cost:0.39234 --- Test cost:0.20825
0.7-0.7 run 0 - epoch 600/700 --- Train cost:0.39357 --- Test cost:0.20547
0.7-0.7 run 0 - epoch 605/700 --- Train cost:0.39525 --- Test cost:0.19908
0.7-0.7 run 0 - epoch 610/700 --- Train cost:0.40385 --- Test cost:0.19959
0.7-0.7 run 0 - epoch 615/700 --- Train cost:0.41621 --- Test cost:0.20243
0.7-0.7 run 0 - epoch 620/700 --- Train cost:0.43105 --- Test cost:0.21817
0.7-0.7 run 0 - epoch 625/700 --- Train cost:0.44099 --- Test cost:0.22861
0.7-0.7 run 0 - epoch 630/700 --- Train cost:0.45350 --- Test cost:0.23422
0.7-0.7 run 0 - epoch 635/700 --- Train cost:0.45594 --- Test cost:0.23467
0.7-0.7 run 0 - epoch 640/700 --- Train cost:0.45833 --- Test cost:0.23053
0.7-0.7 run 0 - epoch 645/700 --- Train cost:0.45528 --- Test cost:0.22126
0.7-0.7 run 0 - epoch 650/700 --- Train cost:0.45358 --- Test cost:0.21023
0.7-0.7 run 0 - epoch 655/700 --- Train cost:0.44874 --- Test cost:0.19694
0.7-0.7 run 0 - epoch 660/700 --- Train cost:0.44231 --- Test cost:0.18700
0.7-0.7 run 0 - epoch 665/700 --- Train cost:0.43445 --- Test cost:0.17822
0.7-0.7 run 0 - epoch 670/700 --- Train cost:0.42189 --- Test cost:0.17231
0.7-0.7 run 0 - epoch 675/700 --- Train cost:0.40009 --- Test cost:0.15920
0.7-0.7 run 0 - epoch 680/700 --- Train cost:0.39000 --- Test cost:0.15155
0.7-0.7 run 0 - epoch 685/700 --- Train cost:0.39403 --- Test cost:0.14920
0.7-0.7 run 0 - epoch 690/700 --- Train cost:0.39859 --- Test cost:0.14477
0.7-0.7 run 0 - epoch 695/700 --- Train cost:0.39514 --- Test cost:0.13717
0.7-0.7 run 1 - epoch 0/700 --- Train cost:0.29358 --- Test cost:0.10004
0.7-0.7 run 1 - epoch 5/700 --- Train cost:0.29562 --- Test cost:0.08927
0.7-0.7 run 1 - epoch 10/700 --- Train cost:0.29830 --- Test cost:0.09813
0.7-0.7 run 1 - epoch 15/700 --- Train cost:0.30584 --- Test cost:0.11427
0.7-0.7 run 1 - epoch 20/700 --- Train cost:0.30768 --- Test cost:0.12313
0.7-0.7 run 1 - epoch 25/700 --- Train cost:0.30571 --- Test cost:0.13305
0.7-0.7 run 1 - epoch 30/700 --- Train cost:0.30763 --- Test cost:0.14353
0.7-0.7 run 1 - epoch 35/700 --- Train cost:0.31223 --- Test cost:0.15247
0.7-0.7 run 1 - epoch 40/700 --- Train cost:0.31444 --- Test cost:0.16273
0.7-0.7 run 1 - epoch 45/700 --- Train cost:0.31952 --- Test cost:0.17322
0.7-0.7 run 1 - epoch 50/700 --- Train cost:0.31981 --- Test cost:0.18617
0.7-0.7 run 1 - epoch 55/700 --- Train cost:0.31634 --- Test cost:0.19106
0.7-0.7 run 1 - epoch 60/700 --- Train cost:0.31258 --- Test cost:0.19226
0.7-0.7 run 1 - epoch 65/700 --- Train cost:0.30884 --- Test cost:0.19185
0.7-0.7 run 1 - epoch 70/700 --- Train cost:0.30403 --- Test cost:0.19113
0.7-0.7 run 1 - epoch 75/700 --- Train cost:0.29350 --- Test cost:0.18588
0.7-0.7 run 1 - epoch 80/700 --- Train cost:0.28498 --- Test cost:0.17491
0.7-0.7 run 1 - epoch 85/700 --- Train cost:0.27136 --- Test cost:0.16793
0.7-0.7 run 1 - epoch 90/700 --- Train cost:0.26303 --- Test cost:0.15911
0.7-0.7 run 1 - epoch 95/700 --- Train cost:0.25530 --- Test cost:0.15196
0.7-0.7 run 1 - epoch 100/700 --- Train cost:0.25767 --- Test cost:0.15182
0.7-0.7 run 1 - epoch 105/700 --- Train cost:0.26665 --- Test cost:0.15105
0.7-0.7 run 1 - epoch 110/700 --- Train cost:0.27480 --- Test cost:0.14792
0.7-0.7 run 1 - epoch 115/700 --- Train cost:0.28630 --- Test cost:0.14845
0.7-0.7 run 1 - epoch 120/700 --- Train cost:0.30122 --- Test cost:0.14967
0.7-0.7 run 1 - epoch 125/700 --- Train cost:0.31297 --- Test cost:0.15004
0.7-0.7 run 1 - epoch 130/700 --- Train cost:0.32264 --- Test cost:0.15090
0.7-0.7 run 1 - epoch 135/700 --- Train cost:0.32352 --- Test cost:0.14529
0.7-0.7 run 1 - epoch 140/700 --- Train cost:0.32180 --- Test cost:0.14048
0.7-0.7 run 1 - epoch 145/700 --- Train cost:0.32766 --- Test cost:0.13811
0.7-0.7 run 1 - epoch 150/700 --- Train cost:0.34203 --- Test cost:0.14134
0.7-0.7 run 1 - epoch 155/700 --- Train cost:0.35293 --- Test cost:0.14463
0.7-0.7 run 1 - epoch 160/700 --- Train cost:0.35675 --- Test cost:0.14175
0.7-0.7 run 1 - epoch 165/700 --- Train cost:0.35355 --- Test cost:0.13170
0.7-0.7 run 1 - epoch 170/700 --- Train cost:0.35530 --- Test cost:0.12645
0.7-0.7 run 1 - epoch 175/700 --- Train cost:0.36155 --- Test cost:0.12634
0.7-0.7 run 1 - epoch 180/700 --- Train cost:0.36238 --- Test cost:0.12274
0.7-0.7 run 1 - epoch 185/700 --- Train cost:0.35822 --- Test cost:0.12516
0.7-0.7 run 1 - epoch 190/700 --- Train cost:0.35456 --- Test cost:0.12251
0.7-0.7 run 1 - epoch 195/700 --- Train cost:0.35061 --- Test cost:0.12055
0.7-0.7 run 1 - epoch 200/700 --- Train cost:0.34387 --- Test cost:0.11710
0.7-0.7 run 1 - epoch 205/700 --- Train cost:0.33607 --- Test cost:0.11503
0.7-0.7 run 1 - epoch 210/700 --- Train cost:0.32664 --- Test cost:0.11193
0.7-0.7 run 1 - epoch 215/700 --- Train cost:0.31188 --- Test cost:0.10780
0.7-0.7 run 1 - epoch 220/700 --- Train cost:0.30542 --- Test cost:0.10449
0.7-0.7 run 1 - epoch 225/700 --- Train cost:0.30320 --- Test cost:0.10244
0.7-0.7 run 1 - epoch 230/700 --- Train cost:0.30621 --- Test cost:0.10352
0.7-0.7 run 1 - epoch 235/700 --- Train cost:0.31252 --- Test cost:0.10736
0.7-0.7 run 1 - epoch 240/700 --- Train cost:0.31187 --- Test cost:0.11024
0.7-0.7 run 1 - epoch 245/700 --- Train cost:0.31513 --- Test cost:0.11634
0.7-0.7 run 1 - epoch 250/700 --- Train cost:0.31237 --- Test cost:0.11715
0.7-0.7 run 1 - epoch 255/700 --- Train cost:0.31365 --- Test cost:0.11557
0.7-0.7 run 1 - epoch 260/700 --- Train cost:0.31508 --- Test cost:0.11481
0.7-0.7 run 1 - epoch 265/700 --- Train cost:0.31632 --- Test cost:0.11552
0.7-0.7 run 1 - epoch 270/700 --- Train cost:0.31537 --- Test cost:0.11706
0.7-0.7 run 1 - epoch 275/700 --- Train cost:0.30798 --- Test cost:0.12300
0.7-0.7 run 1 - epoch 280/700 --- Train cost:0.29972 --- Test cost:0.12489
0.7-0.7 run 1 - epoch 285/700 --- Train cost:0.29876 --- Test cost:0.12912
0.7-0.7 run 1 - epoch 290/700 --- Train cost:0.30667 --- Test cost:0.13576
0.7-0.7 run 1 - epoch 295/700 --- Train cost:0.31459 --- Test cost:0.14385
0.7-0.7 run 1 - epoch 300/700 --- Train cost:0.32012 --- Test cost:0.14676
0.7-0.7 run 1 - epoch 305/700 --- Train cost:0.31801 --- Test cost:0.14874
0.7-0.7 run 1 - epoch 310/700 --- Train cost:0.31168 --- Test cost:0.15247
0.7-0.7 run 1 - epoch 315/700 --- Train cost:0.31095 --- Test cost:0.15273
0.7-0.7 run 1 - epoch 320/700 --- Train cost:0.30766 --- Test cost:0.14921
0.7-0.7 run 1 - epoch 325/700 --- Train cost:0.31012 --- Test cost:0.15022
0.7-0.7 run 1 - epoch 330/700 --- Train cost:0.31308 --- Test cost:0.15089
0.7-0.7 run 1 - epoch 335/700 --- Train cost:0.31083 --- Test cost:0.14378
0.7-0.7 run 1 - epoch 340/700 --- Train cost:0.31134 --- Test cost:0.13989
0.7-0.7 run 1 - epoch 345/700 --- Train cost:0.31198 --- Test cost:0.13972
0.7-0.7 run 1 - epoch 350/700 --- Train cost:0.32131 --- Test cost:0.14785
0.7-0.7 run 1 - epoch 355/700 --- Train cost:0.33550 --- Test cost:0.16138
0.7-0.7 run 1 - epoch 360/700 --- Train cost:0.33708 --- Test cost:0.16262
0.7-0.7 run 1 - epoch 365/700 --- Train cost:0.33248 --- Test cost:0.15744
0.7-0.7 run 1 - epoch 370/700 --- Train cost:0.33540 --- Test cost:0.15838
0.7-0.7 run 1 - epoch 375/700 --- Train cost:0.33821 --- Test cost:0.16209
0.7-0.7 run 1 - epoch 380/700 --- Train cost:0.33374 --- Test cost:0.16230
0.7-0.7 run 1 - epoch 385/700 --- Train cost:0.32498 --- Test cost:0.15911
0.7-0.7 run 1 - epoch 390/700 --- Train cost:0.32137 --- Test cost:0.15766
0.7-0.7 run 1 - epoch 395/700 --- Train cost:0.31979 --- Test cost:0.16168
0.7-0.7 run 1 - epoch 400/700 --- Train cost:0.31886 --- Test cost:0.17175
0.7-0.7 run 1 - epoch 405/700 --- Train cost:0.31401 --- Test cost:0.17555
0.7-0.7 run 1 - epoch 410/700 --- Train cost:0.31555 --- Test cost:0.17807
0.7-0.7 run 1 - epoch 415/700 --- Train cost:0.31647 --- Test cost:0.18183
0.7-0.7 run 1 - epoch 420/700 --- Train cost:0.31898 --- Test cost:0.18739
0.7-0.7 run 1 - epoch 425/700 --- Train cost:0.31567 --- Test cost:0.19097
0.7-0.7 run 1 - epoch 430/700 --- Train cost:0.31468 --- Test cost:0.19269
0.7-0.7 run 1 - epoch 435/700 --- Train cost:0.30736 --- Test cost:0.19150
0.7-0.7 run 1 - epoch 440/700 --- Train cost:0.29837 --- Test cost:0.19498
0.7-0.7 run 1 - epoch 445/700 --- Train cost:0.28992 --- Test cost:0.20124
0.7-0.7 run 1 - epoch 450/700 --- Train cost:0.28765 --- Test cost:0.21018
0.7-0.7 run 1 - epoch 455/700 --- Train cost:0.28457 --- Test cost:0.22184
0.7-0.7 run 1 - epoch 460/700 --- Train cost:0.28292 --- Test cost:0.23305
0.7-0.7 run 1 - epoch 465/700 --- Train cost:0.27106 --- Test cost:0.22972
0.7-0.7 run 1 - epoch 470/700 --- Train cost:0.26564 --- Test cost:0.22419
0.7-0.7 run 1 - epoch 475/700 --- Train cost:0.26370 --- Test cost:0.21683
0.7-0.7 run 1 - epoch 480/700 --- Train cost:0.25979 --- Test cost:0.20745
0.7-0.7 run 1 - epoch 485/700 --- Train cost:0.25938 --- Test cost:0.20060
0.7-0.7 run 1 - epoch 490/700 --- Train cost:0.25192 --- Test cost:0.18966
0.7-0.7 run 1 - epoch 495/700 --- Train cost:0.24650 --- Test cost:0.17946
0.7-0.7 run 1 - epoch 500/700 --- Train cost:0.24864 --- Test cost:0.17283
0.7-0.7 run 1 - epoch 505/700 --- Train cost:0.25101 --- Test cost:0.16612
0.7-0.7 run 1 - epoch 510/700 --- Train cost:0.25201 --- Test cost:0.15781
0.7-0.7 run 1 - epoch 515/700 --- Train cost:0.26089 --- Test cost:0.15945
0.7-0.7 run 1 - epoch 520/700 --- Train cost:0.27329 --- Test cost:0.15753
0.7-0.7 run 1 - epoch 525/700 --- Train cost:0.27786 --- Test cost:0.14996
0.7-0.7 run 1 - epoch 530/700 --- Train cost:0.28370 --- Test cost:0.14662
0.7-0.7 run 1 - epoch 535/700 --- Train cost:0.28911 --- Test cost:0.14295
0.7-0.7 run 1 - epoch 540/700 --- Train cost:0.28622 --- Test cost:0.14291
0.7-0.7 run 1 - epoch 545/700 --- Train cost:0.28351 --- Test cost:0.13921
0.7-0.7 run 1 - epoch 550/700 --- Train cost:0.27121 --- Test cost:0.13668
0.7-0.7 run 1 - epoch 555/700 --- Train cost:0.24713 --- Test cost:0.12633
0.7-0.7 run 1 - epoch 560/700 --- Train cost:0.24302 --- Test cost:0.12539
0.7-0.7 run 1 - epoch 565/700 --- Train cost:0.24177 --- Test cost:0.12594
0.7-0.7 run 1 - epoch 570/700 --- Train cost:0.24071 --- Test cost:0.12040
0.7-0.7 run 1 - epoch 575/700 --- Train cost:0.24747 --- Test cost:0.11985
0.7-0.7 run 1 - epoch 580/700 --- Train cost:0.24997 --- Test cost:0.11531
0.7-0.7 run 1 - epoch 585/700 --- Train cost:0.25019 --- Test cost:0.11012
0.7-0.7 run 1 - epoch 590/700 --- Train cost:0.24599 --- Test cost:0.09890
0.7-0.7 run 1 - epoch 595/700 --- Train cost:0.24409 --- Test cost:0.08938
0.7-0.7 run 1 - epoch 600/700 --- Train cost:0.24055 --- Test cost:0.08287
0.7-0.7 run 1 - epoch 605/700 --- Train cost:0.24055 --- Test cost:0.08124
0.7-0.7 run 1 - epoch 610/700 --- Train cost:0.23607 --- Test cost:0.08007
0.7-0.7 run 1 - epoch 615/700 --- Train cost:0.23199 --- Test cost:0.07462
0.7-0.7 run 1 - epoch 620/700 --- Train cost:0.23038 --- Test cost:0.07189
0.7-0.7 run 1 - epoch 625/700 --- Train cost:0.23302 --- Test cost:0.07066
0.7-0.7 run 1 - epoch 630/700 --- Train cost:0.23930 --- Test cost:0.06843
0.7-0.7 run 1 - epoch 635/700 --- Train cost:0.24611 --- Test cost:0.06503
0.7-0.7 run 1 - epoch 640/700 --- Train cost:0.24974 --- Test cost:0.06417
0.7-0.7 run 1 - epoch 645/700 --- Train cost:0.25392 --- Test cost:0.06473
0.7-0.7 run 1 - epoch 650/700 --- Train cost:0.25927 --- Test cost:0.06840
0.7-0.7 run 1 - epoch 655/700 --- Train cost:0.26451 --- Test cost:0.07206
0.7-0.7 run 1 - epoch 660/700 --- Train cost:0.27431 --- Test cost:0.07613
0.7-0.7 run 1 - epoch 665/700 --- Train cost:0.28359 --- Test cost:0.07886
0.7-0.7 run 1 - epoch 670/700 --- Train cost:0.28954 --- Test cost:0.08042
0.7-0.7 run 1 - epoch 675/700 --- Train cost:0.29998 --- Test cost:0.08122
0.7-0.7 run 1 - epoch 680/700 --- Train cost:0.30792 --- Test cost:0.08440
0.7-0.7 run 1 - epoch 685/700 --- Train cost:0.31818 --- Test cost:0.08598
0.7-0.7 run 1 - epoch 690/700 --- Train cost:0.32664 --- Test cost:0.08584
0.7-0.7 run 1 - epoch 695/700 --- Train cost:0.33422 --- Test cost:0.08522
0.7-0.7 run 2 - epoch 0/700 --- Train cost:0.37401 --- Test cost:0.16714
0.7-0.7 run 2 - epoch 5/700 --- Train cost:0.38369 --- Test cost:0.15754
0.7-0.7 run 2 - epoch 10/700 --- Train cost:0.39699 --- Test cost:0.16756
0.7-0.7 run 2 - epoch 15/700 --- Train cost:0.40373 --- Test cost:0.17435
0.7-0.7 run 2 - epoch 20/700 --- Train cost:0.40979 --- Test cost:0.18412
0.7-0.7 run 2 - epoch 25/700 --- Train cost:0.41868 --- Test cost:0.19635
0.7-0.7 run 2 - epoch 30/700 --- Train cost:0.42532 --- Test cost:0.20321
0.7-0.7 run 2 - epoch 35/700 --- Train cost:0.43590 --- Test cost:0.21194
0.7-0.7 run 2 - epoch 40/700 --- Train cost:0.44526 --- Test cost:0.21365
0.7-0.7 run 2 - epoch 45/700 --- Train cost:0.45004 --- Test cost:0.21677
0.7-0.7 run 2 - epoch 50/700 --- Train cost:0.45354 --- Test cost:0.22390
0.7-0.7 run 2 - epoch 55/700 --- Train cost:0.45082 --- Test cost:0.22317
0.7-0.7 run 2 - epoch 60/700 --- Train cost:0.45462 --- Test cost:0.22701
0.7-0.7 run 2 - epoch 65/700 --- Train cost:0.46895 --- Test cost:0.23407
0.7-0.7 run 2 - epoch 70/700 --- Train cost:0.47450 --- Test cost:0.23077
0.7-0.7 run 2 - epoch 75/700 --- Train cost:0.48380 --- Test cost:0.23090
0.7-0.7 run 2 - epoch 80/700 --- Train cost:0.49063 --- Test cost:0.24205
0.7-0.7 run 2 - epoch 85/700 --- Train cost:0.49676 --- Test cost:0.24845
0.7-0.7 run 2 - epoch 90/700 --- Train cost:0.49630 --- Test cost:0.24431
0.7-0.7 run 2 - epoch 95/700 --- Train cost:0.50739 --- Test cost:0.24749
0.7-0.7 run 2 - epoch 100/700 --- Train cost:0.51176 --- Test cost:0.24729
0.7-0.7 run 2 - epoch 105/700 --- Train cost:0.50988 --- Test cost:0.23966
0.7-0.7 run 2 - epoch 110/700 --- Train cost:0.50731 --- Test cost:0.22839
0.7-0.7 run 2 - epoch 115/700 --- Train cost:0.49640 --- Test cost:0.21617
0.7-0.7 run 2 - epoch 120/700 --- Train cost:0.50322 --- Test cost:0.21768
0.7-0.7 run 2 - epoch 125/700 --- Train cost:0.51728 --- Test cost:0.22410
0.7-0.7 run 2 - epoch 130/700 --- Train cost:0.51751 --- Test cost:0.22351
0.7-0.7 run 2 - epoch 135/700 --- Train cost:0.51789 --- Test cost:0.22254
0.7-0.7 run 2 - epoch 140/700 --- Train cost:0.50712 --- Test cost:0.20596
0.7-0.7 run 2 - epoch 145/700 --- Train cost:0.50923 --- Test cost:0.20111
0.7-0.7 run 2 - epoch 150/700 --- Train cost:0.51981 --- Test cost:0.19884
0.7-0.7 run 2 - epoch 155/700 --- Train cost:0.52774 --- Test cost:0.20481
0.7-0.7 run 2 - epoch 160/700 --- Train cost:0.52958 --- Test cost:0.21109
0.7-0.7 run 2 - epoch 165/700 --- Train cost:0.53263 --- Test cost:0.21732
0.7-0.7 run 2 - epoch 170/700 --- Train cost:0.53219 --- Test cost:0.21988
0.7-0.7 run 2 - epoch 175/700 --- Train cost:0.53100 --- Test cost:0.22886
0.7-0.7 run 2 - epoch 180/700 --- Train cost:0.52766 --- Test cost:0.23419
0.7-0.7 run 2 - epoch 185/700 --- Train cost:0.52484 --- Test cost:0.23345
0.7-0.7 run 2 - epoch 190/700 --- Train cost:0.52674 --- Test cost:0.23846
0.7-0.7 run 2 - epoch 195/700 --- Train cost:0.52637 --- Test cost:0.23948
0.7-0.7 run 2 - epoch 200/700 --- Train cost:0.51790 --- Test cost:0.23756
0.7-0.7 run 2 - epoch 205/700 --- Train cost:0.51252 --- Test cost:0.23939
0.7-0.7 run 2 - epoch 210/700 --- Train cost:0.50808 --- Test cost:0.24008
0.7-0.7 run 2 - epoch 215/700 --- Train cost:0.49692 --- Test cost:0.23191
0.7-0.7 run 2 - epoch 220/700 --- Train cost:0.47958 --- Test cost:0.22474
0.7-0.7 run 2 - epoch 225/700 --- Train cost:0.47624 --- Test cost:0.22279
0.7-0.7 run 2 - epoch 230/700 --- Train cost:0.46946 --- Test cost:0.21351
0.7-0.7 run 2 - epoch 235/700 --- Train cost:0.46608 --- Test cost:0.20953
0.7-0.7 run 2 - epoch 240/700 --- Train cost:0.47163 --- Test cost:0.20564
0.7-0.7 run 2 - epoch 245/700 --- Train cost:0.47409 --- Test cost:0.20331
0.7-0.7 run 2 - epoch 250/700 --- Train cost:0.47398 --- Test cost:0.20205
0.7-0.7 run 2 - epoch 255/700 --- Train cost:0.48132 --- Test cost:0.20843
0.7-0.7 run 2 - epoch 260/700 --- Train cost:0.48416 --- Test cost:0.21112
0.7-0.7 run 2 - epoch 265/700 --- Train cost:0.48746 --- Test cost:0.20545
0.7-0.7 run 2 - epoch 270/700 --- Train cost:0.49397 --- Test cost:0.20514
0.7-0.7 run 2 - epoch 275/700 --- Train cost:0.50312 --- Test cost:0.19975
0.7-0.7 run 2 - epoch 280/700 --- Train cost:0.50932 --- Test cost:0.19193
0.7-0.7 run 2 - epoch 285/700 --- Train cost:0.51182 --- Test cost:0.18140
0.7-0.7 run 2 - epoch 290/700 --- Train cost:0.51096 --- Test cost:0.17216
0.7-0.7 run 2 - epoch 295/700 --- Train cost:0.51677 --- Test cost:0.16731
0.7-0.7 run 2 - epoch 300/700 --- Train cost:0.52956 --- Test cost:0.17133
0.7-0.7 run 2 - epoch 305/700 --- Train cost:0.54421 --- Test cost:0.17103
0.7-0.7 run 2 - epoch 310/700 --- Train cost:0.54871 --- Test cost:0.17076
0.7-0.7 run 2 - epoch 315/700 --- Train cost:0.55535 --- Test cost:0.16884
0.7-0.7 run 2 - epoch 320/700 --- Train cost:0.56483 --- Test cost:0.17044
0.7-0.7 run 2 - epoch 325/700 --- Train cost:0.56122 --- Test cost:0.17665
0.7-0.7 run 2 - epoch 330/700 --- Train cost:0.55180 --- Test cost:0.18125
0.7-0.7 run 2 - epoch 335/700 --- Train cost:0.55243 --- Test cost:0.19562
0.7-0.7 run 2 - epoch 340/700 --- Train cost:0.55877 --- Test cost:0.21321
0.7-0.7 run 2 - epoch 345/700 --- Train cost:0.56604 --- Test cost:0.23505
0.7-0.7 run 2 - epoch 350/700 --- Train cost:0.56562 --- Test cost:0.24703
0.7-0.7 run 2 - epoch 355/700 --- Train cost:0.56474 --- Test cost:0.24705
0.7-0.7 run 2 - epoch 360/700 --- Train cost:0.56264 --- Test cost:0.23709
0.7-0.7 run 2 - epoch 365/700 --- Train cost:0.56610 --- Test cost:0.23458
0.7-0.7 run 2 - epoch 370/700 --- Train cost:0.57724 --- Test cost:0.23848
0.7-0.7 run 2 - epoch 375/700 --- Train cost:0.58381 --- Test cost:0.24055
0.7-0.7 run 2 - epoch 380/700 --- Train cost:0.58681 --- Test cost:0.22896
0.7-0.7 run 2 - epoch 385/700 --- Train cost:0.57849 --- Test cost:0.21829
0.7-0.7 run 2 - epoch 390/700 --- Train cost:0.57526 --- Test cost:0.21713
0.7-0.7 run 2 - epoch 395/700 --- Train cost:0.57066 --- Test cost:0.21360
0.7-0.7 run 2 - epoch 400/700 --- Train cost:0.55831 --- Test cost:0.20889
0.7-0.7 run 2 - epoch 405/700 --- Train cost:0.54510 --- Test cost:0.20701
0.7-0.7 run 2 - epoch 410/700 --- Train cost:0.54627 --- Test cost:0.20956
0.7-0.7 run 2 - epoch 415/700 --- Train cost:0.55227 --- Test cost:0.21839
0.7-0.7 run 2 - epoch 420/700 --- Train cost:0.55974 --- Test cost:0.22540
0.7-0.7 run 2 - epoch 425/700 --- Train cost:0.56738 --- Test cost:0.23546
0.7-0.7 run 2 - epoch 430/700 --- Train cost:0.57224 --- Test cost:0.23986
0.7-0.7 run 2 - epoch 435/700 --- Train cost:0.57561 --- Test cost:0.25008
0.7-0.7 run 2 - epoch 440/700 --- Train cost:0.57805 --- Test cost:0.25512
0.7-0.7 run 2 - epoch 445/700 --- Train cost:0.56517 --- Test cost:0.25398
0.7-0.7 run 2 - epoch 450/700 --- Train cost:0.54523 --- Test cost:0.25012
0.7-0.7 run 2 - epoch 455/700 --- Train cost:0.52495 --- Test cost:0.24045
0.7-0.7 run 2 - epoch 460/700 --- Train cost:0.51111 --- Test cost:0.23039
0.7-0.7 run 2 - epoch 465/700 --- Train cost:0.49880 --- Test cost:0.21973
0.7-0.7 run 2 - epoch 470/700 --- Train cost:0.48413 --- Test cost:0.20893
0.7-0.7 run 2 - epoch 475/700 --- Train cost:0.47716 --- Test cost:0.19987
0.7-0.7 run 2 - epoch 480/700 --- Train cost:0.48024 --- Test cost:0.19714
0.7-0.7 run 2 - epoch 485/700 --- Train cost:0.47504 --- Test cost:0.19677
0.7-0.7 run 2 - epoch 490/700 --- Train cost:0.46867 --- Test cost:0.19144
0.7-0.7 run 2 - epoch 495/700 --- Train cost:0.46943 --- Test cost:0.19224
0.7-0.7 run 2 - epoch 500/700 --- Train cost:0.46463 --- Test cost:0.19479
0.7-0.7 run 2 - epoch 505/700 --- Train cost:0.45287 --- Test cost:0.18943
0.7-0.7 run 2 - epoch 510/700 --- Train cost:0.44250 --- Test cost:0.18984
0.7-0.7 run 2 - epoch 515/700 --- Train cost:0.43649 --- Test cost:0.19257
0.7-0.7 run 2 - epoch 520/700 --- Train cost:0.43326 --- Test cost:0.19549
0.7-0.7 run 2 - epoch 525/700 --- Train cost:0.43221 --- Test cost:0.20513
0.7-0.7 run 2 - epoch 530/700 --- Train cost:0.43072 --- Test cost:0.20765
0.7-0.7 run 2 - epoch 535/700 --- Train cost:0.42460 --- Test cost:0.20938
0.7-0.7 run 2 - epoch 540/700 --- Train cost:0.41998 --- Test cost:0.21191
0.7-0.7 run 2 - epoch 545/700 --- Train cost:0.41134 --- Test cost:0.21242
0.7-0.7 run 2 - epoch 550/700 --- Train cost:0.40963 --- Test cost:0.21842
0.7-0.7 run 2 - epoch 555/700 --- Train cost:0.40604 --- Test cost:0.22251
0.7-0.7 run 2 - epoch 560/700 --- Train cost:0.39802 --- Test cost:0.21908
0.7-0.7 run 2 - epoch 565/700 --- Train cost:0.39163 --- Test cost:0.21332
0.7-0.7 run 2 - epoch 570/700 --- Train cost:0.39213 --- Test cost:0.21169
0.7-0.7 run 2 - epoch 575/700 --- Train cost:0.39527 --- Test cost:0.21466
0.7-0.7 run 2 - epoch 580/700 --- Train cost:0.39338 --- Test cost:0.21445
0.7-0.7 run 2 - epoch 585/700 --- Train cost:0.38505 --- Test cost:0.20274
0.7-0.7 run 2 - epoch 590/700 --- Train cost:0.37672 --- Test cost:0.19304
0.7-0.7 run 2 - epoch 595/700 --- Train cost:0.37163 --- Test cost:0.19102
0.7-0.7 run 2 - epoch 600/700 --- Train cost:0.36788 --- Test cost:0.19128
0.7-0.7 run 2 - epoch 605/700 --- Train cost:0.36246 --- Test cost:0.18895
0.7-0.7 run 2 - epoch 610/700 --- Train cost:0.35591 --- Test cost:0.18291
0.7-0.7 run 2 - epoch 615/700 --- Train cost:0.35000 --- Test cost:0.17632
0.7-0.7 run 2 - epoch 620/700 --- Train cost:0.34414 --- Test cost:0.16742
0.7-0.7 run 2 - epoch 625/700 --- Train cost:0.34351 --- Test cost:0.15873
0.7-0.7 run 2 - epoch 630/700 --- Train cost:0.34201 --- Test cost:0.15686
0.7-0.7 run 2 - epoch 635/700 --- Train cost:0.34434 --- Test cost:0.15672
0.7-0.7 run 2 - epoch 640/700 --- Train cost:0.34492 --- Test cost:0.15906
0.7-0.7 run 2 - epoch 645/700 --- Train cost:0.35059 --- Test cost:0.16424
0.7-0.7 run 2 - epoch 650/700 --- Train cost:0.35887 --- Test cost:0.16778
0.7-0.7 run 2 - epoch 655/700 --- Train cost:0.37010 --- Test cost:0.16903
0.7-0.7 run 2 - epoch 660/700 --- Train cost:0.37748 --- Test cost:0.16429
0.7-0.7 run 2 - epoch 665/700 --- Train cost:0.38475 --- Test cost:0.16418
0.7-0.7 run 2 - epoch 670/700 --- Train cost:0.38857 --- Test cost:0.16255
0.7-0.7 run 2 - epoch 675/700 --- Train cost:0.38648 --- Test cost:0.15830
0.7-0.7 run 2 - epoch 680/700 --- Train cost:0.38868 --- Test cost:0.16170
0.7-0.7 run 2 - epoch 685/700 --- Train cost:0.39062 --- Test cost:0.16472
0.7-0.7 run 2 - epoch 690/700 --- Train cost:0.38889 --- Test cost:0.16444
0.7-0.7 run 2 - epoch 695/700 --- Train cost:0.38060 --- Test cost:0.16110
Performance evaluation
Let’s compare the difference in performance with a plot:
fig, axs = plt.subplots(1, 2, figsize=(12, 4))
plt.subplots_adjust(wspace=0.05)
axs[0].set_title("MSE train")
for k, v in train_history.items():
train_losses = np.array(v)
mean_train_history = np.mean(train_losses, axis=0)
std_train_history = np.std(train_losses, axis=0,)
mean_train_history = mean_train_history.reshape((epochs,))
std_train_history = std_train_history.reshape((epochs,))
# shadow standard deviation
axs[0].fill_between(
range(epochs),
mean_train_history - std_train_history,
mean_train_history + std_train_history,
alpha=0.2,
)
# average trend
axs[0].plot(range(epochs), mean_train_history, label=f"{k}") # Avg Loss
axs[1].set_title("MSE test")
for k, v in test_history.items():
test_losses = np.array(v)
mean_test_history = np.mean(test_losses, axis=0)
std_test_history = np.std(test_losses, axis=0,)
mean_test_history = mean_test_history.reshape((epochs,))
std_test_history = std_test_history.reshape((epochs,))
# shadow standard deviation
axs[1].fill_between(
range(epochs),
mean_test_history - std_test_history,
mean_test_history + std_test_history,
alpha=0.2,
)
# averange trend
axs[1].plot(range(epochs), mean_test_history, label=f"{k}") # Avg Loss
axs[0].legend(loc="upper center", bbox_to_anchor=(1.01, 1.25), ncol=4, fancybox=True, shadow=True)
for ax in axs.flat:
ax.set_xlabel("Epochs")
ax.set_ylabel("MSE")
ax.set_yscale("log")
ax.set_ylim([1e-3, 0.6])
ax.label_outer()
plt.subplots_adjust(bottom=0.3)
plt.show()
On the left you can see that without dropout there is a deep minimization of the training loss, moderate values of dropout converge, whereas high drop probabilities impede any learning. On the right, we can see the difference in generalization during the optimization process. Standard training without dropout initially reaches a low value of generalization error, but as the model starts to learn the noise in the training data (overfitting), the generalization error grows back. Oppositely, moderate values of dropout enable generalization errors comparable to the respective training ones. As the learning is not successful for elevated drop probabilities, the generalization error is huge. It is interesting to notice that the “not-learning” error is very close to the final error of the QNN trained without dropout.
Hence, one can conclude that low values of dropout greatly improve the generalization performance of the model and remove overfitting, even if the randomness of the technique inevitably makes the training a little noisy. On the other hand, high drop probabilities only hinder the training process.
Validation
To validate the technique we can also check how the model predicts in the whole \([-1,1]\) range with and without quantum dropout.
X, X_test, y, y_test = make_sin_dataset(dataset_size=20, test_size=0.25)
# spanning the whole range
x_ax = jnp.linspace(-1, 1, 100).reshape(100, 1)
# selecting which run we want to plot
run = 1
fig, ax = plt.subplots()
styles = ["dashed", "-.", "solid", "-."]
for i, k in enumerate(train_history.keys()):
if k[0] == 0.3:
alpha = 1
else:
alpha = 0.5
# predicting and rescaling
yp = scaler.inverse_transform(qnn(x_ax, opt_params[k][run], keep_rot).reshape(-1, 1))
plt.plot([[i] for i in np.linspace(-1, 1, 100)], yp, label=k, alpha=alpha, linestyle=styles[i])
plt.scatter(X, y, label="Training", zorder=10)
plt.scatter(X_test, y_test, label="Test", zorder=10)
ylabel = r"$y = \sin(\pi\cdot x) + \epsilon$"
plt.xlabel("x", fontsize="medium")
plt.ylabel(ylabel, fontsize="medium")
plt.legend()
ax.xaxis.set_major_locator(ticker.MultipleLocator(0.5))
ax.yaxis.set_major_locator(ticker.MultipleLocator(0.5))
plt.show()
The model without dropout overfits the noisy data by trying to exactly predict each of them, whereas dropout actually mitigates overfitting and makes the approximation of the underlying sinusoidal function way smoother.
Conclusion
In this demo, we explained the basic idea behind quantum dropout and how to avoid overfitting by randomly “dropping” some rotation gates of a QNN during the training phase. We invite you to check out the paper 1 for more dropout techniques and additional analysis. Try it yourself and develop new dropout strategies.
References
- 1(1,2)
Scala, F., Ceschini, A., Panella, M., & Gerace, D. (2023). A General Approach to Dropout in Quantum Neural Networks. Adv. Quantum Technol., 2300220.
- 2
Hinton, G., Srivastava, N., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2012). Improving neural networks by preventing co-adaptation of feature detectors. arXiv:1207.0580..
- 3
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research, 15(56):1929−1958..
- 4
Kiani,B. T., Lloyd, S., & Maity, R. (2020). Learning Unitaries by Gradient Descent. arXiv: 2001.11897..
- 5
Larocca, M., Ju, N., García-Martín, D., Coles, P. J., & Cerezo, M. (2023). Theory of overparametrization in quantum neural networks. Nat. Comp. Science, 3, 542–551..
About the author
Francesco Scala
Francesco Scala is a Ph.D. student in QML at the University of Pavia, Italy. His main research deals with both theoretical and numerical aspects of QML algorithms and Quantum Neural Networks, focusing in particular on overparametrization and regulari...
Total running time of the script: (6 minutes 47.790 seconds)
