Full Zero2Neuro Example: Exclusive OR
For this example, we will create a fully-connected network to learn a simple binary function – the Exclusive OR (XOR) function. This function has 4 possible cases (one on each row):
| In 0 | In 1 | Out 0 |
|---|---|---|
| 0 | 0 | 0 |
| 1 | 0 | 1 |
| 0 | 1 | 1 |
| 1 | 1 | 0 |
In 0 and In 1 are the two inputs into the network, and the Out 0 column shows the correct value that the network should produce for each of the cases.
Data Configuration
We specify the data with two different files:
- The XOR data itself is specified using a CSV file format: xor_data.csv
- The data configuration file states how the CSV file will be interpreted: data.txt
data.txt looks like this:
# Data is from a table
--data_format=tabular
--data_file=xor_data.csv
# Our one file becomes our training set
--data_set_type=fixed
# Two inputs
--data_inputs
In 0
In 1
# One output
--data_outputs
Out 0
Notes:
-
In any of the configuration files, characters following the # symbol are ignored. Typically, we use this feature to add textual explanations about the following lines.
-
Given that there is a single input file and –data_set_type=fixed, all of the examples in the file will be used for training. We do not have a validation or test data set in this example.
Network Configuration
In this example we are implementing a fully connected network with two hidden (intermediate) layers of neurons before the network produces its output. For XOR, at least one hidden layer is required to solve the problem.
The network.txt configuration file looks like this:
# Fully connected network
--network_type=fully_connected
# Two inputs
--input_shape
2
# Two hidden layers
--number_hidden_units
20
10
--hidden_activation=elu
# One output unit
--output_shape
1
--output_activation=sigmoid
Here, we declare the full structure of the network:
- input_shape: each example has two inputs; these form a vector (or 1-dimensional tensor)
- number_hidden_units: a list of integers (20 and 10 in this case) that declare how many hidden units there are in each of the two hidden layers. All input units connect to all units in the first hidden layer; all units in the first hidden layer connect to all units in the second hidden layer
- hidden_activation: declares which non-linear function is used for each of the hidden units
- output_shape: the output in this example is a single value (this must match the number of –data_outputs)
- output_activation: declares the non-linear function for the output unit. Here, we choose sigmoid because we want our output values to fall within the 0…1 range.
Experiment Configuration
The final configuration file describes the details of the training and reporting process.
The experiment.txt configuration file looks like this:
# Name used for files and wandb
--experiment_name=xor
# Loss function is what we minimize during training; mse=Mean Squared Error
--loss=mse
# We might also care about other metrics; mae=Mean Absolute Error
--metrics
mae
mse
# How fast we adjust the parameters
--learning_rate=0.001
# Maximum number of steps for training
--epochs=5000
# Stop the training early if no improvement is observed
# Note: we typically monitor val_loss when we have a validation data set
--early_stopping
--early_stopping_monitor=loss
--early_stopping_patience=2000
# Where to place trained networks, reports, and stored results
--results_path=./results
# Expanded file name
--output_file_base={args.experiment_name}_R{args.data_rotation:02d}
# Save the trained model to a file
--save_model
# Create a picture of the model
--render_model
# Save the training set results
--log_training_set
# Report the results to a XLSX file
--report
--report_training
--report_training_ins
Key arguments:
- loss: this defines the function that will be minimized during training
- metrics: a list of functions that measure model performance
- learning_rate: defines how quickly the training process will proceed (but too high can be devastating)
- epochs: the maximum number of training steps (subject to early stopping)
- save_model: write the final Keras3 model to a file. This can be loaded at a later time
- render_model: create a picture of your model and write it to a file
- log_training_set: write the details of training set inputs and corresponding network outputs to a pickle file for later analysis
- report: generate a report in XLSX file format of the performance of the trained model
- report_training: include in the report a sheet that contains the true and model output for each example
- report_training_ins: include on the above sheet the example inputs
Execution in the Bash Shell
- Activate Python Environment. The details will vary depending on your system. On the OU Supercomputer, this is done as follows:
. /home/fagg/tf_setup.sh
conda activate dnn
- Configure your NEURO_REPOSITORY_PATH. The location will depend your local configuration. On the OU Supercomputer:
export NEURO_REPOSITORY_PATH=/home/fagg/zero2neuro
- Change your current working directory to the location of the example. If you are using the XOR example:
cd examples/xor
- Execute Your Experiment
python $NEURO_REPOSITORY_PATH/zero2neuro/src/zero2neuro.py @network.txt @data.txt @experiment.txt -vvv
- @network.txt: load the arguments from the network.txt file
- @data.txt: load the arguments from data.txt
- @experiment.txt: load experiment.txt
- -vvv: be very verbose about reporting the state of the experiment
For this experiment, the Zero2Neuro engine will generate text output
Training Outputs
The outputs are placed in the results/ subdirectory:
- xor_R00_model.keras: reloadable model
- xor_R00_model_plot.png: image that describes the model
- xor_R00_report.xlsx: report file in excel format
- xor_R00_results.pkl: pickle file that contains the model training information and predictions for the training set examples. This file can be read using a Python program