File Loader
Data Format
Data can be loaded from a set of different files. However, all files in the set must be of the same Data File Format. The format is specified using the following argument:
--data_format=XXX
where XXX is one of tabular (csv or xlsx), tabular-indirect, (csv or xlsx), pickle, tf-dataset
Data File location
Location of the set of data files:
--data_set_directory=PATH
where PATH is a relative or absolute path to the set of data files (default is the local directory).
Data Files
File name(s) can be specified in one of two ways:
Single File
--data_file=YYY
where YYY is the name of a file relative to the data_set_directory
Multiple Files
--data_files
YYY0
YYY1
YYY2
:
where YYY0, YYY1, … is a list of file names.
Multiple Files with Fold Assignment
--data_files
YYY0,F0
YYY1,F1
YYY2,F2
:
where YYY0, YYY1, … is a list of file names, and F0, F1, … are the integer fold numbers (0, 1, …) that the contents of each file may be assigned to (see Data Set Generator for more detail).
Selecting Fields from Files
For tabular, tabular-indirect, and pickle formatted files, one must specify which of the fields within the file to use for different purposes. For tabular files, the fields are different columns.
Data Inputs
The list of fields from the file to include as inputs to the model
--data_inputs
in0
in1
:
Notes:
- tabular, tabular-indirect: specifies the columns in the table to include
- pickle: specifies the keys from the dictionary to include
Data Outputs
The list of fields from the file to include as the desired outputs values from the model
--data_outputs
out0
out1
:
Notes:
- tabular, tabular-indirect: specifies the columns in the table to include
- pickle: specifies the keys from the dictionary to include
Data Weights (Advanced; optional)
Each example in a dataset can be weighted for the purposes of computing the loss function. In unbalanced data sets, the common cases can dominate the loss, resulting in a model that essentially ignores the rare cases. When using weights, the rare cases should be assigned a weight of 1, and the common cases should be assigned a low weight (e.g., the ratio of rare to common). All weights should fall within the range 0…1.
--data_weights=FIELD
where FIELD is the string name of the field that contains the weights.
Data Groups (Advanced; optional)
The field to be used to possibly determine the fold that each example belongs to.
--data_weights=FIELD
The values in FIELD are natural numbers (integers: 0, 1, ….). See Data Set Generator for more detail.
Advanced Topics
- Options for Tabular Files: Fine-level control for reading tabular files
- Data Translation: Translating categorical variables (including strings) into machine learning-ready representations