Gathering infomation¶
Assist
is the only class in ExAssist framework.
This section provides basic usage of Assist
.
Create an Assist¶
In order to use ExAssist to assist your experiment, you first need to instantiate an intance of Assist
:
import ExAssist as EA
assist = EA.getAssist('Test')
When calling getAssist
method, ExAssist will instantiate a new Assist
instance if it does not exist.
The name Test
is the unique identifier for the new Assist
instance.
Just like logging you can access the same instance anywhere in your source code:
assist = EA.getAssist('Test')
After getting an instance of Assist
, you need to provide
the following information:
The root path of your experiemnt records.(default:
./Experiments/
)A config variable that keeps all the configurations of your experiments. This config variable could be:
- A
dict
object that contains all the configurations as (key, value)- A
argparse.namespace
object. This namespace object should contains all the configurations.- A
ConfigParser
object that loads configurations from config files.
A simple example is like this:
# Get the Assist instance
assist = EA.getAssist('Test')
# Setup the root path of experiment records
assist.ex_dir = 'tests/Experiments/'
# Setup the config variable
assist.set_config(config)
Once setting up all the informaion, you can start your experiments.
Observe an Experiment¶
The first function of ExAssist is to observe en experiment automatically. ExAssist uses context manager to observe your experiment:
with EA.start(assist) as assist:
# Your experiment happens here
- When you entering this context, ExAssist will automatically:
- Create a uniqe directory which will be used to save all the information about this experiment.
- Gather meta information about your experiment, like your starting time and environment information.
Note
Once entering the context, you can not modify the basic information about Assist
covered in last section, see Create an Assist.
- When you finish running your experiment and leaving this context, ExAssist will automatically:
- Record the status (success, failed or interrupted) of this experiment
- Generate an html file that contains all the information of this experiment.
ExAssist collects lots of information about an experiment:
- time it was started, time it stopped and cpu time it used.
- the used configuration
- status of this experiment
- basic information about the machine it runs on
- packages the experiment depends on and their versions
- data added with
assist.info
- data added with
assist.result
Directory Structure¶
All the information (except the last two points) above is gathered and saved automatically, you don’t need to write any code.
For each experiment running, ExAssist will create a new sub-directory in the path of ex_dir
and stores several files in there:
Experiments/
├── 0
│ ├── config.json
│ ├── index.html
│ ├── info.json
│ └── run.json
├── 1
│ ├── config.json
│ ├── index.html
│ ├── info.json
│ └── run.json
As we can see above, ExAssist will also generate a report (index.html
) for each run.
Assist an Experiment¶
The second function of ExAssist is to assist your experiment. It gives the abilities:
- Record the running information without writing extra IO functions. ExAssis can help you save all the temporary information during the experiment, such as loss and gradients.
import ExAssis as EA assist = EA.getAssist('Test') with EA.start(assist) as assist: # Here starts your experiments. for i in range(100): assist.info['loss'] = 100 - i assist.step()In the code above, we record
loss
value for each iteration. Methodstep()
tells ExAssist that the current iteration is finished.assis.info
is dictionary which means you can put anything you want into this variable. Theinfo
dictionary is meant to store temporary information about the experiment, like training loss for each epoch or the total number of parameters. It is updated once you invokestep
method. You can add whatever information you like toinfo
. Code in the above will generate a list like this:[{'loss':100}, {'loss':99}, {'loss':98}, ...]Once you entering the context, you can access and update following variables:
assist.info
: You can useinfo
to save any temporary value that you need to analysis, like traning loss.assist.result
:result
are designed to keep the evaluation results of this experiment.result
does not affeced bystep()
method.assist.run_path
: Read-only. You can access the path of current experiment data. This is useful when you want to save your model in the same directory with its meta information.assist.epoch
: Read-only. Indicates the internal epoch number of ExAssist. It increases every time when you invokestep()
method.
Deactivate¶
When publishing the code, you usually do not want ExAssist to observe any experiments. You can deactivate ExAssist by:
import ExAssis as EA assist = EA.getAssist('Test') assist.deactivate() with EA.start(assist) as assist: # Here starts your experiments. for i in range(100): assist.info['loss'] = 100 - i assist.step()
By invoking deactivate()
, ExAssis will not do anything
during run as if it does not exist.