lognflow Module contents

lognflow

lognflow

lognflow makes logging easy in Python. It is so simple you can code it yourself, so, why would you?!

lognflow logs all files into a directory by taking care of directories and files names. This saves you a lot of coding and makes your code readable

when you say:

logger = lognflow(logs_root = 'root_for_time_tagged_log_directories')
logger.log_single('variables/variable1', variable1)
logger('I just logged a variable.')

another_logger = lognflow(log_dir = 'specific_dir')
another_logger.log_plot('final_plot', final_plot_is_a_np_1d_array)

The next syntax is an easy way of just logging a numpy array. It will make a new directory within the log_dir, called variables and make a npy file named variable1 and put variable1 in it. The third line of the code above prints the given text to the __call__ routine in the main txt file made in the log_dir.

As you can see, first you give it a root (logs_root) to make a log directory in it or give it the directory itself (log_dir). Then start dumping data by giving the variable name and the data with the type and you are set.

Multiple processes in parallel can make as many instances as they want.

There is an option to keep the logged variables in memory for a long time and then dump them when they reach a ceratin size. This reduces the network load.

for this the txt logs can be buffered for a chosable amount of time and numpy variables that don’t change size can be buffered up to a certain size before storing into the directory using log_var(name, var).

class lognflow.lognflow.getLogger(logs_root: Path | None = None, log_dir: Path | None = None, log_dir_prefix: str | None = None, log_dir_suffix: str | None = None, exist_ok: bool = True, time_tag: bool | str = True, print_text: bool = True, main_log_name: str = 'log', log_flush_period: int = 10, enabled: bool = True)

Bases: object

Initialization

lognflow.getLogger creates a directory called log_dir and puts all logs in there.

Where? 1: if logs_root is given, it makes a log_dir in it with a time_stamp. 2: if log_dir is given, it uses it directly. 3: If you type:

logger = lognflow()

it will try to open a dialog to select a directory, if error occurs, it will get a temp directory from the os and continues.

The lognflow allows setting global settings that can be overridden later by calling each of its methods as follows.

Parameters:

• logs_root (pathlib.Path) – This is the root directory for all logs. We will use the time.time() to create a log directory for each instance of the lognflow.

• log_dir (pathlib.Path) – This is the final directory path for the log files.

• log_prefix (str) – this string will be put before the time tag for log_dir, when only logs_root is given.

• log_suffix – if given, time tag will not be used and this string will be put at the end of the log_dir name.

• exist_ok – if False, if any logging directory exists, it raises an error.

• print_text (bool) – If True, everything that is logged as text will be printed as well

• main_log_name (str) – main log file name, by default: ‘main_log’

• log_flush_period (int) – The period between flushing the log files into HDD. By not flushing, you can reduce network or HDD overhead.

• time_tag (bool) – File names can carry time_tags in time.time() format or indices. This is pretty much the most fundamental contribution of lognflow beside carrying the folders and files paths around. By default it is True and all file names will not have time tags if you set it to False. so, you can give time_tag argument for all logger functions, whose default is this default. It can also be a string: options are ‘index’, ‘time_and_index’ and ‘keep_initial’. If you use ‘index’, instead of time stamps, it will simply put an index that counts up after each logging. If you use ‘time_and_index’, it will use both time_tag and an index. If you use ‘keep_initial’, it will save the first instance with suffix ‘_initial’ and then it will make a new one for the next instance and update that file onwards.

assert_log_dir()

copy(parameter_name=None, source=None, suffix=None, time_tag=False)

copy into a new file Given a parameter_name, the second argument will be copied into the first. We will try syntaxes os_system(‘cp’) and ‘copy’ for Windows.

Parameters:

• parameter_name – str examples: myvar or myscript/myvar parameter_name can be just a name e.g. myvar, or could be a path like name such as myscript/myvar.

• source – str if source.is_file() then it is copied into its new location. Otherwise, we use logger.get_flist(source, suffix) to obtain a list of files matching the source and copy them into their new location.

critical()

debug(text_to_log)

disable()

enable()

error()

exception()

file_from_name(parameter_name)

file from name given a parameter_name, it returns log_dir / parameter_name

flush_all()

get_common_files(var_name_A, var_name_B, suffix=None, flist_A=None, flist_B=None)

get common files in two directories

It happens often in ML that there are two directories, A and B, and we are interested to get the flist in both that is common between them. returns a tuple of two lists of files.

Parameters

param var_name_A:

directory A name

param var_name_B:

directory B name

get_flist(var_name, suffix=None)

get list of files return the list of files for a saved variable.

Parameters

param var_name:

variable name

param suffix:

If there are different suffixes availble for a variable this input needs to be set. npy, npz, mat, and torch are supported.

get_namelist(var_name, suffix=None)

get logger names of files return the list of names for a saved variable.

Parameters

param var_name:

variable name

param suffix:

If there are different suffixes availble for a variable this input needs to be set. npy, npz, mat, and torch are supported.

get_record(parameter_name: str, suffix: str | None = None) → tuple

Get the buffered numpy arrays If you need the buffered variable back. :param parameter_name: str

examples: myvar or myscript/myvar

parameter_name can be just a name e.g. myvar, or could be a path like name such as myscript/myvar.

Returns:

A tuple including two np.ndarray. The first on is 1d time and the second one is nd buffered data.

Return type:

tuple of two nd.arrays

get_stack_from_files(var_name=None, flist=[], suffix=None, read_func=None, return_flist=False)

Get list or data of all files in a directory

This function gives the list of paths of all files in a directory for a single variable.

Parameters

param var_name:

The directory or variable name to look for the files

type var_name:

str

param flist:

list of Paths, if data is returned, this flist input can limit the data requested to this list.

type flist:

list

param suffix:

the suffix of files to look for, e.g. ‘txt’

type siffix:

str

param read_func:

the function that takes the posix path of a file and returns the data in there.

Output

It returns a list of data in all files or a numpy array if concatenation of all is possible.

get_stack_from_names(var_names=None, read_func=None, return_flist=False)

get_text(log_name='main_log', flist=None, suffix='txt', file_index=-1)

get text log files Given the log_name, this function returns the text therein.

Parameters

param log_name:

the log name. If not given then it is the main log.

param flist:

you can give a file list in Posix paths, for text files

param suffix:

str to search for specifi files

param file_index:

int or list[int] a number or a list of numbers for the index of the file to include, default: -1

hexbin(parameter_name: str, parameter_value, gridsize=20, image_format='jpg', dpi=1200, title=None, time_tag: bool | None = None, return_figure=False)

log a 2D histogram The 2D histogram is made out of hexagonals

Parameters:

• parameter_name – str examples: myvar or myscript/myvar parameter_name can be just a name e.g. myvar, or could be a path like name such as myscript/myvar.

• parameter_value – np.array An np array of size 2 x n, to make the 2D histogram

• gridsize – int grid size is the number of bins in 2D

• time_tag – bool Wheather if the time stamp is in the file name or not.

hist(parameter_name: str, parameter_value_list, bins=10, alpha=0.5, labels_list=None, normalize=False, image_format='jpg', dpi=1200, title=None, time_tag=None, return_figure=False, **kwargs)

log a single histogram If you have a numpy array or a list of arrays (or indexable by first dimension, an array of 1D arrays), use this to log a hist if multiple inputs are given they will be plotted on top of each other using the alpha opacity.

Parameters:

• parameter_name – str examples: myvar or myscript/myvar parameter_name can be just a name e.g. myvar, or could be a path like name such as myscript/myvar.

• parameter_value_list – np.array An np array or a list of np arrays or indexable-by-0th-dim np arrays

• bins – number or np.array used to set the bins for making of the histogram

• alpha – float the opacity of histograms, a flot between 0 and 1. If you have multiple histograms on top of each other, use 1/number_of_your_variables.

• time_tag – bool Wheather if the time stamp is in the file name or not.

images_to_pdf(parameter_name: str, parameter_value: list, time_tag: bool | None = None, dpi=1200, **kwargs)

imshow(parameter_name: str, parameter_value, frame_shape: tuple | None = None, colorbar=True, remove_axis_ticks=True, image_format='jpg', dpi=1200, cmap='viridis', title=None, time_tag: bool | None = None, borders=0, return_figure=False, figsize=None, **kwargs)

log an image The image is logged using plt.imshow Accepted shapes are:

• (n, m)

• (n, m, 3)

• (n_im, n_r, n_c)

• (n_im, n_r, 3, 1)

• (n_im, n_r, n_c, 3)

Parameters:

• parameter_name – str examples: myvar or myscript/myvar parameter_name can be just a name e.g. myvar, or could be a path like name such as myscript/myvar.

• parameter_value – np.array An np array of shape amongst the following: * (n, m) * (n, m, 3) * (n_im, n_r, n_c) * (n_im, n_r, 3, 1) * (n_im, n_r, n_c, 3)

• time_tag – bool Wheather if the time stamp is in the file name or not.

imshow_series(parameter_name: str, list_of_stacks, list_of_masks=None, figsize=None, text_as_colorbar=False, colorbar=False, cmap='viridis', list_of_titles_columns=None, list_of_titles_rows=None, fontsize=None, vmin=None, vmax=None, title=None, colorbar_last_only=True, colorbar_fraction=0.046, colorbar_pad=0.04, colorbar_labelsize=1, grid_width_space=0.0, remove_axis_ticks=True, aspect='equal', image_format='jpg', dpi=1200, time_tag: bool | None = None, return_figure=False, **kwargs)

log a cavas of stacks of images One way to show many images and how they change is to make stacks of images and put them in a list. Then each element of the list is supposed to be iteratable by the first dimension, which should be the same size for all elements in the list. This function will start putting them in rows of a canvas. If you have an image with many channels.

Each element of the list must appear as either: n_frm x n_row x n_clm if there are n_frm images

for all elements of stack

n_frm x n_row x n_clm x 3 if channels are in RGB

if you have multiple images as channels such as the following, call the prepare_stack_of_images.

Parameters:

parameter_name – str examples: myvar or myscript/myvar parameter_name can be just a name e.g. myvar, or could be a path like name such as myscript/myvar.

Displays a grid of image series for comparison with optional customization for annotations, colorbars, and formatting.

Parameters:

list_of_stacks (list):

A list of 3D or 4D arrays, each representing a stack of images. All stacks must have the same number of images.

list_of_masks (list, optional):

A list of masks corresponding to the stacks. Each mask should have the same shape as the images in its respective stack. If provided, masked areas will be ignored when calculating statistics. Defaults to None.

figsize (tuple, optional):

The overall size of the figure in inches. If None, it is determined based on the number of stacks and images. Defaults to None.

text_as_colorbar (bool, optional):

If True, displays the maximum, mean, and minimum values of each image as text in place of a colorbar. Defaults to False.

colorbar (bool, optional):

If True, displays a colorbar for each subplot. Defaults to False.

cmap (str, optional):

The colormap to use for displaying the images. Defaults to ‘viridis’.

list_of_titles_columns (list, optional):

Titles for each column in the grid. Must have a length equal to the number of images in each stack. Defaults to None.

list_of_titles_rows (list, optional):

Titles for each row in the grid. Must have a length equal to the number of stacks. Defaults to None.

fontsize (int, optional):

Font size for the text annotations. If None, it is determined based on the figure size. Defaults to None.

vmin (float, optional):

The minimum value for image normalization. If None, it is automatically calculated from the image data. Defaults to None.

vmax (float, optional):

The maximum value for image normalization. If None, it is automatically calculated from the image data. Defaults to None.

title (str, optional):

The title for the entire figure. Defaults to None.

colorbar_last_only (bool, optional):

If True, displays a colorbar only for the last column. Defaults to False.

colorbar_fraction (float, optional):

Fraction of the original axis allocated for the colorbar. Defaults to 0.046.

colorbar_pad (float, optional):

Padding between the image and colorbar. Defaults to 0.04.

colorbar_labelsize (int, optional):

Label size for the colorbar. Defaults to 1.

grid_width_space (float, optional):

Horizontal spacing between grid columns. Defaults to 0.0.

remove_axis_ticks (bool, optional):

If True, removes axis ticks from all subplots. Defaults to True.

aspect (str, optional):

Aspect ratio of the displayed images. Defaults to ‘equal’.

**kwargs:

Additional keyword arguments to pass to the imshow function.

Parameters:

time_tag – bool Wheather if the time stamp is in the file name or not.

imshow_subplots(parameter_name: str, images: ndarray, frame_shape=None, grid_locations=None, figsize=None, image_format='jpg', dpi=1200, time_tag: bool | None = None, colorbar=False, remove_axis_ticks=True, titles=None, cmaps=None, return_figure=False, **kwargs)

log multiple images in a tiled frame The frame image is logged using plt.imshow

Accepted shapes are:

• (n, m)

• (n, m, 3)

• (n_im, n_r, n_c)

• (n_im, n_r, 3, 1)

• (n_im, n_r, n_c, 3)

Parameters:

• parameter_name – str examples: myvar or myscript/myvar parameter_name can be just a name e.g. myvar, or could be a path like name such as myscript/myvar.

• images – np.array An np array of size n_f x n_r x n_c, to be shown by imshow as a square tile of side length of n_ch**0.5

• frame_shape – n_f images will be tiles according to thi tuple as shape.

• grid_locations – if this is of shape n_images x 2, then each subplot will be located at a specific given location. To make it beautiful, you better proveide figsize and im_sizes or im_size_factor to merely scale them to cover a small region between 0 and 1.

• time_tag – bool Wheather if the time stamp is in the file name or not.

info()

is_file(var_name, file_index=None, suffix=None, verbose=False)

check if a single variable file exists returns True is available else returns False

Parameters

param var_name:

variable name

param file_index:

If there are many snapshots of a variable, this input can limit the returned to a set of indices.

param suffix:

If there are different suffixes availble for a variable this input needs to be set. npy, npz, mat, and torch are supported.

load(var_name, file_index=-1, suffix=None, read_func=None, verbose=False, return_fpath=False, return_collection=False)

get a single variable return the value of a saved variable.

Parameters

param var_name:

variable name

param file_index:

If there are many snapshots of a variable, this input can limit the returned to a set of indices.

param suffix:

If there are different suffixes availble for a variable this input needs to be set. npy, npz, mat, and torch are supported.

param read_func:

a function that takes the Posix path and returns data

Note

when reading a MATLAB file, the output is a dictionary. Also when reading a npz except if it is made by record

load_configs()

load_torch(name)

log_animation(parameter_name: str, stack, interval=50, blit=False, repeat_delay=None, dpi=100, time_tag: bool | None = None)

Make an animation from a stack of images

Parameters:

• parameter_name – str examples: myvar or myscript/myvar parameter_name can be just a name e.g. myvar, or could be a path like name such as myscript/myvar.

• stack – np.array of shape n_f x n_r x n_c or n_f x n_r x n_c x 3 stack[cnt] needs to be plotable by plt.imshow()

• time_tag – bool Wheather if the time stamp is in the file name or not.

log_code(code_fpath=None)

log code, pass __file__

log_confusion_matrix(parameter_name: str, cm, target_names=None, title='Confusion matrix', cmap=None, figsize=None, image_format='jpg', dpi=1200, time_tag=False, close_plt=True)

log a confusion matrix given a sklearn confusion matrix (cm), make a nice plot

Parameters:

• cm – confusion matrix from sklearn.metrics.confusion_matrix

• target_names – given classification classes such as [0, 1, 2] the class names, for example: [‘high’, ‘medium’, ‘low’]

• title – the text to display at the top of the matrix

• cmap – the gradient of the values displayed from matplotlib.pyplot.cm (http://matplotlib.org/examples/color/colormaps_reference.html) plt.get_cmap(‘viridis’) or plt.cm.Blues

• time_tag – if True, the file name will be stamped with time

Usage::

from lognflow import lognflow logger = lognflow(log_roots or log_dir) logger.plot_confusion_matrix( cm = cm, # confusion matrix created by

# sklearn.metrics.confusion_matrix

target_names = y_labels_vals, # list of names of the classes title = best_estimator_name) # title of graph

Credit

http://scikit-learn.org/stable/auto_examples/

model_selection/plot_confusion_matrix.html

name_from_file(fpath)

Given an fpath inside the logger log_dir, what would be its equivalent parameter_name?

plot(parameter_name: str, parameter_value_list, *plt_plot_args, x_values_list=None, image_format='jpg', dpi=1200, title=None, labels=[], time_tag: bool | None = None, fig_ax=None, return_figure=False, **kwargs)

log a single plot If you have a numpy array or a list of arrays (or indexable by first dimension, an array of 1D arrays), use this to log a plot

Parameters:

• parameter_name – str examples: myvar or myscript/myvar parameter_name can be just a name e.g. myvar, or could be a path like name such as myscript/myvar.

• parameter_value_list – np.array An np array or a list of np arrays or indexable-by-0th-dim np arrays

• x_values_list – np.array if set, must be a list of one or many np.array of same size of all y values or a list for each vector in y values where every element of x-values list is the same as the y-values element in their list

• time_tag – bool Wheather if the time stamp is in the file name or not.

printv(var, **kwargs)

record(parameter_name: str, parameter_value, flush=False, suffix=None, log_size_limit: int = 100000000, savefig=False, plot_start_ago=None, plot_win_length=10, time_tag=False)

log a numpy array in buffer then dump It can be the case that we need to take snapshots of a numpy array over time. The size of the array would not change and this is hoing to happen frequently. This log_ver makes a buffer in RAM and keeps many instances of the array along with their time stamp and then when the size of the array reaches a threhshold flushes it into HDD with a file that has an initial time stamp. The benefit of using this function over save is that it does not use the connection to the directoy all time and if that is on a network, there will be less overhead.

Parameters:

• parameter_name – str examples: myvar or myscript/myvar parameter_name can be just a name e.g. myvar, or could be a path like name such as myscript/myvar.

• parameter_value – np.array An np array whose size doesn’t change

• suffix – str can be ‘npz’ or ‘txt’ which will save it as text.

• log_size_limit – int log_size_limit in bytes, default: 1e+8.

record_flush(parameter_name: str, suffix: str | None = None)

Flush the buffered numpy arrays If you have been using log_ver, this will flush all the buffered arrays. It is called using log_size_limit for a variable and als when the code that made the logger ends. :param parameter_name: str

examples: myvar or myscript/myvar

parameter_name can be just a name e.g. myvar, or could be a path like name such as myscript/myvar.

rename(new_name: str, append: bool = False)

renaming the log directory It is possible to rename the log directory while logging is going on. This is particulary useful when at the end of an experiment, it is necessary to put some variables in the name of the directory, which is very realistic in the eyes of an experimentalist.

There is only one input and that is the new name of the directory.

Parameters:

• new_name (str) – The new name of the directory (without parent path)

• append (bool) – keep the time tag for the folder and append it to the right side of the new name. Default: False.

replace_time_with_index(var_name, verbose=False)

index in file var_names lognflow uses time stamps to make new log files for a variable. That is done by putting time stamp after the name of the variable. This function changes all of the time stamps, sorted ascendingly, by indices.

Parameters:

var_name – variable name

save(parameter_name: str, parameter_value, suffix=None, mat_field=None, time_tag: bool | None = None, verify=False)

log a single variable The most frequently used function would probably be this one.

if you call the logger object as a function and give it a parameter name and something to be logged, the __call__ referes to this function.

Parameters:

• parameter_name – str examples: myvar or myscript/myvar parameter_name can be just a name e.g. myvar, or could be a path like name such as myscript/myvar.

• parameter_value – np.array Could be anything and np.save will be used. If it is a dictionary, np.savez will be used. As you may know, np.save can save all pickalables.

• suffix – str can be ‘npz’, ‘npy’, ‘mat’, ‘pth’ for pytorch models or ‘txt’ or anything else which will save it as text. This includes ‘json’, ‘pdb’, or …

• mat_field – str when saving as ‘mat’ file, the field can be set. otherwise it will be the parameter_name

• time_tag – bool Wheather if the time stamp is in the file name or not.

save_configs(configs_dict, max_array_size=256)

save_torch(name, x)

savefig(parameter_name: str, image_format='jpg', dpi=1200, time_tag: bool | None = None, close_plt=True)

log a single plt log a plt that you have on the screen.

Parameters:

• parameter_name – str examples: myvar or myscript/myvar parameter_name can be just a name e.g. myvar, or could be a path like name such as myscript/myvar.

• time_tag – bool Wheather if the time stamp is in the file name or not.

savez(parameter_name: str, parameter_value, time_tag: bool | None = None)

scatter3(parameter_name: str, data_N_by_3, elev_list=None, azim_list=None, image_format='jpg', dpi=300, title=None, time_tag: bool | None = None, return_figure=False, make_animation=False, log_animation_kwargs={}, **kwargs)

log a single scatter in 3D Scatter plotting in 3D

Parameters:

• parameter_name – str examples: myvar or myscript/myvar parameter_name can be just a name e.g. myvar, or could be a path like name such as myscript/myvar.

• data_N_by_3 – np.array An np array of size 3 x n, to sctter n data points in 3D

• elev_list – list Must be an iterable even if has only one number for elev

• azim_list – list Must be an iterable even if has only one number for azim

• time_tag – bool Wheather if the time stamp is in the file name or not.

setLevel(level='info.txt')

surface(parameter_name: str, parameter_value, image_format='jpg', dpi=1200, title=None, time_tag: bool | None = None, return_figure=False, **kwargs)

log a surface in 3D surface plotting in 3D exactly similar to imshow but in 3D

Parameters:

• parameter_name – str examples: myvar or myscript/myvar parameter_name can be just a name e.g. myvar, or could be a path like name such as myscript/myvar.

• parameter_value – np.array An np array of size n x m, to plot surface in 3D

• time_tag – bool Wheather if the time stamp is in the file name or not.

rest of the parameters (**kwargs) will be passed to plot_surface()

text(log_name: str | None = None, to_be_logged='', log_time_stamp=True, print_text=None, log_size_limit: int = 10000000, time_tag: bool | None = None, log_flush_period: int | None = None, flush=False, end='\n', new_file=False, suffix=None)

log a string into a text file You can shose a name for the log and give the text to put in it. Also you can pass a small numpy array. You can ask it to put time stamp in the log and in the log file name, you can disable printing the text. You can set the log size limit to split it into another file with a new time stamp.

Parameters:

• log_name – str examples: mylog or myscript/mylog log_name can be just a name e.g. mylog, or could be a pathlike name such as myscript/mylog.

• to_be_logged – str, nd.array, list, dict the string to be logged, could be a list or numpy array or even a dictionary. It uses str(…).

• log_time_stamp – bool Put time stamp for every entry of the log

• print_text – bool if False, what is logged will not be printed.

• log_size_limit – int log size limit in bytes.

• time_tag – bool put time stamp in file names.

• log_flush_period – int How often flush the log in seconds, if time passes this given period, it will flush the first time a text is logged, or if the logger is finilized.

• flush – bool force flush into the log file

• end – str The last charachter for this call.

• new_file – bool if a new file is needed. If time_tag is True, it will make a new file with a new name that has a time tag. If False, it closees the current text file and overwrites on it.

• suffix – str suffix is the extension of the file name.

text_flush(log_name=None, flush=False, suffix=None)

Flush the text logs Writing text to open(file, ‘a’) does not constantly happen on HDD. There is an OS buffer in between. This funciton should be called regularly. lognflow calls it once in a while when text is called multiple times. but use needs to also call it once in a while. In later versions, a timer will be used to call it automatically. :param flush:

force the flush regardless of when the last time was. default: False

property time_stamp

Current time stamp Gives the time after the start of the lognflow

variables_to_pdf(parameter_name: str, parameter_value: list, time_tag: bool | None = None, dpi=1200, **kwargs)

warning()

class lognflow.lognflow.textinlog(to_be_logged: str, log_fpath: pathlib.Path, log_size_limit: int, log_size: int, last_log_flush_time: float, log_flush_period: int)

Bases: object

last_log_flush_time: float

log_flush_period: int

log_fpath: Path

log_size: int

log_size_limit: int

to_be_logged: str

class lognflow.lognflow.varinlog(data_array: numpy.ndarray, time_array: numpy.ndarray, curr_index: int, file_start_time: float, suffix: str, log_counter_limit: int, savefig: bool, plot_start_ago: float, plot_win_length: float, time_tag: bool)

Bases: object

curr_index: int

data_array: ndarray

file_start_time: float

log_counter_limit: int

plot_start_ago: float

plot_win_length: float

savefig: bool

suffix: str

time_array: ndarray

time_tag: bool

logviewer

class lognflow.logviewer.logviewer(log_dir: ~pathlib.Path, logger=<built-in function print>, not_exist_ok=False)

Bases: object

log viewer Since lognflow makes lots of files and folders, maybe it is nice to have a logviewer that loads those information. In this module we provide a set of functions for a logged object that can load variables, texts, file lists and etc.. Use it simply by:

from lognflow import logviewer
logged = logviewer(log_dir = 'dir_contatining_files')
var = logged.get_single('variable_name')

assert_log_dir()

disable_logger()

get_common_files(var_name_A, var_name_B, suffix=None, flist_A=None, flist_B=None)

get common files in two directories

It happens often in ML that there are two directories, A and B, and we are interested to get the flist in both that is common between them. returns a tuple of two lists of files.

Parameters

param var_name_A:

directory A name

param var_name_B:

directory B name

get_flist(var_name, suffix=None)

get list of files return the list of files for a saved variable.

Parameters

param var_name:

variable name

param suffix:

If there are different suffixes availble for a variable this input needs to be set. npy, npz, mat, and torch are supported.

get_namelist(var_name, suffix=None)

get logger names of files return the list of names for a saved variable.

Parameters

param var_name:

variable name

param suffix:

If there are different suffixes availble for a variable this input needs to be set. npy, npz, mat, and torch are supported.

get_single(var_name, file_index=-1, suffix=None, read_func=None, verbose=False, return_fpath=False)

get a single variable return the value of a saved variable.

Parameters

param var_name:

variable name

param file_index:

If there are many snapshots of a variable, this input can limit the returned to a set of indices.

param suffix:

If there are different suffixes availble for a variable this input needs to be set. npy, npz, mat, and torch are supported.

param read_func:

a function that takes the Posix path and returns data

Note

when reading a MATLAB file, the output is a dictionary. Also when reading a npz except if it is made by log_var

get_stack_from_files(var_name=None, flist=[], suffix=None, read_func=None)

Get list or data of all files in a directory

This function gives the list of paths of all files in a directory for a single variable.

Parameters

param var_name:

The directory or variable name to look for the files

type var_name:

str

param flist:

list of Paths, if data is returned, this flist input can limit the data requested to this list.

type flist:

list

param suffix:

the suffix of files to look for, e.g. ‘txt’

type siffix:

str

param read_func:

the function that takes the posix path of a file and returns the data in there.

Output

It returns a list of data in all files or a numpy array if concatenation of all is possible.

get_stack_from_names(var_names=None, read_func=None, return_flist=False)

get_text(log_name='main_log', flist=None, suffix='txt', file_index=-1)

get text log files Given the log_name, this function returns the text therein.

Parameters

param log_name:

the log name. If not given then it is the main log.

param flist:

you can give a file list in Posix paths, for text files

param suffix:

str to search for specifi files

param file_index:

int or list[int] a number or a list of numbers for the index of the file to include, default: -1

name_from_file(fpath)

Given an fpath inside the logger log_dir, what would be its equivalent parameter_name?

replace_time_with_index(var_name, verbose=False)

index in file var_names lognflow uses time stamps to make new log files for a variable. That is done by putting time stamp after the name of the variable. This function changes all of the time stamps, sorted ascendingly, by indices.

Parameters:

var_name – variable name

printprogress

lognflow.printprogress.needs_s(number)

class lognflow.printprogress.printprogress(int_or_iterable, numTicks=78, title=None, desc=None, method='linear', print_function=<built-in function print>, ETA_only=False, **print_function_kwargs)

Bases: object

While there are packages that use

to show a progress bar,

there are cases e.g. a print_function or an ssh terminal that does not support

. In such cases, if something is typed at the end of

the line, it cannot be deleted. The following code provides a good looking progress bar with a simple title and limits and is very simple to use. Define the object with number of steps of the loop and then call it in every iteration of the loop. If you’d like it to go faster, call it with give number of steps that have passed.