reda.utils package¶
Utility functions
Submodules¶
reda.utils.data module¶
reda.utils.datetimes module¶
reda.utils.decorators_and_managers module¶
Utility class for context managers and decorators
- class reda.utils.decorators_and_managers.LogDataChanges(container, filter_action='default', filter_query='')[source]¶
Bases:
object
Context manager that observes the DataFrame of a data container for changes in the number of rows.
Examples
>>> from reda.testing.containers import get_simple_ert_container_nor >>> ERTContainer = get_simple_ert_container_nor() >>> from reda.containers.ERT import LogDataChanges >>> with LogDataChanges(ERTContainer): ... # now change the data ... ERTContainer.data.loc[0, "r"] = 22 ... ERTContainer.data.query("r < 10", inplace=True) >>> # ERTContainer.print_log() 2... - root - INFO - Data size changed from 22 to 21
reda.utils.eit_fzj_utils module¶
- reda.utils.eit_fzj_utils.apply_correction_factors(df, correction_file)[source]¶
Apply correction factors for a pseudo-2D measurement setup. See Weigand and Kemna, 2017, Biogeosciences, for more information:
https://doi.org/10.5194/bg-14-921-2017
- Parameters:
- df
pandas.DataFrame
Data container
- correction_filestring
Path to correction file. The file must have 5 columns: a,b,m,n,correction_factor
- df
- Returns:
- corr_dataNx5
numpy.ndarray
Correction files as imported from the file. Columns: a,b,m,n,correction_factor
- corr_dataNx5
- reda.utils.eit_fzj_utils.check_resistor_board_measurements(data_file, reference_data_file=None, create_plot=True, **kwargs)[source]¶
To check basic system function a test board was built with multiple resistors attached to for connectors each. Measurements can thus be validated against known electrical (ohmic) resistances.
Note that the normal-reciprocal difference is not yet analyzed!
The referenc_data_file should have the following structure: The file contains the four-point spreads to be imported from the measurement. This file is a text file with four columns (A, B, M, N), separated by spaces or tabs. Each line denotes one measurement and its expected resistance, the allowed variation, and its allow difference towards its reciprocal counterpart:
1 2 4 3 1000 1 20 4 3 2 1 1000 1 20
test frequency: 1Hz
- Parameters:
- data_filestring
path to mnu0 data file
- reference_data_file: string, optional
path to reference data file with structure as describe above. Default data is used if set to None
- create_plotbool, optional
if True, create a plot with measured and expected resistances
- kwargsdict, optional
kwargs will be redirected to the sEIT.import_eit_fzj call
- Returns:
- figfigure object, optional
if create_plot is True, return a matplotlib figure
- reda.utils.eit_fzj_utils.compute_correction_factors(data, true_conductivity, elem_file, elec_file)[source]¶
Compute correction factors for 2D rhizotron geometries, following Weigand and Kemna, 2017, Biogeosciences
https://doi.org/10.5194/bg-14-921-2017
- Parameters:
- data
pandas.DataFrame
measured data
- true_conductivityfloat
Conductivity in S/m
- elem_filestring
path to CRTomo FE mesh file (elem.dat)
- elec_filestring
path to CRTomo FE electrode file (elec.dat)
- data
- Returns:
- correction_factorsNx5 :py:class.`numpy.ndarray`
measurement configurations and correction factors (a,b,m,n,correction_factor)
- reda.utils.eit_fzj_utils.get_md_data_2018a(filename)[source]¶
Return the md data of a given FZJ EIT 2018a LI calibration data file.
This function should probably go into the importers, but for now will reside here until it can be properly integrated.
- Parameters:
- filenamestr
Path to eit_data.mat file generated for an LI-‘calibration’ run
- Returns:
- mdpandas.DataFrame
MD data
- reda.utils.eit_fzj_utils.testboard_evaluation(datapath, configdat, outputname, frequencies=array([1.00000000e-01, 1.34339933e-01, 1.80472177e-01, 2.42446202e-01, 3.25702066e-01, 4.37547938e-01, 5.87801607e-01, 7.89652287e-01, 1.06081836e+00, 1.42510267e+00, 1.91448198e+00, 2.57191381e+00, 3.45510729e+00, 4.64158883e+00, 6.23550734e+00, 8.37677640e+00, 1.12533558e+01, 1.51177507e+01, 2.03091762e+01, 2.72833338e+01, 3.66524124e+01, 4.92388263e+01, 6.61474064e+01, 8.88623816e+01, 1.19377664e+02, 1.60371874e+02, 2.15443469e+02, 2.89426612e+02, 3.88815518e+02, 5.22334507e+02, 7.01703829e+02, 9.42668455e+02, 1.26638017e+03, 1.70125428e+03, 2.28546386e+03, 3.07029063e+03, 4.12462638e+03, 5.54102033e+03, 7.44380301e+03, 1.00000000e+04]), error_percentage=1)[source]¶
A testboard with resistors and capacitors was built to test the basic operation performance of eit-systems from FZJ. This function plots the results of measurements on this board in terms of impedance magnitude and phase.
- Parameters:
- datapathstr
Path to the eit_data_mnu0.mat file containing the measurements.
- configdat: np.ndarray|txt-file|int
input configuration of the used testboard configurations, e.g. for first two rows of the board: 1 4 2 3 2 3 1 4 5 8 6 7 6 7 5 8 9 12 10 11 10 11 9 12 Note that normal and reciprocal measurements have to be measured. If this parameter is an integer, this indicates the number of electrodes used (should be a multiple of 4).
- outputname: str
output name of plot in png-format
- frequencies: numpy array
frequency range (in log10-space) to compare the measurements to; default range is from 0.1 Hz to 10 kHz
- error_percentage: float
percentage of allowed measurement error. The range inside this limit will be shown as a grey shadow in the plot.
- Returns:
- fig: figure object
Saves the plot with the given output name in the execution location of the script.
- seit: reda.sEIT
object with the measurement data
reda.utils.electrode_manager module¶
An electrode manager that deals with the relationships between electrode numbers and electrode positions. It can be used while importing data, but also to manage electrode positions for data that does not come with actual electrode positions.
## Possible future features
- ## Implemented features
[done] Generate electrode numbers from a set of electrode coordinates
[done] Add new positions, including a .assume_regular_electrodes (spacing, axis=(x|y|z)) function that conforms electrode numbers to a fixed distance of electrodes.
[done] Assume reciprocal measurement by reversing
[done] The electrode numbering should follow an intuitive heuristic, i.e. from left to right, top to bottom (think of rhizotrons or borehole settings). This behavior should be changeable by means of a custom sort function that takes (x,z) or (x,y,z) tuples and returns a given order.
[done] 2D and 3D
[done] .from_existing_ordering(iterable(numbers), iterable(coordinates))
[done] reverse lookup: given a fixed electrode assignment table, return
[done] test/account for positions not registered yet, but for which an electrode number is requested electrode positions for given electrode numbers
[done] need proper tests for all sorters
[done] how to deal with incorrect electrode positions, e.g. for the case where default Syscal settings are used?
[done] Merge two datasets by their electrode coordinates:
## User Stories
Import Syscal Data with unused electrodes but correct electrode positions
Syscal data contains only electrode positions (x,y,z for binary, but often only x coordinates for ascii export). As such we need to assign electrode numbers to those coordinates. This is relatively straightforward in case all electrodes attached to the multichannel system were used and a regular spacing was set in the system. However, if electrodes were skipped, things can get messy.
Two possible procedures can now be employed:
Add all encountered electrode positions (for a,b,m,n) to the electrode manager.
Sort the positions in ascending order for z,y,x coordinates (in ascending priority)
(optional) add missing electrode positions to conform the resulting assignments to a fixed grid of electrodes, e.g., 48, 72, or 96 electrodes.
Use the resulting electrode number assignment to generate logical electrode numbers for a,b,m,n
Provide a precomputed assignment table for electrode numbers and positions to the electrode manager
Use the resulting electrode number assignment to generate logical electrode numbers for a,b,m,n
Import Syscal Data with unused electrodes but incorrect electrode positions
Same as in (1), but use a replacement function to replace the incorrect electrode positions with the correct ones.
- class reda.utils.electrode_manager.electrode_manager(electrode_positions=None)[source]¶
Bases:
object
- Attributes:
- electrode_positions
Methods
add_by_position
(data_raw[, remove_duplicates])Add electrodes by using only their positions (1D/2D/3D).
add_fixed_assignments
(data_raw)Use a pre-determined electrode numbering scheme.
align_assignments
(pos1, pos2, abmn1, abmn2)Align the positions and logical electrode numbers of two datasets.
conform_to_regular_x_spacing
(spacing_x[, ...])Assume electrodes are located on a regular grid of electrodes with spacing spacing_x.
Return the assignment between electrode numbers and coordinates
For a given set of coordinates, return electrode coordinates
get_position_of_number
(number)Return the position associated with a given electrode number.
reflect_on_position_x
(reflect_on_x)Reflect all electrode positions on a given x position.
register_custom_ordering_scheme
(function)Register a user-supplied function to assign electrode numbers to a given set of coordinates.
Replace the coordinates of the given electrode number (in the current ordering) with the new coordinates.
Using the current ordering scheme, replace the x coordinates with multiples of spaxing_x, starting with 0.
Helper function that activates the sorter that assigns electrode numbers in the order the coordinates were registered.
Helper function that activates the sorter that assigns electrode numbers in the order the coordinates were registered.
Helper function that activates the sorter that actually sorts electrode coordinates before assigning the electrode numbers.
shift_positions_xyz
(shift_by_xyz)Shift electrode positions by adding the vector 'shift_by_xyz' to each position.
__call__
plot_coordinates_x_z_to_ax
- add_by_position(data_raw, remove_duplicates=True, **kwargs)[source]¶
Add electrodes by using only their positions (1D/2D/3D).
Electrode positions are rounded to the self.round_to_decimals decimal to ensure we can do proper comparison and identification with the floats.
Multiple formats are possible:
pandas.DataFrame with columns (x/z) or (x/y/z)
list (one position) or nested list (multiple positions)
numpy array
WARNING: For lists/tuples/arrays the number of supplied dimensions is determined by the second dimension after applying:
nr_dimensions = np.atleast_2d(input_data).shape[1]
As such, make sure to properly format 1d arrays!
- Parameters:
- data_rawnumpy.ndarray|tuple|list
Coordinate(s) for the electrodes to add to the pool
- remove_duplicatesbool, optional
If True, only add coordinates not already registered
- add_fixed_assignments(data_raw)[source]¶
Use a pre-determined electrode numbering scheme.
This is accomplished by the following steps:
clean all existing electrode positions already registered
set a lock that prevents the addition of additional
- align_assignments(pos1, pos2, abmn1, abmn2)[source]¶
Align the positions and logical electrode numbers of two datasets.
- Parameters:
- pos1pandas.DataFrame
Dataframe containing the electrode positions of the first dataset. Required columns: x, y, z and either a column ‘electrode_number’ or an index named ‘electrode_number’.
- pos2pandas.DataFrame
Dataframe containing the electrode positions of the second dataset. Required columns: x, y, z and either a column ‘electrode_number’ or an index named ‘electrode_number’.
- abmn1pandas.DataFrame
Dataframe containing the logical electrode numbers a,b,m,n for the data set 1.
- abmn2pandas.DataFrame
Dataframe containing the logical electrode numbers a,b,m,n for the data set 2.
- Returns:
- electrode_positions_alignedpandas.DataFrame
Aligned electrode positions with new electrode numbers (index is named ‘electrode_number’).
- abmn1_alignedpandas.DataFrame
Dataframe containing the logical electrode numbers a,b,m,n of the first data set after merging (aligning) the electrode positions of both datasets.
- abmn2_alignedpandas.DataFrame
Dataframe containing the logical electrode numbers a,b,m,n of the second data set after merging (aligning) the electrode positions of both datasets.
- conform_to_regular_x_spacing(spacing_x, nr_electrodes=None, y_level=0, z_level=0)[source]¶
Assume electrodes are located on a regular grid of electrodes with spacing spacing_x. Add missing electrode positions and thus ensure that the resulting electrode numbers conform to this full set of electrodes.
This is useful when dealing with multiple measurements using multi-channel/multiplexer systems where not all electrodes are used in every measurement.
This function works for simple setups - for specialized cases it is recommended to provide a complete electrode position list using the XXX method.
Electrode positions are assumed to start at 0.
- Parameters:
- spacing_xfloat
Spacing of electrode meash
- nr_electrodesint|None
If set, only fill in electrode positions up to this electrode. Otherwise the maximal x position is used to determine the electrode number.
- y_levelfloat, optional
y coordinate used for the additional points. Defaults to 0
- z_levelfloat, optional
z coordinate used for the additional points. Defaults to 0
- Raises:
- Exception in case where existing electrode positions are not multiples
- of spacing_x.
- property electrode_positions¶
- get_all_electrode_numbers()[source]¶
Return the assignment between electrode numbers and coordinates
- get_electrode_numbers_for_positions(positions_raw)[source]¶
For a given set of coordinates, return electrode coordinates
In order to prevent problems with floating point representation and comparison, positions are rounded to the decimal given in self.round_to_decimals.
- Returns:
- positionnumpy.ndarray|None
- get_position_of_number(number)[source]¶
Return the position associated with a given electrode number.
- reflect_on_position_x(reflect_on_x)[source]¶
Reflect all electrode positions on a given x position:
x_new = reflect_on_x - x_i
This procedure modifies the actual position data and cannot be reversed.
It is useful to treat reciprocal data measured with devices that can be configured to measure reciprocal data easily, while retaining the electrode coordinates of the normal spread (e.g., possible with the Iris Instruments Syscal devices.
- register_custom_ordering_scheme(function)[source]¶
Register a user-supplied function to assign electrode numbers to a given set of coordinates.
The function must take one argument, a pandas.DataFrame with electrode coordinates in columns x,y,z. It then returns a ordered copy of that DataFrame, whose index values denote the electrode numbers. It is recommended to adjust the index to start with electrode number 1.
- replace_coordinate_of_electrode_number(electrode_number, coordinates)[source]¶
Replace the coordinates of the given electrode number (in the current ordering) with the new coordinates.
Note that, depending on the chosen ordering scheme this can lead to the electrode assignments changing. This is the case for the ‘sort_coordinates_zyx’ scheme. However, ‘as_is’ and ‘as_is_plus_one’ will retain the electrode order.
- Parameters:
- electrode_numberint
The electrode number to replace
- coordinatestuple|list|numpy.ndarray of size 2 or 3
The new coordinates
- replace_coordinates_with_fixed_regular_grid_x(spacing_x)[source]¶
Using the current ordering scheme, replace the x coordinates with multiples of spaxing_x, starting with 0.
- Parameters:
- spacing_xfloat
New spacing of x coordinates
- set_ordering_to_as_is()[source]¶
Helper function that activates the sorter that assigns electrode numbers in the order the coordinates were registered. Useful if coordinates are input by hand.
- set_ordering_to_as_is_plus_one()[source]¶
Helper function that activates the sorter that assigns electrode numbers in the order the coordinates were registered. Useful if coordinates are input by hand.
This function will add 1 to the index values to ensure we do not zero index.
- set_ordering_to_sort_zyx()[source]¶
Helper function that activates the sorter that actually sorts electrode coordinates before assigning the electrode numbers. Useful to assign electrode numbers for automatically generated electrode numbers, such as encountered for some measurement systems.
- shift_positions_xyz(shift_by_xyz)[source]¶
Shift electrode positions by adding the vector ‘shift_by_xyz’ to each position.
- Parameters:
- shift_by_xyztuple|list|numpy.ndarray of size 1 or 2 or 3
The vector to shift the data by. Length of 1 assumes that only the x coordinate of the vector differs from zero. Length of 2 assume a shift in (x,z) direction.
reda.utils.enter_directory module¶
- class reda.utils.enter_directory.CreateEnterDirectory(directory)[source]¶
Bases:
EnterDirectory
This is a context manager that enters a given directory and returns to the initial current directory after finishing. If the target directory does not exist, create it.
reda.utils.filter_config_types module¶
Sort configurations into four-point spread types.
Sorting is done by subsequently applying filters to the configurations, with removal of selected configurations. Thus, each filter sees only configurations not ‘chosen’ by any previously applied filters.
dipole-dipole
Schlumberger
Wenner
misc
- reda.utils.filter_config_types.filter(configs, settings)[source]¶
Main entry function to filtering configuration types
- Parameters:
- configs: Nx4 array
array containing A-B-M-N configurations
- settings: dict
‘only_types’: [‘dd’, ‘other’], # filter only for those types
- Returns:
- dict
results dict containing filter results (indices) for all registered filter functions. All remaining configs are stored under the keywords ‘remaining’
reda.utils.fix_sign_with_K module¶
Fix signs in resistance measurements using the K factors. The sign of negative resistance measurements can be switched if the geometrical factor is negative.
- reda.utils.fix_sign_with_K.fix_sign_with_K(dataframe, **kwargs)[source]¶
Swap electrode denotations so that geometrical (K) factors become positive. Also, swap signs of all parameters affected by this process.
Affected parameters, at the moment, are:
k
r
Vmn
Zt
rho_a
rpha
- Parameters:
- dataframepandas.DateFrame
dataframe holding the data
- Returns:
- dataframepandas.DateFrame
the fixed dataframe
reda.utils.geom_fac_crtomo module¶
Compute geometric factors (also referred to as K) using CRMod/CRTomo
- reda.utils.geom_fac_crtomo.compute_K(dataframe, settings, keep_dir=False)[source]¶
- Parameters:
- dataframepandas.DataFrame
dataframe that contains the data
- settingsdict
with required settings, see below
- keep_dirpath
if not None, copy modeling dir here
- settings = {
‘rho’: 100, # resistivity to use for homogeneous model, [Ohm m] ‘elem’ ‘elec’ ‘2D’ : True|False ‘sink_node’: ‘100’,
- }
reda.utils.geometric_factors module¶
select and execute functions to compute geometric factors according to the rcParams variable.
- reda.utils.geometric_factors.apply_K(df, k, **kwargs)[source]¶
Apply the geometric factors to the dataset and compute (apparent) resistivities/conductivities
- Parameters:
- dfpandas.DataFrame
The dataframe containing the data to which geometric factors shall be applied
- knumpy.ndarray
The geometric factors
- **kwargs: {}
No kwargs at the moment
- Returns:
- dfpandas.DataFrame
The data with geometric factors applied
- reda.utils.geometric_factors.compute_K_analytical(dataframe, spacing, **kwargs)[source]¶
Given an electrode spacing, compute geometrical factors using the equation for the homogeneous half-space (Neumann-equation)
If a dataframe is given, use the column (a, b, m, n). Otherwise, expect an Nx4 arrray.
- Parameters:
- dataframepandas.DataFrame or numpy.ndarray
Configurations, either as DataFrame
- spacingfloat or numpy.ndarray
distance between electrodes. If array, then these are the x-coordinates of the electrodes
- Returns:
- Knumpy.ndarray
The geometric factors corresponding to the provided configurations
- reda.utils.geometric_factors.compute_K_numerical(dataframe, settings=None, keep_dir=None)[source]¶
Use a finite-element modeling code to infer geometric factors for meshes with topography or irregular electrode spacings.
- Parameters:
- dataframepandas.DataFrame
the data frame that contains the data
- settingsdict
The settings required to compute the geometric factors. See examples down below for more information in the required content.
- keep_dirpath
if not None, copy modeling dir here
- Returns:
- K
numpy.ndarray
K factors (are also directly written to the dataframe)
- K
Examples
settings = { 'rho': 100, 'elem': 'elem.dat', 'elec': 'elec.dat', 'sink_node': '100', '2D': False, }
reda.utils.helper_functions module¶
Useful helper functions.
- reda.utils.helper_functions.has_multiple_timesteps(data)[source]¶
Return True if data container has multiple timesteps.
- reda.utils.helper_functions.opt_import(module, requiredFor='use the full functionality')[source]¶
Import and return module only if it exists. If module cannot be imported, a warning is printed followed by the requiredFor string. Otherwise, the imported module will be returned. This function should be used to import optional dependencies in order to avoid repeated try/except statements.
- Parameters:
- modulestr
Name of the module to be imported.
- requiredForstr, optional
Info string for the purpose of the dependency.
Examples
>>> from reda.utils import opt_import >>> reda = opt_import("reda") >>> reda.__name__ 'reda' >>> opt_import("doesNotExist", requiredFor="do something special") No module named 'doesNotExist'. You need to install this optional dependency to do something special.
- reda.utils.helper_functions.search(what)[source]¶
Utility function to search docstrings for string what.
reda.utils.mpl module¶
This file set ups matplotlib plot functions for the whole package.
Import all necessary Matplotlib modules and set default options To use this module, import * from it:
Examples¶
>>> import reda.utils.mpl
>>> plt, mpl = reda.utils.mpl.setup()
- reda.utils.mpl.mpl_get_cb_bound_below_plot(ax)[source]¶
Return the coordinates for a colorbar axes below the provided axes object. Take into account the changes of the axes due to aspect ratio settings.
Important: Use only AFTER fig.subplots_adjust(…)
- reda.utils.mpl.mpl_get_cb_bound_next_to_plot(ax)[source]¶
Return the coordinates for a colorbar axes next to the provided axes object. Take into account the changes of the axes due to aspect ratio settings.
Parts of this code are taken from the transforms.py file from matplotlib
Important: Use only AFTER fig.subplots_adjust(…)
Examples
>>> import matplotlib as mpl >>> import matplotlib.pyplot as plt >>> from reda.utils.mpl import mpl_get_cb_bound_next_to_plot >>> fig, ax = plt.subplots() >>> fig.subplots_adjust(right=0.8) >>> plt_obj = ax.plot([1, 2, 3], [1, 2, 3], '.-') >>> cb_pos = mpl_get_cb_bound_next_to_plot(ax) >>> ax1 = fig.add_axes(cb_pos, frame_on=True) >>> cmap = mpl.cm.jet_r >>> norm = mpl.colors.Normalize(vmin=float(23), vmax=float(33)) >>> cb1 = mpl.colorbar.ColorbarBase( ... ax1, ... cmap=cmap, ... norm=norm, ... orientation='vertical' ... ) >>> cb1.locator = mpl.ticker.FixedLocator([23, 28, 33]) >>> cb1.update_ticks() >>> # cb1.ax.artists.remove(cb1.outline)
(
Source code
,png
,pdf
)
- reda.utils.mpl.setup(use_latex=False, overwrite=False)[source]¶
Set up matplotlib imports and settings.
- Parameters:
- use_latex: bool, optional
Determine if Latex output should be used. Latex will only be enable if a ‘latex’ binary is found in the system.
- overwrite: bool, optional
Overwrite some matplotlib config values.
- Returns:
- plt:
pylab
pylab module imported as plt
- mpl:
matplotlib
matplotlib module imported as mpl
- plt:
reda.utils.norrec module¶
Normal-reciprocal functionality
- reda.utils.norrec.assign_norrec_diffs(df, diff_list)[source]¶
Compute and write the difference between normal and reciprocal values for all columns specified in the diff_list parameter.
Note that the DataFrame is directly written to. That is, it is changed during the call of this function. No need to use the returned object.
- Parameters:
- df: pandas.DataFrame
Dataframe containing the data
- diff_list: list
list of columns to compute differences for.
- Returns:
- df_new: pandas.DataFrame
The data with added columns
- reda.utils.norrec.assign_norrec_to_df(df)[source]¶
Determine normal-reciprocal pairs for a given dataframe.
- Parameters:
- df: pandas.DataFrame
The data
- Returns:
- df_new: pandas.DataFrame
The data with two new columns: “id” and “norrec”
- reda.utils.norrec.average_repetitions(df, keys_mean)[source]¶
average duplicate measurements. This requires that IDs and norrec labels were assigned using the assign_norrec_to_df function.
- Parameters:
- df
DataFrame
- keys_mean: list
list of keys to average. For all other keys the first entry will be used.
- reda.utils.norrec.compute_norrec_differences(df, keys_diff)[source]¶
DO NOT USE ANY MORE - DEPRECIATED!
- reda.utils.norrec.get_test_df()[source]¶
Return a test dataframe suitable to test the normal-reciprocal functions
- reda.utils.norrec.get_test_df_advanced()[source]¶
Return a test dataframe suitable to test the normal-reciprocal functions
reda.utils.pseudo_positions module¶
Various ways to compute pseudo-positions of given electrode configurations.