reda.importers package

Importers for different measurement devices

Subpackages

• reda.importers.legacy package
  • Submodules
  • reda.importers.legacy.eit160 module
    • import_medusa_data()
  • reda.importers.legacy.eit40 module
    • apply_correction_factors()
    • import_medusa_data()
• reda.importers.utils package
  • Submodules
  • reda.importers.utils.decorators module
    • enable_result_transforms()
  • reda.importers.utils.transforms module
    • transform_electrodes_roll_along
      • transform_electrodes_roll_along.transform()

Submodules

reda.importers.bert module

Importer to load the unified data format used in pyGIMLi, BERT, and dc2dinvres.

reda.importers.bert.import_ohm(filename, verbose=False, reciprocals=False, **kwargs)

Construct pandas data frame from BERT`s unified data format (.ohm).

Parameters:

filenamestring

File path to .ohm file

verbosebool, optional

Enables extended debug output

reciprocalsint, optional

if provided, then assume that this is a reciprocal measurement where only the electrode cables were switched. The provided number N is treated as the maximum electrode number, and denotations are renamed according to the equation \(X_n = N - (X_a - 1)\)

Returns:

datapandas.DataFrame

The measurement data

elecspandas.DataFrame

Electrode positions (columns: X, Y, Z)

topographyNone

No topography information is provided at the moment

reda.importers.crtomo module

Import data in the CRMod/CRTomo format

reda.importers.crtomo.load_mod_file(filename)

Load a .mod file (sometimes also called volt.dat or data.crt). This file contains the number of measurements in the first line, and in the following lines measurements. Each line consists of 4 columns:

• the first column contains the current injection electrodes a and b, stored as one integer using the equation: a * 1e4 + b

• the second column contains the voltage measurement electrodes m and n, stored as one integer using the equation: m * 1e4 + n

• the third column contains the measured resistance [Ohm]

• the fourth column contains the measured phase value [mrad]

Parameters:

filenamestr

Path of filename to import

Returns:

dfpandas.DataFrame

A reda-conform DataFrame

Examples

import reda import reda.importers.crtomo as cexp df = cexp.load_mod_file(‘volt_01_0.1Hz.crt’) ert = reda.ERT(data=df)

reda.importers.crtomo.load_seit_data(directory, frequency_file='frequencies.dat', data_prefix='volt_', **kwargs)

Load sEIT data from data directory. This function loads data previously exported from reda using reda.exporters.crtomo.write_files_to_directory

Parameters:

directorystring

input directory

frequency_filestring, optional

file (located in directory) that contains the frequencies

data_prefix: string, optional

for each frequency a corresponding data file must be present in the input directory. Frequencies and files are matched by sorting the frequencies AND the filenames, retrieved using glob and the data_prefix

Returns:

dfpandas.DataFrame

A DataFrame suitable for the sEIT container

electrodesNone

No electrode data is imported

topographyNone

No topography data is imported

reda.importers.eit_fzj module

Import data from the EIT-systems built at the Research Center Jülich (FZJ).

As there is an increasing number of slightly different file formats in use, this module acts as an selector for the appropriate import functions.

reda.importers.eit_fzj.MD_ConfigsPermutate(df_md)

Given a MD DataFrame, return a Nx4 array which permutes the current injection dipoles.

reda.importers.eit_fzj.apply_correction_factors(df, correction_data)

Apply correction factors for a pseudo-2D measurement setup. See Weigand and Kemna, 2017, Biogeosciences, for detailed information.

Parameters:

dfpandas.DataFrame

DataFrame containing the data

correction_datastring|iterable of strings|:py:class:numpy.ndarray

Correction data, either as a filename, a list of filenames to be merged, or directly as a numpy array

Returns:

dfpandas.DataFrame

Corrected data

corr_datanumpy.ndarray

Correction factors used

reda.importers.eit_fzj.compute_data_errors(data_emd_4p, data_md_raw, adc_data, binary_file, **kwargs)

Compute data errors based on a noise-level analysis and subsequent linear error propagation.

reda.importers.eit_fzj.compute_magnitude_errors(phi_m, phi_n, current, dphi_m, dphi_n, dcurrent, **kwargs)

Compute magnitude errors based on linear error propagation

reda.importers.eit_fzj.compute_phase_errors(phi_m, phi_n, current, dphi_m, dphi_n, dcurrent, **kwargs)

Compute the phase error based on linear error propagation of the transfer impedance equation:

Zt = U_mn / I_ab = (phi_m - phi_n) / I_ab

Parameters:

phi_mnumpy.ndarray|complex float

Complex potential at electrode m

phi_nnumpy.ndarray|complex float

Complex potential at electrode n

currentnumpy.ndarray|complex float

Complex current injected at electrodes a and b

dphi_mnumpy.ndarray|complex float

potential error at electrode m. Real part corresponds to error of the real part of the potential, vice versa for the imaginary part

dphi_nnumpy.ndarray|complex float

potential error at electrode m. Real part corresponds to error of the real part of the potential, vice versa for the imaginary part

dcurrentnumpy.ndarray|complex float

current error at electrode m. Real part corresponds to error of the real part of the potential, vice versa for the imaginary part

Returns:

error_phasenumpy.ndarray|float

Phase error in [rad]

reda.importers.eit_fzj.compute_quadrupoles(df_emd, config_file, df_md=None)

Parameters:

df_emdpandas.DataFrame

The EMD data, as imported from the .mat file (3P-data)

config_filestring

filename for configuration file. The configuration file contains N rows with 4 columns each (A, B, M, N)

df_mdpandas.DataFrame (optional)

The MD data

Returns:

reda.importers.eit_fzj.get_adc_data(filename, **kwargs)

Import data and return the adc-related data from the MD (i.e., injection) structure

Parameters:

filenamestring (usually: eit_data_mnu0.mat)

filename of matlab file

Returns:

data_adc_rawpandas.DataFrame|None

adc-MD data (sometimes this data is not imported, then we return None here)

reda.importers.eit_fzj.get_md_data(filename, **kwargs)

Import data and return the MD (i.e., injection) data

Parameters:

filenamestring (usually: eit_data_mnu0.mat)

filename of matlab file

Returns:

data_md_rawpandas.DataFrame|None

MD data (sometimes this data is not imported, then we return None here)

reda.importers.eit_fzj.get_mnu0_data(filename, configs, return_3p=False, **kwargs)

Import data post-processed as 3P data (NMU0), i.e., measured towards common ground.

Parameters:

filenamestring (usually: eit_data_mnu0.mat)

filename of matlab file

configsNx4 numpy.ndarray|filename|function

4P measurements configurations (ABMN) to generate out of the data. If this parameter is a callable, then call it with the MD DataFrame as its sole parameter and expect a Nx4 numpy.ndarray as return value

return_3pbool, optional

also return 3P data

Returns:

data_emd_4ppandas.DataFrame

The generated 4P data

data_md_rawpandas.DataFrame|None

MD data (sometimes this data is not imported, then we return None here)

data_emd_3ppandas.DataFrame

The imported 3P data (only if return_3p==True)

reda.importers.eit_fzj.read_3p_data(filename, configs, return_3p=False, **kwargs)

Import data post-processed as 3P data (NMU0), i.e., measured towards common ground.

Parameters:

filenamestring (usually: eit_data_mnu0.mat)

filename of matlab file

configsNx4 numpy.ndarray|filename|function

4P measurements configurations (ABMN) to generate out of the data. If this parameter is a callable, then call it with the MD DataFrame as its sole parameter and expect a Nx4 numpy.ndarray as return value

return_3pbool, optional

also return 3P data

Returns:

data_emd_4ppandas.DataFrame

The generated 4P data

data_md_rawpandas.DataFrame|None

MD data (sometimes this data is not imported, then we return None here)

data_emd_3ppandas.DataFrame

The imported 3P data (only if return_3p==True)

reda.importers.eit_version_2010 module

Research Center Jülich - EIT40 system importer (2010 version)

reda.importers.eit_version_2013 module

reda.importers.eit_version_2017 module

reda.importers.eit_version_2018a module

reda.importers.eit_version_2018a.convert_electrode(i)

With 3 multiplexers, electrode numbers are running from 1 - 120, in 30 electrode increments. We need to convert those numberings back.

reda.importers.eit_version_20200609 module

reda.importers.fzj_readbin module

This is a helper class to read the primary binary measurement data of the FZJ SIP and EIT systems SIP-04 and EIT40 (Zimmermann et al. 2008 a, b).

This is not a regular REDA-Importer as the time-domain data contained in these binary files is not usable for geoelectric processing. However, looking at this primary digitized data (i.e., the first digital representation of the analog measurement signal) can help in understanding and analyzing the final SIP/sEIT data and associated problems.

class reda.importers.fzj_readbin.fzj_readbin(filename=None)

Bases: object

Methods

characterize()	Print a few characteristics of the loaded data
fft_analysis_one_channel(measurement_index, ...)	On one specific measurement at one channel, conduct an FFT analysis to estimate the noise level.
find_swapped_measurement_indices(a, b, frequency)	For a given set of injection electrodes and a frequency, try to find the two injections that will make up the final measurement (i.e., the regular injection (a,b) and its swapped injection (b,a).
get_ts_abm(a, b, m, frequency)	Return the time series for a given set of (a, b, m electrodes)
import_file(filename)	Parameters:
list_injections()	List the available injections
plot_noise_level_for_one_injection(...[, nch])	measurement_index can be found by using the search function:
plot_timeseries_to_axes(axes, ...[, ...])	injection_number is 1-indexed

get_sample_times
plot_per_frequency
plot_timeseries

characterize()

Print a few characteristics of the loaded data

fft_analysis_one_channel(measurement_index, channel, split_into_three=False, plot=False, **kwargs)

On one specific measurement at one channel, conduct an FFT analysis to estimate the noise level.

Parameters:

measurement_indexint

Index of injection related to index in self.frequency_data.

channelint

Channel to analyze. 1-indexed.

split_into_threebool, optional (default: False)

If True, apply analysis to each third of the time-series separately.

plot: bool, optional (default: False)

If True, generate plots of the time-series and noise level

remove_excitation_frequencybool, optional (default: False)

.

remove_noise_harmonicsbool (default, False)

.

Returns:

noise_levelslist

The estimated white noise leves for the parts of the time-series. If split_into_three is False, then the list contains only one entry

plotslist, optional

If generated return plots in this list

find_swapped_measurement_indices(a, b, frequency, mean_measurement_time=None)

For a given set of injection electrodes and a frequency, try to find the two injections that will make up the final measurement (i.e., the regular injection (a,b) and its swapped injection (b,a).

Parameters:

aint

1. Current electrode

bint

2. Current electrode

frequencyfloat

Measurement frequency

mean_measurement_timedatetime.datetime|pandas.Timestamp

For swapped measurements the datetime entry in the MD and EMD structs will be the mean time between the singular measurements.

Returns:

findices[int, int]

Indices of the rows in fzj_readbin.frequency_data corresponding to the measurement. If only one measurement was found, then the second index is None.

get_sample_times(frequency)

get_ts_abm(a, b, m, frequency)

Return the time series for a given set of (a, b, m electrodes)

All values are 1-indexed!!!

WARNING: This interface always chooses the first result related to the input set in case duplicate measurements are present! This relates to duplicate frequencies and duplicate injections.

import_file(filename)

Parameters:

filenamestr

Filename to either the .mcf or .bin file. It is assumed that the corresponding .mff, .mcf, and .bin files reside in the same location with the same filename.

list_injections()

List the available injections

plot_noise_level_for_one_injection(measurement_index, nch=None, **kwargs)

measurement_index can be found by using the search function:

indices = self.find_swapped_measurement_indices(1, 22, 1) fig = plot_noise_level_for_one_injection(indices[0])

plot_per_frequency()

plot_timeseries(filename, frequency_index, injection_number, channel, range_fraction=1.0, plot_style='.-', index_start=0)

plot_timeseries_to_axes(axes, frequency_index, injection_number, channel, range_fraction=1.0, plot_style='.-', index_start=0)

injection_number is 1-indexed

reda.importers.geotom module

reda.importers.geotom.add_file(filename, settings, container=None)

Parameters:

filename: string

path

settings: dict

{
    timestep: [int|datetime], timestep relating to this measurement
}

container: :class:`reda.container.ERT.ERT`

ERT container to add dataset to

reda.importers.iris_syscal_pro module

Read binary data from the IRIS Instruments Syscal Pro system

reda.importers.iris_syscal_pro.import_bin(filename, **kwargs)

Read a .bin file generated by the IRIS Instruments Syscal Systems and return a curated Pandas dataframe for further processing. This dataframe contains only information currently deemed important. Use the function reda.importers.iris_syscal_pro_binary._import_bin to extract ALL information from a given .bin file.

Note that the .bin files contain electrodes positions for the electrodes. Electrode numbers are then inferred from those.

Parameters:

filenamestring|io.BytesIO

path to input filename or io.BytesIO object containing the binary data

assume_regular_electrodes_xNone|tuple(nr_electrodes, spacing)

If not None, then assume measurements were taken using a profile of regular electrodes in x directions. Fill in any electrodes not used in the data to align the logical electrode numbers wit the physical electrode numbers. Use the electrode spacing set in the system (usually a default of 1 m is used), and then change this to the real spacing using the ‘elecs_transform_reg_spacing_x’ parameter.

elecs_transform_reg_spacing_xtuple(old, new)|None, optional

If not None, then assume a regular electrode spacing in x direction with spacing ‘old’. This is often 1 m, a default of the systems. However, in reality often other spacings are used, and this parameter will lead to a transformation of the old (device-specific) spacing to the true spacing.

shift_by_xyztuple|list|numpy.ndarray of size 1 or 2 or 3, optional

If set, shift electrode positions by adding this vector Length of 1 assumes that only the x coordinate of the vector differs from zero. Length of 2 assume a shift in (x,z) direction.

This parameter is evaluated after the ‘elecs_transform_reg_spacing_x’ parameter - as such you can and must use real spacings in this case.

reciprocalsint, optional

if provided, then assume that this is a reciprocal measurement where only the electrode cables were switched. The provided number N is treated as the maximum electrode number, and denotations are renamed according to the equation \(X_n = N - (X_a - 1)\)

check_meas_numsbool, optional

if True, then check that the measurement numbers are consecutive. Don’t return data after a jump to smaller measurement numbers (this usually indicates that more data points were downloaded than are part of a specific measurement. Default: True

skip_rowsint, optional

Ignore this number of rows at the beginning, e.g., because they were inadvertently imported from an earlier measurement. Default: 0

Returns:

datapandas.DataFrame

Contains the measurement data

elec_mgrreda.utils.electrode_manager.electrode_manager

Electrode manager that manages the electrode positions

topographyNone

No topography information is contained in the text files, so we always return None

reda.importers.iris_syscal_pro.import_txt(filename, **kwargs)

Import Syscal measurements from a text file, exported as ‘Spreadsheet’.

At this point we expect only x-coordinates to be exported. As such the columns Spa.1 to Spa.4 are assumed to be x coordinates for a,b,m,n.

Parameters:

filename: str

Input filename

assume_regular_electrodes_xNone|tuple(nr_electrodes, spacing)

If not None, then assume measurements were taken using a profile of regular electrodes in x directions. Fill in any electrodes not used in the data to align the logical electrode numbers with the physical electrode numbers. Use the electrode spacing set in the system (usually a default of 1 m is used), and then change this to the real spacing using the ‘elecs_transform_reg_spacing_x’ parameter.

elecs_transform_reg_spacing_xtuple(old, new)|None, optional

If not None, then assume a regular electrode spacing in x direction with spacing ‘old’. This is often 1 m, a default of the systems. However, in reality often other spacings are used, and this parameter will lead to a transformation of the old (device-specific) spacing to the true spacing.

shift_by_xyztuple|list|numpy.ndarray of size 1 or 2 or 3, optional

If set, shift electrode positions by adding this vector Length of 1 assumes that only the x coordinate of the vector differs from zero. Length of 2 assume a shift in (x,z) direction.

This parameter is evaluated after the ‘elecs_transform_reg_spacing_x’ parameter - as such you can and must use real spacings in this case.

reciprocals: int, optional

if provided, then assume that this is a reciprocal measurements where only the electrode cables were switched. The provided number N is treated as the maximum electrode number, and denotations are renamed according to the equation \(X_n = N - (X_a - 1)\)

Returns:

data: pandas.DataFrame

Contains the measurement data

elec_mgrreda.utils.electrode_manager.electrode_manager

Electrode manager that manages the electrode positions

topography: None

No topography information is contained in the text files, so we always return None

reda.importers.mpt_das1 module

Read MPT DAS-1 data files.

TODO:

reda.importers.mpt_das1.get_frequencies(filename, header_row)

Read the used frequencies in header of DAS-1 SIP data set.

Parameters:

filenamestr

input filename

header_rowint

row number of header row

Returns:

frequencieslist

Contains the measured frequencies

reda.importers.mpt_das1.get_measurement_type(filename)

Given an MPT DAS-1 result file, try to determine the type of measurement.

Currently supported/detected types:

• TDIP (i.e., time-domain measurement)

• FD (i.e., complex measurement)

• SIP (i.e., sEIT measurements

Parameters:

filenamestr

Path to data file

Returns:

measurement_typestr

The type of measurement: tdip,cr,sip

reda.importers.mpt_das1.import_das1(filename, **kwargs)

Reads a any MPT DAS-1 data file (.Data), e.g. TD/FD/SIP, and prepares information in pandas DataFrame for further processing.

Parameters:

filenamestring

path to input file

Returns:

datapandas.DataFrame

Contains the measurement data

electrodespandas.DataFrame

Contains electrode positions (None at the moment)

topographyNone

No topography information is contained in the text files, so we always return None

reda.importers.mpt_das1.import_das1_fd(filename, **kwargs)

Reads a frequency domain (single frequency) MPT DAS-1 data file (.Data) and pre pares information in pandas DataFrame for further processing.

Parameters:

filenamestr

path to input file

corr_arraylist, optional

used to correct the electrode numbers [a, b, m, n], eg. for cable layouts which separated current and potential cables, hence, 64 electrodes for a measurement profile of 32 electrodes

Returns:

datapandas.DataFrame

Contains the measurement data

electrodespandas.DataFrame

Contains electrode positions (None at the moment)

topographyNone

No topography information is contained in the text files, so we always return None

reda.importers.mpt_das1.import_das1_sip(filename, **kwargs)

Reads a spectral induced polarization MPT DAS-1 data file (.Data) and prepares information in pandas DataFrame for further processing.

Parameters:

filenamestring

path to input file

Returns:

datapandas.DataFrame

Contains the measurement data

electrodespandas.DataFrame

Contains electrode positions (None at the moment)

topographyNone

No topography information is contained in the text files, so we always return None

reda.importers.mpt_das1.import_das1_td(filename, **kwargs)

Reads a time domain MPT DAS-1 data file (.Data) and prepares information in pandas DataFrame for further processing.

Parameters:

filenamestr

path to input file

Returns:

datapandas.DataFrame

Contains the measurement data

electrodespandas.DataFrame

Contains electrode positions (None at the moment)

topographyNone

No topography information is contained in the text files, so we always return None

reda.importers.radic_sip256c module

reda.importers.radic_sip256c.compute_quadrupoles(reading_configs, readings, settings)

reda.importers.radic_sip256c.parse_radic_file(filename, settings, selection_mode='after', reciprocal=None)

Import one result file as produced by the SIP256c SIP measuring device (Radic Research)

Full settings dictionary:

settings = {
    'filter_skip': (integer) skip dipoles we are interested in
    'quadrupole_mode': ['after'|'between'|'before'| 'all']
                       which dipoles to use from the file
}

Parameters:

filenamestring

input filename, usually with the ending “.RES”

settingsdict

Settings for the data import, see code snippet above

selection_modedict

which voltage dipoles should be returned. Possible choices: “all”|”before”|”after”

reciprocalint|None

If this is an integer, then assume this was a reciprocal measurement and the number denotes the largest RU number, N. Electrode numbers (a,b,m,n) will then be transformed to (N1 - a, N1 - b, N1 - m, N1 - n), with N1 = N + 1

Returns:

sip_data:py:pandas`pandas.DataFrame`

The data contained in a data frame

electrodesNone

No electrode positions are imported

topographyNone

No topography is imported

reda.importers.radic_sip256c.parse_reading(reading_block)

reda.importers.radic_sip256c.write_crmod_file(sipdata, directory)

reda.importers.res2dinv module

Importer for RES2DINV files

Please note that this importer is very rudimentary at the moment. We need test data and real usage to improve upon this.

reda.importers.res2dinv.import_res2dinv(filename, **kwargs)

Read a RES2DINV-style

Parameters:

Returns:

datapandas.DataFrame

Contains the measurement data

elec_mgrreda.utils.electrode_manager.electrode_manager

Electrode manager that manages the electrode positions

topographyNone

No topography information is contained in the text files, so we always return None

reda.importers.sip04 module

Importer functions for data files measured with the SIP-04 SIP system, developed at the research center Jülich.

The result files contain a lot of different, sometimes redundant parameters.

Parameter	Explanation/Meaning of Parameters:
…_1	First measurement
…_2	Second measurement
…_3	Third measurement
…_m	Mean value of that three measurements
…_std	Standard deviation of three three measurements
Zm	complex value of transfer impedance
Zm_mAbs	Absolute value of Zm_m
Zm_mPhi	Phase shift of Zm_m
Zm_mRe	Real part of Zm_m
Zm_mIm	Imaginary Part of Zm_m
Zm_AbsStd	Standard deviation of all 3 Zm_Abs values
Zm_PhiStd	Standard deviation of all 3 Zm_Phi values

reda.importers.sip04.import_sip04_data(data_filename)

Import RELEVANT data from the result files. Refer to the function reda.importers.sip04.import_sip04_data_all() for an importer that imports ALL data.

Exported parameters:

key	description
a	First current electrode
b	Second current electrode
m	First potential electrode
n	Second potential electrode
frequency	Measurement frequency
Temp_1	Temperature sensor 1 (optional)
Temp_2	Temperature sensor 2 (optional)
zt	Complex Transfer Impedance (the measurement), mean value
r	Magnitude of mean measurements (=|zt|)
rpha	Resistance phase [mrad]
zt_1	Complex Transfer Impedance, first repetition
zt_2	Complex Transfer Impedance, second repetition
zt_3	Complex Transfer Impedance, third repetition
ContactResistance	Contact resistance (mean value)
ShuntResistance	Shunt resistance used [Ohm]

Parameters:

data_filenamestring

Path to .mat or .csv file containing SIP-04 measurement results. Note that the .csv file does not contain all data contained in the .mat file!

Returns:

dfpandas.DataFrame

The data, contained in a DataFrame

reda.importers.sip04.import_sip04_data_all(data_filename)

Import ALL data from the result files

Parameters:

data_filenamestring

Path to .mat or .csv file containing SIP-04 measurement results. Note that the .csv file does not contain all data contained in the .mat file!

Returns:

df_allpandas.DataFrame

The data, contained in a DataFrame

reda.importers.tsert_import module

class reda.importers.tsert_import.tsert_import(filename)

Bases: tsert_base

The TSERT file format – export functions

TSERT: Time-series electrical resistivity tomography

Methods

check_format(filename)	Check if a given file is actually a tsert file with correct version number
get_index()	Return the index of the file
import_data(timesteps[, version, ...])	Wrapper for .load_data that merges the data into one dataframe
load_data([timesteps, version, not_before, ...])	Parameters:
summary([print_index])	Short summary of the filename

load_electrode_positions
load_metadata
load_topography

import_data(timesteps, version='base', not_before=None, not_after=None)

Wrapper for .load_data that merges the data into one dataframe

load_data(timesteps='all', version='base', not_before=None, not_after=None)

Parameters:

timestepsiterable|’all’

Timesteps to import, the string ‘all’ will import all data

version

not_beforeNone|datetime.datetime

Only select data sets whose timestep lies before the given datetime. Only evaluated if the timesteps parameter is set to ‘all’.

not_afterNone|datetime.datetime

Only select data sets whose timestep lies after the given datetime. Only evaluated if the timesteps parameter is set to ‘all’.

Returns:

data_listlist

List of dataframes

load_electrode_positions()

load_metadata()

load_topography()

summary(print_index=False)

Short summary of the filename

Parameters:

print_indexbool, optional