A cache based data reader. More...

#include <DataMgrV3_0.h>

Inheritance diagram for VAPoR::DataMgrV3_0:

Classes
class	BlkExts

Public Member Functions
	DataMgrV3_0 (string format, size_t mem_size, int nthreads=0)

virtual	~DataMgrV3_0 ()

virtual int	Initialize (const std::vector< string > &files)

virtual std::vector< string >	GetDataVarNames () const

virtual std::vector< string >	GetDataVarNames (int ndim, bool spatial) const

virtual std::vector< string >	GetCoordVarNames () const

virtual std::vector< string >	GetCoordVarNames (int ndim, bool spatial) const

void	GetTimeCoordinates (std::vector< double > &timecoords) const

bool	GetDataVarInfo (string varname, VAPoR::DC::DataVar &datavar) const

bool	GetBaseVarInfo (string varname, VAPoR::DC::BaseVar &var) const

bool	GetCoordVarInfo (string varname, VAPoR::DC::CoordVar &cvar) const

bool	IsTimeVarying (string varname) const

bool	IsCompressed (string varname) const

int	GetNumTimeSteps (string varname) const

int	GetNumRefLevels (string varname) const

int	GetCRatios (string varname, std::vector< size_t > &cratios) const

VAPoR::RegularGrid *	GetVariable (size_t ts, string varname, int level, int lod, bool lock=false)

VAPoR::RegularGrid *	GetVariable (size_t ts, string varname, int level, int lod, std::vector< double > min, std::vector< double > max, bool lock=false)

VAPoR::RegularGrid *	GetVariable (size_t ts, string varname, int level, int lod, std::vector< size_t > min, std::vector< size_t > max, bool lock=false)

int	GetVariableExtents (size_t ts, string varname, int level, std::vector< double > &min, std::vector< double > &max)

virtual int	GetDimLensAtLevel (string varname, int level, bool spatial, std::vector< size_t > &dims_at_level, std::vector< size_t > &bs_at_level) const

void	UnlockGrid (const VAPoR::RegularGrid *rg)

void	Clear ()

virtual bool	VariableExists (size_t ts, string varname, int level=0, int lod=0)

bool	IsVariableDerived (string varname) const

bool	IsVariableNative (string varname) const

void	PurgeVariable (string varname)

virtual string	GetMapProjection () const

Public Member Functions inherited from VetsUtil::MyBase
	MyBase ()

const string &	getClassName () const

Additional Inherited Members
Public Types inherited from VetsUtil::MyBase
typedef void(*	ErrMsgCB_T) (const char *msg, int err_code)

typedef void(*	DiagMsgCB_T) (const char *msg)

Static Public Member Functions inherited from VetsUtil::MyBase
static void	SetErrMsg (const char *format,...)
	Record a formatted error message. More...

static void	SetErrMsg (int errcode, const char *format,...)
	Record a formatted error message and an error code. More...

static const char *	GetErrMsg ()

static void	SetErrCode (int err_code)
	Record an error code. More...

static int	GetErrCode ()
	Retrieve the current error code. More...

static void	SetErrMsgCB (ErrMsgCB_T cb)

static ErrMsgCB_T	GetErrMsgCB ()

static void	SetErrMsgFilePtr (FILE *fp)

static const FILE *	SetErrMsgFilePtr ()

static void	SetDiagMsg (const char *format,...)
	Record a formatted diagnostic message. More...

static const char *	GetDiagMsg ()

static void	SetDiagMsgCB (DiagMsgCB_T cb)

static DiagMsgCB_T	GetDiagMsgCB ()

static void	SetDiagMsgFilePtr (FILE *fp)

static bool	EnableErrMsg (bool enable)

Static Public Attributes inherited from VetsUtil::MyBase
static char *	ErrMsg

static int	ErrCode

static int	ErrMsgSize

static FILE *	ErrMsgFilePtr

static ErrMsgCB_T	ErrMsgCB

static char *	DiagMsg

static int	DiagMsgSize

static FILE *	DiagMsgFilePtr

static DiagMsgCB_T	DiagMsgCB

static bool	Enabled

Protected Member Functions inherited from VetsUtil::MyBase
void	SetClassName (const string &name)

Detailed Description

A cache based data reader.

Author: John Clyne

The DataMgr class is an abstract class that defines public methods for accessing (reading) 1D, 2D and 3D field variables. The class implements a memory cache to speed data access – once a variable is read it is stored in cache for subsequent access. The DataMgr class is abstract: it declares a number of public and protected pure virtual methods that must be implemented by specializations of this class to allow access to particular file formats.

This class inherits from VetsUtil::MyBase. Unless otherwise documented any method that returns an integer value is returning status. A negative value indicates failure. Error messages are logged via VetsUtil::MyBase::SetErrMsg().

Methods that return a boolean do not, unless otherwise documented, log an error message upon failure (return of false).

Parameters

level

Grid refinement level for multiresolution variables. Compressed variables in the VDC have a multi-resolution representation: the sampling grid for multi-resolution variables is hierarchical, and the dimension lengths of adjacent levels in the hierarchy differ by a factor of two. The level parameter is used to select a particular depth of the hierarchy.

To provide maximum flexibility as well as compatibility with previous versions of the VDC the interpretation of level is somewhat complex. Both positive and negative values may be used to specify the refinement level and have different interpretations.

For positive values of level, a value of 0 indicates the coarsest member of the grid hierarchy. A value of 1 indicates the next grid refinement after the coarsest, and so on. Using postive values the finest level in the hierarchy is given by GetNumRefLevels() - 1. Values of level that are greater than GetNumRefLevels() - 1 are treated as if they were equal to GetNumRefLevels() - 1.

For negative values of level a value of -1 indicates the variable's native grid resolution (the finest resolution available). A value of -2 indicates the next coarsest member in the hierarchy after the finest, and so on. Using negative values the coarsest available level in the hierarchy is given by negating the value returned by GetNumRefLevels(). Values of level that are less than the negation of GetNumRefLevels() are treated as if they were equal to the negation of the GetNumRefLevels() return value.

Parameters

lod

The level-of-detail parameter, lod, selects the approximation level for a compressed variable. The lod parameter is similar to the level parameter in that it provides control over accuracy of a compressed variable. However, instead of selecting the grid resolution the lod parameter controls the compression factor by indexing into the cratios vector (see below). As with the level parameter, both positive and negative values may be used to index into cratios and different interpretations.

For positive values of lod, a value of 0 indicates the the first element of cratios, a value of 1 indicates the second element, and so on up to the size of the cratios vector (See DC::GetCRatios()).

For negative values of lod a value of -1 indexes the last element of cratios, a value of -2 indexes the second to last element, and so on. Using negative values the first element of cratios - the greatest compression rate - is indexed by negating the size of the cratios vector.

Definition at line 100 of file DataMgrV3_0.h.

Constructor & Destructor Documentation

VAPoR::DataMgrV3_0::DataMgrV3_0	(	string	format,
		size_t	mem_size,
		int	nthreads = `0`
	)

Constructor for the DataMgrV3_0 class.

The DataMgrV3_0 will attempt to cache previously read data and coordinate variables in memory. The mem_size specifies the requested cache size in MEGABYTES!!!

Parameters

[in]	format	A string indicating the format of data collection.
[in]	mem_size	Size of memory cache to be created, specified in MEGABYTES!!
[in]	numthreads	Number of parallel execution threads to be run during encoding and decoding of compressed data. A value of 0, the default, indicates that the thread count should be determined by the environment in a platform-specific manner, for example using sysconf(_SC_NPROCESSORS_ONLN) under *nix OSes.

virtual VAPoR::DataMgrV3_0::~DataMgrV3_0 ( )

virtual

Member Function Documentation

void VAPoR::DataMgrV3_0::Clear ( )

Clear the memory cache

This method clears the internal memory cache of all entries

bool VAPoR::DataMgrV3_0::GetBaseVarInfo	(	string	varname,
		VAPoR::DC::BaseVar &	var
	)		const

Return metadata about a data or coordinate variable

If the variable varname is defined as either a data or coordinate variable its metadata will be returned in var.

Return values

bool	If true the method was successful. If false the named variable is invalid (unknown)

See also: GetDataVarInfo(), GetCoordVarInfo()

bool VAPoR::DataMgrV3_0::GetCoordVarInfo	(	string	varname,
		VAPoR::DC::CoordVar &	cvar
	)		const

Return a coordinate variable's definition

Return a reference to a DC::CoordVar object describing the coordinate variable named by varname

Parameters

[in]	varname	A string specifying the name of the coordinate variable.
[out]	coordvar	A CoordVar object containing the definition of the named variable.

Return values

bool	If true the method was successful. If false the named variable is invalid (unknown)

See also: GetDataVarInfo()

virtual std::vector<string> VAPoR::DataMgrV3_0::GetCoordVarNames ( ) const

virtual

Return a list of names for all of the defined coordinate variables.

This method returns a list of all coordinate variables defined in the data set.

Return values

list	A vector containing a list of all the coordinate variable names

See also: GetDataVarNames()

virtual std::vector<string> VAPoR::DataMgrV3_0::GetCoordVarNames	(	int	ndim,
		bool	spatial
	)		const

virtual

Return a list of coordinate variables with a given dimension rank

This method returns a list of all coordinate variables defined in the coordinate set with the specified dimensionality

Parameters

[in]	ndim	Variable rank
[in]	spatial	A boolean indicating whether only the variables spatial rank should be compared against `ndim`

Return values

list	A vector containing a list of all the coordinate variable names with the specified number of dimensions.

int VAPoR::DataMgrV3_0::GetCRatios	(	string	varname,
		std::vector< size_t > &	cratios
	)		const

Return the compression ratio vector for the indicated variable

Return the compression ratio vector for the indicated variable. The vector returned contains an ordered list of available compression ratios for the variable named by variable. If the variable is not compressed, the cratios parameter will contain a single element, one.

Parameters

[in]	varname	Data or coordinate variable name.
[out]	cratios	Ordered vector of compression ratios

Return values

status A negative int is returned on failure

See also: DC::BaseVar::GetCRatios();

bool VAPoR::DataMgrV3_0::GetDataVarInfo	(	string	varname,
		VAPoR::DC::DataVar &	datavar
	)		const

Return a data variable's definition

Return a reference to a DC::DataVar object describing the data variable named by varname

Parameters

[in]	varname	A string specifying the name of the variable.
[out]	datavar	A DataVar object containing the definition of the named Data variable.

Return values

bool	If true the method was successful. If false the named variable is invalid (unknown)

See also: GetCoordVarInfo()

virtual std::vector<string> VAPoR::DataMgrV3_0::GetDataVarNames ( ) const

virtual

Return a list of names for all of the defined data variables.

This method returns a list of all data variables defined in the data set.

Return values

list	A vector containing a list of all the data variable names

See also: GetCoordVarNames()

Test:: New in 3.0

virtual std::vector<string> VAPoR::DataMgrV3_0::GetDataVarNames	(	int	ndim,
		bool	spatial
	)		const

virtual

Return a list of data variables with a given dimension rank

This method returns a list of all data variables defined in the data set with the specified dimension rank (number of dimensions). Data variables may have 0 to 3 spatial dimensions, and 0 of 1 time dimension.

Parameters

[in]	ndim	Variable rank (number of dimensions)
[in]	spatial	A boolean, if true, indicates that only the variable's spatial rank should be compared against `ndim`. Otherwise the sum of the number of spatial dimensions plus the time dimension (if time varying) is compared agains `ndim`.

Return values

list	A vector containing a list of all the data variable names with the specified number of dimensions (rank).

Test:: New in 3.0

virtual int VAPoR::DataMgrV3_0::GetDimLensAtLevel	(	string	varname,
		int	level,
		bool	spatial,
		std::vector< size_t > &	dims_at_level,
		std::vector< size_t > &	bs_at_level
	)		const

virtual

Return a variable's dimension lengths at a specified refinement level

Compressed variables have a multi-resolution grid representation. This method returns the variable's ordered spatial and temporal dimension lengths, and block dimensions at the multiresolution refinement level specified by level.

If the variable named by varname is not compressed the variable's native dimensions are returned.

Parameters

[in]	varname	Data or coordinate variable name.
[in]	level	Specifies a member of a multi-resolution variable's grid hierarchy as described above.
[out]	dims_at_level	An ordered vector containing the variable's dimensions at the specified refinement level
[out]	bs_at_level	An ordered vector containing the variable's block dimensions at the specified refinement level

Return values

status Zero is returned upon success, otherwise -1.

See also: VAPoR::DC, DC::BaseVar::GetBS(), DC::BaseVar::GetDimensions()

Parameters

[in] spatial A boolean, if true, indicates that only the variable's spatial dimensions and block size should be returned. A time varying dimension, if one exists, is not included

virtual string VAPoR::DataMgrV3_0::GetMapProjection ( ) const

inlinevirtual

Todo:: Need to define

Definition at line 528 of file DataMgrV3_0.h.

int VAPoR::DataMgrV3_0::GetNumRefLevels ( string varname ) const

Return the number of refinement levels for the indicated variable

Compressed variables have a multi-resolution grid representation. This method returns the number of levels in the hiearchy. A value of one indicates that only the native resolution is available. A value of two indicates that two levels, the native plus the next coarsest are available, and so on.

Parameters

[in] varname Data or coordinate variable name.

Return values

num	If `varname` is unknown zero is returned. if `varname` is not compressed (has no multi-resolution representation) one is returned. Otherwise the total number of levels in the multi-resolution hierarchy are returned.

int VAPoR::DataMgrV3_0::GetNumTimeSteps ( string varname ) const

Return the time dimension length for a variable

Returns the number of time steps (length of the time dimension) for which a variable is defined. If varname does not have a time coordinate 1 is returned. If varname is not defined as a variable a negative int is returned.

Parameters

[in] varname A string specifying the name of the variable.

Return values

count The length of the time dimension, or a negative int if varname is undefined.

See also: IsTimeVarying()

void VAPoR::DataMgrV3_0::GetTimeCoordinates ( std::vector< double > & timecoords ) const

inline

Get time coordinates

Get a sorted list of all time coordinates defined for the data set. Multiple time coordinate variables may be defined. This method collects the time coordinates from all time coordinate variables and sorts them into a single, global time coordinate variable.

Note: Need to deal with different units (e.g. seconds and days).; Should methods that take a time step argument ts expect "local" or "global" time steps

Definition at line 208 of file DataMgrV3_0.h.

VAPoR::RegularGrid* VAPoR::DataMgrV3_0::GetVariable	(	size_t	ts,
		string	varname,
		int	level,
		int	lod,
		bool	lock = `false`
	)

Read and return variable data

Reads all data for the data or coordinate variable named by varname for the time step, refinement level, and leve-of-detail indicated by ts, level, and lod, respectively.

Parameters

[in]	ts	An integer offset into the time coordinate variable returned by GetTimeCoordinates() indicting the time step of the variable to access.
[in]	varname	The name of the data or coordinate variable to access
[in]	level	Grid refinement level. See DataMgr
[in]	lod	The level-of-detail parameter, `lod`, selects the approximation level. See DataMgr.

VAPoR::RegularGrid* VAPoR::DataMgrV3_0::GetVariable	(	size_t	ts,
		string	varname,
		int	level,
		int	lod,
		std::vector< double >	min,
		std::vector< double >	max,
		bool	lock = `false`
	)

Read and return a variable hyperslab

This method is identical to the GetVariable() method, however, a subregion is specified in the user coordinate system. min and max specify the minimum and maximum extents of an axis-aligned bounding box containing the region of interest. The VAPoR::RegularGrid object returned contains the intersection between the specified hyperslab and the variable's spatial domain (which is not necessarily rectangular or axis-aligned). If the requested hyperslab lies entirely outside of the domain of the requested variable NULL is returned

Parameters

[in]	min	A one, two, or three element array specifying the minimum extents, in user coordinates, of an axis-aligned box defining the region-of-interest. The spatial dimensionality of the variable determines the number of elements in `min`.
[in]	max	A one, two, or three element array specifying the maximum extents, in user coordinates, of an axis-aligned box defining the region-of-interest. The spatial dimensionality of the variable determines the number of elements in `max`.

Note: The RegularGrid structure returned is allocated from the heap. it is the caller's responsiblity to delete the returned object when it is no longer in use.

VAPoR::RegularGrid* VAPoR::DataMgrV3_0::GetVariable	(	size_t	ts,
		string	varname,
		int	level,
		int	lod,
		std::vector< size_t >	min,
		std::vector< size_t >	max,
		bool	lock = `false`
	)

int VAPoR::DataMgrV3_0::GetVariableExtents	(	size_t	ts,
		string	varname,
		int	level,
		std::vector< double > &	min,
		std::vector< double > &	max
	)

Compute the coordinate extents of a variable

This method finds the spatial domain extents of a variable

This method returns the min and max extents specified in user coordinates, of the smallest axis-aligned bounding box that is guaranteed to contain the variable indicated by varname, and the given refinement level, level

virtual int VAPoR::DataMgrV3_0::Initialize ( const std::vector< string > & files )

virtual

Initialize the class

This method must be called to initialize the DataMgrV3_0 class with a list of input data files.

Parameters

[in] files A list of file paths

Return values

status A negative int is returned on failure and an error message will be logged with MyBase::SetErrMsg()

bool VAPoR::DataMgrV3_0::IsCompressed ( string varname ) const

Return a boolean indicating whether a variable is compressed

This method returns true if the variable named by varname is defined and it has a compressed representation. If either of these conditions is not true the method returns false.

Parameters

[in] varname A string specifying the name of the variable.

Return values

bool	Returns true if variable `varname` exists and is compressed

See also: DefineCoordVar(), DefineDataVar(), DC::BaseVar::GetCompressed()

bool VAPoR::DataMgrV3_0::IsTimeVarying ( string varname ) const

Return a boolean indicating whether a variable is time varying

This method returns true if the variable named by varname is defined and it has a time axis dimension. If either of these conditions is not true the method returns false.

Parameters

[in] varname A string specifying the name of the variable.

Return values

bool	Returns true if variable `varname` exists and is time varying.

bool VAPoR::DataMgrV3_0::IsVariableDerived ( string varname ) const

Return true if the named variable is the output of a pipeline

This method returns true if varname matches a variable name in the output list (PipeLine::GetOutputs()) of any pipeline added with NewPipeline()

See also: NewPipeline()

bool VAPoR::DataMgrV3_0::IsVariableNative ( string varname ) const

Return true if the named variable is availble from the derived classes data access methods.

A return value of true does not imply that the variable can be read (

See also: VariableExists()), only that it is part of the data set known to the derived class; NewPipeline()

void VAPoR::DataMgrV3_0::PurgeVariable ( string varname )

Purge the cache of a variable

Parameters

[in] varname is the variable name

void VAPoR::DataMgrV3_0::UnlockGrid ( const VAPoR::RegularGrid * rg )

Unlock a floating-point region of memory

Decrement the lock counter associatd with a region of memory, and if zero, unlock region of memory previously locked with GetVariable(). When the lock counter reaches zero the region is simply marked available for internal garbage collection during subsequent GetVariable() calls

Parameters

[in] rg A pointer to a RegularGrid previosly returned by GetVariable()

Return values

status Returns a non-negative value on success

See also: GetVariable()

virtual bool VAPoR::DataMgrV3_0::VariableExists	(	size_t	ts,
		string	varname,
		int	level = `0`,
		int	lod = `0`
	)

virtual

Returns true if indicated data volume is available

Returns true if the variable identified by the timestep, variable name, refinement level, and level-of-detail is present in the data set. Returns false if the variable is not available.

Parameters

[in]	ts	A valid time step between 0 and GetNumTimesteps()-1
[in]	varname	A valid variable name
[in]	level	Refinement level requested.
[in]	lod	Compression level of detail requested. refinement level contained in the DC.

The documentation for this class was generated from the following file:

/Users/clyne/src/vapor-3.0/include/vapor/DataMgrV3_0.h

Classes

Public Member Functions

Additional Inherited Members

Detailed Description

Constructor & Destructor Documentation

Member Function Documentation