data_observations_filter (03/01/16)

CRHM Platform

Observation Filter

    Observation filters allow the observation data to be preprocessed before being used in the model.  Filters are declared in the observation heading after the declaration of variables.  Examples of their use follows:

  • relative humidity is often the meteorological observation, whereas vapour pressure is the common input to model modules.  One solution is to make the individual modules handle the change of variable but this makes the modules more complex and slower.  The conversion has also to be done in every module requiring vapour pressure. A more efficient solution is to make the data conversion once when the data is initially read into the model directly from the input data file.
  • If  wind measurements are made at 10 metres and the model modules require a wind referenced at 2 metres.  One solution is again to make the modules handle the conversion of measurement height but this necessitates defining the relevant translation parameters in every module.

Filters

    The filters are included after the data variables in the file are defined but before the line of '#' symbols separating the data header from the actual observation data.  Filter lines begin with a '$' followed by the variable name for the generated data, i.e. $ea.

    The data filtering filters; "missing", "missinginter", "missingrepl" and "missingFlagAfter" are best used separately and the results saved and used as a new observation file. This limitation arises because all the data must be read to determine the last "good value" and other filters may used data values before missing values are corrected.

    The filter name follows with the required parameters enclosed in brackets.  The parameter list consists of variables included in the observation file or variables generated by earlier filters and the numerical constants used by the filter.  An example of a filter to modify wind reference height is:

$u2 refwind(u, 10, 2, 1) "this is a  comment", where parameters for refwind are refwind(u, Zm, Z2, Ht)

    where:

u2 = u1*log((Z2 - d)/Z)/log((Zm - d)/Z)

u2 (m/s) is the name of the new variable,

Zm (m) - the actual measurement height = 10,

Z2 (m) - desired reference height = 2,

Ht (m) the vegetation height = 1, and it assumed that

d = 2/3*Ht and

Z = 0.123*Ht.

 

   Both  spaces or commas can be used to delimit parameters.  Any input after the closing bracket is a comment and will appear in the program variable help box.

Similarly the filter for vapour pressure is:

$ea ea(t, rh) where t is the temperature observation name and rh (%) is the relative humidity observation name.

where:

ea = sat _ea(t)*rh/100.0,

t is the temperature measurement and

rh (%) is the relative humidity.

Defined filters

$$ comment line.

$u2 refwind(u, Z2, Zm, Ht) u = wind at reference height Zm,  u2 = desired wind at reference height Z2 and Ht = vegetation height.

$ea ea(t, rh)  ea = desired vapour pressure (kPa) for t = temperature and rh (%) = the relative humidity.

$rh rh(t, ea)  rh = desired relative humidity (%) for t = temperature and ea (kPa) = the vapour pressure.

$RHi RH_WtoI(t, rh) RHi = RH w.r.t. ice of moist air at ambient temperature t, rh = RH w.r.t. water.

$Tc FtoC(Tf) Tc = conversion of Tf(°F) to Tc(°C).

$Tc KtoC(Tk) Tc = conversion of Tk(°K) to Tc(°C).

$Tk CtoK(Tk) Tk = conversion of Tc(°C) to Tk(°K).

$d missingC(var, c1, c2, c3), where var is the data being checked. Values less than or equal c1 or greater than or equal c2 are replaced with c3.

$d missing0(var, c1, c2), where var is the data being checked. Values less than or equal c1 or greater than or equal c2 are replaced with 0.0.

$d missing(var, c1, c2), where var is the data being checked. Values less than or equal c1 or greater than or equal c2 are replaced with the most recent 'good' value.  If first line of data is not 'good' data,  warning is issued.  Data replacement does not begin until after an interval with 'good' data.

$d missinginter(var, c1, c2), where var is the data being checked. Values less than or equal c1 or greater than or equal c2 are replaced with a linearly interpolated value calculated from the   'good' values before and after the missing values.  If first line of data is not 'good' data,  warning is issued.  Data replacement does not begin until the first 'good' data after the 'bad' data.  Missing data at the end of the file is left unchanged.

$d missingrepl(var, c1, c2, var2), where var is the data being checked. Values less than or equal c1 or greater than or equal c2 are replaced with the value from var2.

$d missingFlag(var, c1, c2), where var is the data being checked. Output is 0.0 except when the value is less than or equal c1 or greater than or equal c2, when the output is 1.0.  Value is checked before any replacement filters are executed.

$d missingFlagAfter(var, c1, c2), where var is the data being checked. Output is 0.0 except when the value is less than or equal c1 or greater than or equal c2, when the output is 1.0.  Value is checked after any replacement filters are executed.

$p smear(p, Time1, Time2) p = daily precipitation as first value of day, Time1 = start time or <= 0 to indicate only negative values of precipitation have to be processed.  Time2 is stop time or <= 0 for end of file.

$a abs(var) a = absolute value of variable

$a add(var, c) a = variable var plus constant c.

$s sub(var, c) s = variable var minus constant c.

$m mul(var, c) m = variable var multiplied by constant c.

$d div(var, c) d = variable var divided by constant c.

$pow powV(var, A,  B) pow = A*varB .

$exp expV(var, A, B) exp = A*eB*var .

$log logV(var A B) log = A*ln(var*B).

$C const(c) C = constant c.

$a addV(var, var1) a = variable var plus variable var1.

$s subV(var, var1) s = variable var minus variable var1.

$m mulV(var, var1) m = variable var multiplied by variable var1.

$d divV(var, var1) d = variable var divided by variable var1.

$R random(seed) series of random numbers initialised by seed of last call. Random number generator is shared between all calls.

$TimeShift(Ts), where Ts is the time shift in days.  Negative values move the file time backwards.  The fractional portion should an integral number of time intervals, i.e. n*1/24, n*1/48 etc..

Using ForceInterval to Change Observation File Interval.

    This filter changes the time step interval of an entire observation file.  Depending upon the initial interval the interval length can increase or decrease. Calling,

$ForceInterval(48)

    will create 30 minute interval data. Calling,

$ForceInterval(6)

    will create 4 hourly interval data.

The range of the frequency parameter is 1 to 288.  Special consideration has to be given to "rate" inputs with units of mm/int, MJ/int etc., where the daily sum has to be kept constant despite the change in time interval.  The inputs have to be flagged with a negative column count. N.B. observations like daily precipitation (ppt),   mm/d are not affected and are always positive.

sample header

p -1 (mm/int) N.B. negative sign.  If it had been daily precipitation it would be be "ppt 1 (mm/d)".

Qsi 1 (W/m^2)

QnD 1 (MJ)

rh 1 ()

SWE 1 (mm)

t 1 (°C)

u 1 (m/s)

$ForceInterval(24) change interval to hourly from what ever it was originally.

########

Time Simulation (Can only be used in a separate *.obs file containing no real data).

    Sometimes it would be convenient to generate synthetic data instead using field observations.  A special filter makes this possible.

$Sim(StartTime, EndTime, Interval)  where

StartTime as mm/dd/yy or mm/dd/yyyy

EndTime as mm/dd/yy or mm/dd/yyyy

Interval in hours. Minimum 0.5

When this filter is put in an observation file,  no interval data is ever read but time periods from 'StartTime' to 'EndTime - 1 Interval' are generated at the interval specified.  The time simulation filter does not generate time fields unless additional filters are used to define the data to be generated, e.g. 'const(C)' etc..

Wave Synthesis (Can only be used in a separate *.obs file with $Sim(... ) and not in a regular observation file).

$Fract sine(period phase start end)

$Fract sin(period phase start end)

$Fract cos(period phase start end)

$Fract square(period phase start end)

$Fract ramp(period phase start end)

$Fract pulse(start end), where

period is in days. For less than one day the time format can be used, e.g. 12 hours can be '0.5' or '12:00:00'.

phase is in days. For less than one day the time format can be used, e.g. 12 hours can be '0.5' or '12:00:00'.

start is start date (mm/dd/yy) or (mm/dd/yy_hh:mm:00)

end is end date (mm/dd/yy or (mm/dd/yy_hh:mm:00))

Fract will vary between -1.0 and 1.0 depending on the function.

Value is the value calculated by the function.

 

$Value exp(start end A B)          Value = A*eB(t - t0)

$Value log(start end A B)            Value = A*ln(B*(t - t0))

$Value pow(start end A B)         Value = A*(t - t0)B

$Value poly(start end a0 a1 a2 a3 a4)      where X = (t - t0), Value = a0 + a1*X + a2*X2 + a3*X3 + a4*X4

Example of Wave Synthesis.

Simulation test (06/10/02)

$$ comment line - Test pulse generation

$Sim(01/01/01, 01/05/2001, 1) simulation time period

$Fract pulse(01/02/01, 01/03/01) one day pulse

$P pulse(01/01/01_1:0:0, 01/01/01_12:0:0)

$S sin(12:0:0, 0, 01/01/01, 01/02/01) // legacy sine is also acceptable

$S cos(12:0:0, 0, 01/01/01, 01/02/01)

$R ramp(1, 0, 01/01/01, 01/02/01)

$Q square(1, 0, 01/01/01, 01/02/01)

$$ coment line - delayed functions

$P1 pulse(01/03/01, 01/04/01)

$S1 sine(1, 0, 01/03/01, 01/04/01)

$R1 ramp(1, 0, 01/03/01, 01/04/01)

$Q1 square(1, 0, 01/03/01, 01/04/01)

############################################