Adaptive Plan View Plotting

Note: An updated version of this document is provided with the 5.1.1 documentation.

Table of Contents

1) Introduction.

Adaptive Plan View Plotting refers to a new AWIPS capability that allows one to add new kinds of plan view plots or change the way existing ones look by only changing plain text static metadata.

In order to make this work, the data to be plotted needs to be in a self describing format. To this end, the plotting of all point data sets now is done from netCDF files. This means that the `plot files' that were previously used for this function for metar, raob and profiler data are no longer applicable. Also, existing netCDF files needed to be enhanced by adding some additional variables and attributes. However, none of the existing variables or attributes have been changed so existing applications that read netCDF point data should still work.

While there is a performance penalty for this additional flexibility, it is not prohibitive. The additional flexibility should be extremely useful for LDAD data, allowing each site to plot whatever data ends up in their LDAD netCDF files.

2.0) Enhancements to the netCDF files.

Before one attempts to manipulate the display characteristics of point data sets, it is helpful to understand the enhancements that have been made to the netCDF files to support this. Here, in cdl format, are the additional variables and attributes that have been added: The main reason to be familiar with these enhancements is so that in the event that one needs to add an additional variable or attribute to an existing netCDF file, one can avoid using these reserved variables and attributes. We refer to these enhancements as `record management' variables and attributes. An implementor also needs to understand that the record management variables are not displayable.

Of all the new global attributes, the :cdlDate attribute is the most important. If one ever changes the structure of the data files in a directory (add, delete, or move a variable) the value of the :cdlDate attribute needs to be changed. This allows the software to recognize that it has to treat the new files differently.

3.0) Key Entries for Plan View Plot Displays.

Under the new paradigm, menu entries and product button entries work exactly as before to display a given depict key. Depict key and data key entries are somewhat different. Here is an example depict key entry; we are using the standard metar plot from depictInfo.manual: If you look at depict key entries for plan view plots, several things are noticable. First, all entries have a depictable type (second vertical bar delimited column) of 72. This is because all of these displays use the exact same software module to create the display; what makes them different is the associated static metadata. Second, the third column, which is the list of associated data keys, contains an extra data key (27001 in this case). This new data key points to what is refered to as a `design file'. The first and last data keys point to the data directory and static progressive disclosure file, respectively, as always. For the new adaptive plan view plotting, the first data key must point to the data directory, the second must point to a design file, and the third optional key must point to a static progressive disclosure file.

Here are the associated data keys from dataInfo.manual:

As previously mentioned, these keys for the most part simply refer to the data directory, the design file, and static progressive disclosure file, respectively. The default versions of all of the design files can be found in localization/nationalData/. Notice that the data directory key has the associated cdl file name in the description field. This refers to a version of this cdl residing in localization/nationalData/. It allows the localization to automatically publish this cdl to the data directory, using ncgen, as the template file.

4.0) Design Files.

This is the most important section of this document, because the design files are where one exerts most of the control over how plan view plots are created. All of the default design files exist in nationalData/ and have file names that look like *Design.txt. They are subject to copy override from realization, site specific, and customization files (for more info on file override, see section 5.0 of localization.doc).

We will begin by discussing the overall structure of design files. Then an example of a design file will be presented and discussed in detail.

4.1) Overall structure of design files.

Design files are plain text files. As the text for a design file is interpreted, any line that is all spaces, blank or begins with // is ignored. A backslash at the very end of a line is considered a line continuation. Each interpreted line consists of a keyword followed by one or more space delimited arguments. A keyword and its arguments must appear on the same line; the continuation can be used to make long entries more readable. Because arguments are primarily space delimited, the user is allowed to use a tilde (~) to stand for a space in string arguments. In such strings, an escaped tilde (\~) is used to represent a tilde. Spaces are not allowed in file names, item ids, keywords and other reserved words, so escaping is generally ignored for these.

A design file is broken into two major sections, the global definitions and the item definitions. The global definitions are all entries that refer to the design file as a whole; the item definitions are all entries that refer to a specific item. All global definitions must appear before any item definitions, but the individual global definitions can be in any order. A complete list of keywords for both global and item definitions can be found in appendix I.

The presence of an `item_id' keyword is what triggers the existence of an item. The entries that define an item span from its corresponding item_id keyword to the entry immediately before the next item_id keyword. The item_id keyword is always the first entry of an item definition, but within an item definition other individual entries may appear in any order. Also, the items as a whole may appear in any order.

An item is a logical entity that may contain raw data from a netcdf file, the result of a function, a constant, or a display method. Each item that contains a display method must also contain raw data or a function result. Containing raw data, a function result, or a constant are mutually exclusive, and it is not allowed to have a display method associated with a constant.

Each item is defined by its characteristics and the data it contains. Here we will focus on what these characteristics mean, how they are determined will be discussed later. One primary characteristic of an item is its dimensionality; which can be constant, scalar or list. A constant item contains just one constant value, a scalar item contains one value per station or record, and a list item contains two or more values per station or record. Another primary characteristic of an item is its indexing; most items contain data on a per station basis, but some contain some data on a per netcdf file record basis. Station versus record indexing is meaningless for a constant item. A third primary characteristic is the base data type. The base data types are float, int, short, and string. It is also possible to talk of the major data type of an item, which is either string or numeric. Constants only have a major type, string versus numeric. The most important characteristic of an individual data value is whether or not it is null. Each base data type has a corresponding well defined null value, 9e99 for doubles, 1e37 for floats, 2147483647 for ints, 32767 for shorts, and a zero length string for strings. The null double value is important because numeric constants are held as doubles and all numeric values from lookup tables are presented as doubles. Where it is important to assign an arbitrary non-null value to a data item, 0 is used for numeric values and "0" is used for strings.

A constant item (not shown in the example) contains only one additional entry beside its item_id, with a keyword of `constant'. If the first argument is a single equals sign followed by another argument that is a numeric value, then the constant will be numeric. Otherwise the first argument will be used to define a string constant. Spaces are allowed in string constants, so here a tilde should be used for a space. It is possible to override the value of a constant in a design file directly from the associated depict key entry. This is done in the 13th vertical bar delimited field, the so called extraInfo field, which contains a list of comma delimited strings. In this case, each pair of strings is assumed to be the id of a constant item followed by a override value.

An item is identified as containing raw data primarily by the presence of a `netcdf_id' keyword. This keyword is followed by the name of the netcdf variable that it reads data from. The most common other keywords for a raw data item are `type', `dimension' and `record'. If the argument of the record keyword is `true', then that item will contain values corresponding to records in the netcdf file. Otherwise, it will contain data for each station. This distinction is important because a single station may have data available at multiple times. The possible arguments for the keyword type are `string', `float', `int', and `short'. The reason the user supplies this information rather than it being obtained from the netCDF file is so that the validity of function relationships can be determined when the design file is parsed. The possible arguments for the keyword dimension are `scalar' and `list'. If scalar, there is one value for each record, meaning that variable should be dimensioned in the netcdf file as (recnum), or (recnum,strlen) in the case of a string. If list, then there are multiple values for each record, meaning that variables should be dimemsioned (recnum,listlen), or (recnum,listlen,strlen) in the case of a string. Here recnum must refer to the UNLIMITED dimension, listlen and strlen to any valid dimension appropriate for the variable.

An item is identified as containing a function output primarily by the presence of a `function' keyword. This keyword is followed by the function name (see appendix II for a complete list of available functions). A function item must also contain a entry with an `inputs' keyword. The arguments that follow the inputs keyword are a list of the item_id's that are input to the function. The items that are input to a function may be other function outputs, raw data, or constants, depending on the requirements of the function. There is no need to specify the data type or dimensionality of a function; that is all determined by the characteristics of the inputs and the function used.

An item is identified as having a display method primarily by the presence of the keyword `placement'. The kind of display is determined by the argument to the keyword `method'. For some methods, the placement is predetermined by the method and the argument to the keyword placement is just `free'. Otherwise it is one of nine predefined placement positions (upper_left, center, etc.). While most display methods require only one input, some require multiple inputs. When this is the case, a special function called `gather' is used to hand multiple items to a single display method. The reader is directed to appendix III for a complete list of display methods.

4.2) A design file example.

What follows is a sample design file. Again we use the standard metar plot as an example. In this example the last global definition is `time_step 3600' and the first item definition is `item_id RT'. The entry `size 45' specifies that each plot model is to be treated as 45 pixels in size for the purpose of calculating progressive disclosure. The entry `time_step 3600' means that each graphic of this type in a time sequence will be separated by one hour, which is 3600 seconds. All time periods in design files are in units of seconds.

Normally, when the code figures out which data record to copy to a station, the record with a time closest to the valid time of the whole data set is used. Here, the item RT is used to specify that we prefer to use observations with a report type of METAR if avaliable, instead of a SPECI that may be closer to the data set time. To make this work, this item is specified to be a record item and an entry with the keyword `indexing_table' is supplied. The file `rank_report_type.txt' contains the data for what is called a Lookup Table (see section 5.0). This particular table is a string to number lookup table, and is used to assign arbitrary rankings to the strings METAR and SPECI.

The item TXT is reading in the raw text of the metar. Since the keyword `sample' is there with the argument of true, this text is what is presented for sampling. In order to sample, an item must be a scalar string type with station indexing.

The item CLOUDS is just pulling in the cloud information but not directly displaying it. That will be done by a later item. Note that the display does not even necessarily have to be done by the immediately following item. Note that this item has a `list' argument to its `dimension' keyword because there can be multiple cloud decks.

The items DPTYP and DPVAL are used to pull in the characteristic and amount of the pressure change. The item DP3 has an example of the gather function, which is a special function used only to pass one or more previously defined items to a display method. The display method here is `trend', which is specifically tailored for the task of displaying pressure change data. In the design of the capability, we have tried to avoid specifically tailored functions or display methods, but a few exist. Note that the units conversion from pascals to tenths of millibars is done with the generic `multiplier' keyword rather that being folded into the method.

The items SPD, DIR and GUST are used to pull in the wind speed, the wind direction, and gust speed, respectively. The item WIND uses the gather function to present these to the `barb' method. Note than since this method predetermines the placement of what is being plotted, the placement of `free' is used. The multiplier is used to convert from meters per second to knots.

The item WX is an example of a raw data item that is immediately handed to a display method. The method `translation' specifies that a string to string lookup table (see section 5.1) is expected, and the recursive translation method of that table will be called to produce the text for output. The `table_file' keyword is used to supply the name of the file containing the lookup table. All text that cannot be translated will be shown as plain ascii, but the text that is translated will be displayed using the `weather' character set. See the third character set in the document characterSets.doc.

The item TC is used to pull in the value decoded from the regular temperature field in the metar, which is in degrees celcius. U.S. metars also sometimes have a temperature in tenths (used to recover the precision of the farenheit observation) in the remarks section. The T10 item is used to pull that in. Then the T item pulls these two in using the `or' function and displays them. The order is important; by putting T10 first, it will use the T10 value if available, and revert to using the TC value otherwise. These kelvin temperatures are converted to farenheit using the multiplier and offset of 1.8 and -459.67. The `min_trans' and `max_trans' keywords are used to supply sanity check limits to the results of the units conversion. A completely analogous thing happens with the items TdC, Td10, and Td in order to display the dewpoint.

The item SLP is yet another example of handing raw data directly to a display method. The multiplier of 0.1 turns pascals to tenths of millibars, and `%5d' is used to format that result. The keyword `trim_count' is followed by the number of characters to strip off the front of the formatted string before displaying it. This is how we follow the convention of not displaying the hundreds of millibars in the pressure field.

The item COVER is used to do the display work for the data we pulled in using the CLOUDS item. Note that it does not matter that it does not immediately follow the CLOUDS item. Recall that the item COVER was a list item. The function `select' is used to choose one value from this list. The file named as the argument to the `function_table' keyword is a string to number table that is used to assign arbitrary rankings to each cover type; the cover type with the lowest ranking number is used. The display method used is `lookup', which means use a lookup table to do a simple translation (not the recursive one done for WX) to produce the output text. The table used is a string to string table defined in cloud_chars.txt, and the result is displayed using the `special' character set (fourth character set in characterSets.doc).

5.0) Lookup Table Files.

As mentioned before, lookup table files are a means by which a user can perform arbitrary data converions. Lookup table files are plain ascii files, with blank lines and lines that begin with a `//' ignored, just as with design files. There are four types of lookup table files. The first line of any lookup table file is a three character string that designates its type. All of the default lookup table files exist in nationalData/ and have file names that look like *_*.txt. Just like design files, they are subject to copy override from realization, site specific, and customization files. Since space delimited parsing is used in these files, one needs to use a tilde (~) to designate the existence of a space in an input string. An escaped tilde (\~) can be used to designate a tilde.

5.1) String to string lookup table.

The first line of a string to string lookup table is `s2s'. A string to string lookup table is used to convert some arbitrary string to some other string. Most entries are a line with a lookup string followed by a result string to translate it to. Sometimes, there is a need to have characters in the result string that are unprintable in ascii. In that case, there is the option of expressing the result string with a colon followed by a list of integers designating the character codes (all space delimited). It is also possible to make an entry with only a lookup string, which means that the result of translating that string will be an empty string.

There are four special keywords that are recognized in a string to string table. Once a line has been encountered that does not have one of these keywords on it, further occurences of these keywords will be treated as a regular lookup string. Normally, if an attempt is made to translate an input string that does not exist in the table as a lookup string, the output will be some constant default string. The keyword `pass', if present, indicates that if the input string is not available as a lookup string, then the resulting translation should be the input string. The usual default string is an empty string, but using the keyword `default' allows one to supply a different default string. There is a special type of translation available for the string to string table that will attempt to translate all possible substrings in the input string, as well as the entire string. The keywords `left' and `right' allow one to perform an edit operation on the input string before it is translated in this case. The arguments after the left or right keyword are a lookup and result string expressed exactly as in a regular entry. For the left keyword, the first occurence of the corresponding lookup string is located, and all text up to that occurence is replaced with the corresponding result text. The right keyword works analogously with the last occurence and the end of the string to translate.

5.2) String to number lookup table.

The first line of a string to number lookup table is `s2n'. A string to number lookup table is used to convert some arbitrary string to some numeric value. Most entries are a line with a lookup string followed by a numeric value to translate it to. Internally, numeric values in a string to number table are held as doubles.

There are three special keywords that are recognized in a string to number table. Once a line has been encountered that does not have one of these keywords on it, further occurences of these keywords will be treated as a regular lookup string. If an attempt is made to translate an input string that does not exist in the table as a lookup string, the output will be some constant default value. Normally, this default value is 9e99, universally recognized by the PlotDesign class as a null double value. Using the keyword `default' allows one to supply a different default numeric value. There is a special type of translation available for the string to number table that will attempt to scan all possible substrings for a possible lookup string, as well as the entire string. The keywords `left' and `right' allow one to perform an edit operation on the input string before it is translated in this case. The arguments after the left or right keyword are a lookup string and a translation string. For the left keyword, the first occurence of the corresponding lookup string is located, and all text up to that occurence is replaced with the corresponding translation string. The right keyword works analogously with the last occurence and the end of the string to translate.

5.3) Number to string lookup table.

The first line of a number to string lookup table is `n2s'. A number to string lookup table is used to convert some arbitrary numeric value to a string. Most commonly, entries are a single number or a pair of numbers, space delimited, followed by a result string. A pair of numbers would represent a range of values that would lookup as the result string. If there are overlapping ranges, order will be important because the first range that matches will be used. Sometimes, there is a need to have characters in the result string that are unprintable in ascii. In that case, there is the option of expressing the result string with a colon followed by a list of integers designating the character codes (all space delimited). It is also possible to make an entry with only one or a pair of numbers, which means that the result of that lookup will be an empty string. Internally, numeric values in a number to string table are held as doubles.

There is one special keyword that is recognized in a number to string table. Normally, if an attempt is made to lookup a number that does not exist in the table or is outside all specified ranges, the output will be an empty string. Using the keyword `default' allows one to supply a different string to be output in the case of a failed lookup.

5.4) Number to number lookup table.

The first line of a number to number lookup table is `n2n'. A number to number lookup table is used to convert some arbitrary numeric value to another numeric value. Most commonly, entries are a list of two, three, or four numbers, space delimited. Two numbers means translate the first number to the second. Three numbers means that if the input number falls between the first two numbers, the result is the third number. Four numbers means that if the input number falls between the first two numbers, do a linear interpolation to the range represented by the last two numbers.

There are two special keywords that are recognized in a number to number table. Normally, if an attempt is made to lookup a number that does not exist in the table or is outside all specified ranges, the output will be 9e99, universally recognized by the PlotDesign class as a null double value. Using the keyword `default' allows one to supply a different number to be output in the case of a failed lookup. If the keyword `pass', is present, the a failed lookup results in the number input to the lookup being passed through.

Appendix I.
Keywords.

By convention, all keywords are all lower case with underscores. This discussion will divide keywords into five categories, global, item, raw data, function, and display. Unless otherwise mentioned, the reader should assume that each keyword takes one and only one argument. The reader should also note that all time periods in design files are in seconds.

All global keywords must appear in the design file before any keywords from other categories. All global keywords refer to the file as a whole, all other keyword are specific to their corresponding item.

Global keywords.

size
The argument of the `size' keyword is the number of pixels in size to assume each station model plot is. This is used for progressive disclosure calculations.

time_step
The argument of the `time_step' keyword is the number of seconds separating the valid times of each displayable frame. For example 3600 would be one hour; thus a graphic would be created with valid times on the hours. By default, this could potentially contain observations ranging in time from 30 minutes before to 30 minutes after the hour.

default_period
By default, a display will accept data from a time bin the same size as the value for `time_step'. The `default_period' keyword allows one to set the size of the time bin over which data will be allowed to appear in the graphic. The bin is always centered at the valid time assigned to the display.

predefined_only
If the argument to the keyword `predefined_only' is `true', then only station appearing in the static progressive disclosure file provided will plot. By default, any station for which no static progressive disclosure entry is available will plot if its latitude and longitude can be found in the netcdf file. In this case, these station will never display at a zoom level lower than nearby neighbors that do have static progressive disclosure entries.

dyn_prog_disc
The `dyn_prog_disc' is used to control whether and how progressive disclosure is calculated on the fly based on which data is available. One can divide stations to plot into two categories, those with static progressive disclosure and those without. In the default case, those stations with static progressive disclosure will retain it. Those without will have their progressive disclosure dynamically computed if there are not too many stations, otherwise they will all be assigned some small constant value that guarantees they will only show up at very high zooms or densities. If the argument to dyn_prog_disc is `none' then stations missing progressive disclosure will always be assigned the small constant value. If the argument to dyn_prog_disc is `missing' then stations missing progressive disclosure will always have progressive disclosure dynamically computed. If the argument to dyn_prog_disc is `all', all stations, including those for which static progressive disclosure information was available, will have their progressive disclosure dynamically computed based on which stations actually have data.

stations_path
If this keyword is present, the argument is a fragment of a file name that the list of stations rendered will be written to each time a frame is drawn. It is written into the localizationDataSets/LLL/ directory and has an extension with the process id encoded.

hard_accums
By default, when performing an accumulation in order to convert data from record to station indexing, as long as at least one non-undefined value is available, a value will be created for the accumulation. If the argument of `hard_accums' is `record', then there must have been a record available for each valid period that made up the accumulation. If the argument of `hard_accums' is `true', there must have been a non-undefined value available for each valid period that made up the accumulation.

sample_format
Normally, when sample text is output, whatever text is in the item designated as being the sample item is just written to the screen as is. The `sample_format' can take one or more arguments, each being a separate line of formatting information. The standard ~ is used as a place holder for a space. As the text to be formatted is scanned, each %c results in the next character being used in the output, and each %s results in the next space delimited word being used in the output. If the last format specification is a %s, then that format specification will be replaced with whatever text is left regardless of spaces. If in the course of formatting the text to be formatted is exhausted, the rest of the format specifications will be replaced with empty strings.

diag_dump
The keyword `diag_dump' takes a numerical argument. If present, it can cause diagnostics to be written to stderr as the design file is parsed and used to obtain data. A value >=10 invokes minimal diagnostics. A value >=20 invokes detailed diagnostics. A value >=30 invokes diagnostics that includes information about individual records and stations processed. A ones place of 9 will result in outputting the contents of all the data items.

Item keywords.

item_id
The presence of an `item_id' keyword is what triggers the existence of an item. The entries that define an item span from its corresponding item_id keyword to the entry immediately before the next item_id keyword. The item_id keyword is always the first entry of an item definition, but within an item definition other individual entries may appear in any order.

sample
The `sample' keyword must be applied to an item that has a scalar string type and is station indexed. When present with an argument of `true', it will cause the text from this item to be used for sampling. This keyword should only be used on one item.

remove_by
The `remove_by' keyword must be applied to an item that has scalar dimensionality and is record indexed. All basic data types, string, float, int and short, have a well defined null value. When present with an argument of `true', it will cause all records for which this item has a null value to be removed from consideration for plotting. This keyword should only be used on one item.

indexing_table
The `indexing_table' keyword must be applied to an item that has scalar dimensionality and is record indexed. This allows the value of this data item to be taken into account when selecting which of multiple records applying to the same station should be selected for display. The argument to indexing_table must be the filename of a lookup table with numeric output and input appropriate to the item. This table is used to convert values in this item into arbitrary rankings; lower ranking numbers are most preferred for display. This keyword should only be used on one item.

goodness_table
The `goodness_table' keyword must be applied to an item that has scalar dimensionality and is station indexed. This allows the value of this data item to be taken into account when calculating progressive disclosure. The argument to goodness_table must be the filename of a lookup table with numeric output and input appropriate to the item. This table is used to convert values in this item into goodness values; higher goodness values are most preferred for display. The reader is directed to va_driver.doc for more information on goodness values.

accum_period
The `accum_period' keyword must be applied to an item that is station indexed. Furthermore, it should only be applied to an item that is also raw data or is the result of a call to the `rec_to_sta' function. The value associated with the `valid_period' keyword for this item directly affects the result of using this keyword. When a non-zero value for the accum_period keyword is present (0 is the default), then an alternate method is used to map record indexed data to station indexed data for the associated item. An attempt is made to add up values over a period in time equal to the accum_period, assuming each individual value represents the state over a period in time equal to the valid_period. The number of record indexed values used in the summation is essentially the accum_period divided by the valid_period. If that number is greater than one, then this results in values being added up over some period of time. If there is only one valid_period within the accum_period, then this routine can be used to find a non-undefined value amongst several record indexed values irrespective of what the default record to station mapping may be.

valid_period
The `valid_period' keyword defaults to the value for the `default_period' keyword, which defaults to the value for the `time_step'. It is only meaningful to specify a value for the valid_period keyword when a value has also been supplied for the `accum_period' keyword for the purpose performing an accumulation. For individual records closer in time than this, only one will be considered for accumulation.

constant
A constant item contains only one additional entry beside its item_id, with a keyword of `constant'. If the first argument is a single equals sign followed by another argument that is a numeric value, then the constant will be numeric. Otherwise the first argument will be used to define a string constant. Spaces are allowed in string constants, using the standard tilde as place holder.

Raw data keywords.

netcdf_id
An item is identified as containing raw data primarily by the presence of a `netcdf_id' keyword. This keyword is followed by the name of the netcdf variable that it reads data from.

record
If the argument of the record keyword is `true', then that item will contain values corresponding to records in the netcdf file. Otherwise, it will contain data for each station. There are two main reasons to specify that a data value is record indexed. First, the keywords `remove_by' and `indexing_table' perform functions that require a record indexed item. Second, a user may want to perform some calculations on a variable before doing the record to station conversion for display.

type
The `type' keyword is used to specify the base data type of a raw data variable. The possible arguments for the keyword type are `string', `float', `int', and `short'. The reason the user supplies this information rather than it being obtained from the netCDF file is so that the validity of function relationships can be determined when the design file is parsed.

dimension
The possible arguments for the `dimension' keyword are `scalar' and `list'. If scalar, there is one value for each record, meaning that variable should be dimensioned in the netcdf file as (recnum), or (recnum,strlen) in the case of a string. If list, then there are multiple values for each record, meaning that variables should be dimemsioned (recnum,listlen), or (recnum,listlen,strlen) in the case of a string. Here recnum must refer to the UNLIMITED dimension, listlen and strlen to any valid dimension appropriate for the variable.

subset
All raw data read from a netcdf file must have the UNLIMITED dimension as its first dimension. Normally beyond that, there can only be one more dimension on the variable (two if string). The `subset' keyword allows one to read data with more dimensions by fixing some dimensions and letting others float. For any item, the subset keyword must have an argument for each dimension beyond the UNLIMITED dimension. An argument of -1 allows a dimension to float, and an argument zero or greater fixes the dimension at that index. Considering numeric variables, if all dimensions are fixed by a subset keyword, then the item becomes scalar, and if all but one dimension is fixed it becomes list. For string data, one floating dimension means scalar, two means list.

time_offset
Normally, records for an item come from a time period centered at the time of the graphic being created. Using the `time_offset' keyword, it is possible to specify an arbitrary offset for the center of that time period for any item. For example, a time_offset argument of -86400 would allow one to read data one day previous to the time of graphic.

min_valid
Each base data type understood by design files also has a well defined null value. The `min_valid' keyword has no meaning for strings. For numeric types, any value read that is less than the argument to min_valid will be automatically converted to the null value for the base type.

max_valid
Each base data type understood by design files also has a well defined null value. The `max_valid' keyword has no meaning for strings. For numeric types, any value read that is greater than the argument to max_valid will be automatically converted to the null value for the base type.

Function keywords.

function
An item is identified as containing a function output primarily by the presence of a `function' keyword. This keyword is followed by the function name (see appendix II for a complete list of functions.)

inputs
Every item that has a `function' keyword must also have an `inputs' keyword. The arguments to this keyword is a list of item ids that represent the data that is input to the function. Depending on the function being used, order is usually very important amongst these arguments.

function_table
Some functions may need a lookup table to do their work and others may optionally be able to apply one. For any of these functions, the argument to the `function_table' keyword is the name of that table.

parameters
The arguments to the `parameters' keyword are one or more strings, the interpretation of which is dependent on which function is being used. A function is set up to take parameters when there is a desire to change the behavior of a function in a way that is not suitable to do with arguments, but does not result in a behavior that is fundementally different enough to warrant an entirely different function. Parameters are never required to make a function work, the code will always provide meaningful defaults.

Display keywords.

placement
An item is identified as having a display method primarily by the presence of the keyword `placement'. For some methods, the placement is predetermined by the method, and at other times the user may decide to supply specific coordinates. In these cases, the argument to the keyword placement is just `free'. Otherwise it is one of nine predefined placement positions. These are upper_left, top, upper_right, left, center, right, lower_left, bottom, and lower_right.

method
The argument to the `method' keyword determines the manner in which the data in the item is to be displayed. Most methods require only one input, although some require multiple inputs. When this is the case, a special function called `gather' is used to hand multiple items to a single display method. The reader is directed to appendix III for a complete list of display methods.

location
The arguments to the `location' keyword are a pair of coordinates that are in units of numbers of characters offset from the center of the station model, right and up being positive. It is meaningful to use non-integral numbers for these coordinates. The value of the `placement' keyword must be `free' for these coordinates to be used.

format
Some display methods require a C format specification. The argument to the `format' keyword is where this is supplied. When formating string values, the core of the format must be a %s type. When formating numeric values, the core of the format can be %d, %f or %g type. The default format is simply `%d'.

trim_count
The argument to the `trim_count' method is the number of characters to strip off the front of a formatted string before displaying it. It defaults to zero, which of course means do not trim any characters.

multiplier
The `multiplier' keyword works hand in hand with the `offset' keyword to allow units conversions to be performed on numeric values before displaying them. Numeric values are multiplied by the multiplier keyword argument and the added to by the offset keyword argument before being formatted for display. The default value for the multiplier is one. For units conversion with vector display, only the multiplier keyword is applicable.

offset
The `offset' keyword works hand in hand with the `multiplier' keyword to allow units conversions to be performed on numeric values before displaying them. Numeric values are multiplied by the multiplier keyword argument and the added to by the offset keyword argument before being formatted for display. The default value for the offset is zero. The offset keyword is not applicable for units conversion perse when displaying vector data.

min_trans
The `min_trans' keyword allows one to supply a lower bound to a final sanity check on a value to display after it has gone through the units conversion check. The value for min_trans is not applicable for vector display.

max_trans
The `max_trans' keyword allows one to supply an upper bound to a final sanity check on a value to display after it has gone through the units conversion check. The value for max_trans is applied to the magnitude for vector display.

table_file
Some display methods make use of a lookup table. The argument to the `table_file' is how one supplies the name of the file containing the data for such a lookup table.

undef_string
By default, when a display method is given a null value to display or the value fails the final sanity check, nothing will be displayed. The argument to `undef_string' allows the user to supply a string that should be displayed in that case.

magnification
As the name suggests, the argument to the `magnification' keyword is a magnification factor for whatever is being plotted. The default is one and non-integral values are meaningful.

alt_char_set
The allowable arguments to the `alt_char_set' keyword are `ascii', `large_ascii', `weather', `special', `large_special'. This allows the user to specify which character set should be used to display text; the default is ascii. The reader is directed to characterSets.doc for more information.

Appendix II.
Functions.

The data type of a function item is determined by the specific function used and the inputs. With some execptions, most functions will take any base data type as input. There are some broadly applicable rules about compatibility among inputs when there are multiple inputs to a function. One can never mix record indexed and station indexed items, and furthmore one can never mix record indexed items that have different values for the time_offset. With rare exceptions, one cannot mix list inputs that have different list sizes. However, one is usually free to mix scalar, list, and constant items as inputs, with the exception that it is usually not meaningful to have all inputs be constant. When mixing input items of different dimensionality, the result has the largest dimensionality. Inputs to functions can be raw data or other function results.

Here we will divide functions into six broad categories; special, conversion, mathematical, meteorological, logical, and list handling.

Special functions.

gather
The `gather' function exists solely for presenting a list of items to a display method. The most common use is to present multiple items to a display method that needs more than one item, such as displaying vectors. Another use is to pass a single item to an additional display method when the item as defined already has a display method. The gather function cannot take constant or record indexed inputs, and only rarely takes list inputs. The specifics of what is allowed is dependent on the particular display method used (see appendix III).

rec_to_sta
The `rec_to_sta' function is used to convert one record indexed item to a station indexed item. It is the only function that can generate a result with an indexing type different from its inputs. If `accum_period' keyword is not present, it will be by whatever default record to station mapping has been established for the time_offset of the item. If an `accum_period' keyword is present, then an accumulation will be performed, depending on the argument values for accum_period and valid_period.

sta_id
The `sta_id' function takes no arguments. The output is a scalar station indexed item of string base type, which contains the id of each station as obtained from static metadata. The fact that this function exists does not prevent one from reading the station id from a netcdf variable.

state_check
The `state_check' function is a very specialized function. It takes at least three inputs. The last input is a mask for bits to check and must be a numeric constant. The second to last input must be a string constant that is a list of characters corresponding to each possible state. The number of possible states is determined by the number of additional inputs before the last two constant inputs. For example, if there were five total inputs, there would be three inputs besides the last two constant inputs, and there would be eight possible states (two to the third power). Whether each input has any of the bits on in the mask is determined, and this yes/no answer is converted to a digit in a binary number, starting with the least significant bit, until all the leading non-constant inputs have been checked. This number is then used to index into the string constant to get a single character, which is the function result.

Conversion functions.

Conversion functions take one non-constant input and convert it to an item with a specified base data type, but with the same indexing and dimensions. By default, the conversion will be done by casting among numeric types, and through standard C format conversion between string and numeric types. The conversion can also be done with a lookup table, assuming compatibility between the input major type, requested output major type, and the flavor of the lookup table. If a value is null in the input, it will become null in the output unless a lookup table has been designed to provide a non-null translation for a null input.

string
The `string' function will convert any arbitrary input to a string item. This function is the only conversion function that can take parameters. The first parameter is the length of the strings in the output item and the second item is the format to use for data conversion. If not provided, meaningful defaults will be provided for the parameters. The parameters are meaningless if a usable lookup table is provided.

float
The `float' function will convert any arbitrary input to a float item.

int
The `int' function will convert any arbitrary input to a int item.

short
The `short' function will convert any arbitrary input to a short item.

Mathematical functions.

Mathematical functions will operate on any numeric type as input, but not on strings. When base data types are mixed, the output type will be whichever of the input types has the greatest range of values. For a given station/record and list index (if applicable), if any of the inputs are null, then the result will be null, with one exception.

diff
The `diff' function takes two inputs, and the result is the first input minus the second input.

div
The `div' function takes two inputs, and the result is the first input divided the second input.

add
The `add' function takes two or more inputs, and the result is the sum of all the inputs.

accum
The `accum' function takes two or more inputs, and the result is the sum of all the non-null inputs. This differs from the `add' function, where the presence of any null input will result in a null output. However, if all inputs are null, the result will still be null.

mult
The `mult' function takes two or more inputs, and the result is the product of all the inputs.

lintrans
The `lintrans' function takes two or more inputs. If I1, I2, etc., are the input and R is the result, then R=I1*I2+I3*I4... As in standard mathematical precedence, all the multiplications take place before the addtions. It is perfectly OK to have an odd number of inputs, in which case the last is simply added to the result without being multiplied by anything.

Meteorological functions.

Meteorological functions will only operate on float or numeric constant item types, and any null inputs will result in a null output.

ucomp
The `ucomp' function takes two inputs, a magnitude and a direction, and outputs the u component of that vector.

vcomp
The `vcomp' function takes two inputs, a magnitude and a direction, and outputs the v component of that vector.

heat_index
The `heat_index' function takes two inputs, temperature and dewpoint in degrees K and returns the Heat Index in degrees K.

wind_chill
The `heat_index' function takes two inputs, temperature in degrees K and wind speed in meters per second, and returns the Wind Chill temperature in degrees K.

dewpoint
The `dewpoint' function takes two inputs, temperature in degrees K and relative humidity in percent, and returns the dewpoint in degrees K.

temperature
The `temperature' function takes two inputs, potential temperature in degrees K and pressure in millibars, and returns the temperature in degrees K.

theta
The `theta' function takes two inputs, temperature in degrees K and pressure in millibars, and returns the potential temperature in degrees K.

alt2press
The `alt2press' function takes two inputs, altimeter setting in millibars and elevation in meters, and returns the surface station pressure in millibars.

height_of
The `height_of' function takes one input, and that input must be a numeric constant which is a pressure in millibars. The output is a scalar station indexed item of float base type, which contains the corresponding height of that pressure for each station, interpolated from gridded data. The user has the option of supplying a list of gridded data source names to try to use with a `paramters' keyword; if not supplied a default list will be tried. If gridded data access fails, the result will be based on a standard atmoshpere.

elev
The `elev' function takes no arguments. The output is a scalar station indexed item of float base type, which contains the elevation of each station as obtained from static metadata. The fact that this function exists does not prevent one from reading the elevation from a netcdf variable if such a variable exists in the data set.

Logical functions.

Logical functions have the same output base type as the first argument, but the maximum dimensionality of all the arguments. With a few exceptions, they will take any mix of base types as input. In general, logical functions test the truth of some relationship among the inputs, and if it is true the first non-null value among the inputs is placed in the output; if false a null value ends up in the output. Where the first non-null value is string and the output is numeric, the value of 0 will be placed in the output. Where the first non-null value is numeric and the output is string, the string "0" will be placed in the output.

or
The `or' function takes two or more inputs. The first non-null value among the inputs is placed in the output. The or function is different from all other functions in that it will work even if some of its inputs are not definable because of mismatches between assumed and actual netcdf variable names.

and
The `and' function takes two or more inputs. If all inputs are non-null, then the value from the first input will be placed in the output.

nor
The `nor' function takes two inputs. If both inputs are null, then a non-null (0 for numeric, "0" for string) will be placed in the output.

xor
The `xor' function takes two inputs. If exactly one input is null and one input is non-null, then the value of the non-null input will be placed in the output.

not
The `not' function takes one input. If the input is null, then a non-null (0 for numeric, "0" for string) will be placed in the output.

==
The `==' function takes two inputs, for which it is not allowed to mix string and numeric types. If the two inputs test as equivalent, then the first input is copied to the output.

!=
The `!=' function takes two inputs, for which it is not allowed to mix string and numeric types. If the two inputs test as not equivalent, then the first input is copied to the output.

>
The `>' function takes two inputs, for which it is not allowed to mix string and numeric types. If the first input tests as greater than the second input, then the first input is copied to the output.

<
The `<' function takes two inputs, for which it is not allowed to mix string and numeric types. If the first input tests as less than the second input, then the first input is copied to the output.

>=
The `>=' function takes two inputs, for which it is not allowed to mix string and numeric types. If the first input tests as greater than or equal to the second input, then the first input is copied to the output.

<=
The `<=' function takes two inputs, for which it is not allowed to mix string and numeric types. If the first input tests as less than or equal to the second input, then the first input is copied to the output.

min
The `min' function takes two or more inputs for which it is not allowed to mix string and numeric types. If any values are non-null, then the first non-null value that tests as being less than or equal to all the others is copied to the output.

max
The `max' function takes two or more inputs for which it is not allowed to mix string and numeric types. If any values are non-null, then the first non-null value that tests as being greater than or equal to all the others is copied to the output.

bit_none
The `bit_none' function take two inputs, which must have a short or int base type or be a numeric constant. If there are no bits that are turned on in both inputs, then a value of zero will be placed in the output, otherwise a null output will result.

bit_any
The `bit_any' function take two inputs, which must have a short or int base type or be a numeric constant. If there are any bits that are turned on in both inputs, then a value of zero will be placed in the output, otherwise a null output will result.

bit_all
The `bit_any' function take two inputs, which must have a short or int base type or be a numeric constant. If every bit that is on in the first input is also on in the second output, then a value of zero will be placed in the output, otherwise a null output will result.

none_there
The `none_there' function take two or more inputs which must be string. If none of the remaining inputs appears in the first input as a substring, then the first input will be placed in the output, otherwise a null output will result.

any_there
The `any_there' function take two or more inputs which must be string. If any of the remaining inputs appears in the first input as a substring, then the first input will be placed in the output, otherwise a null output will result.

all_there
The `all_there' function take two or more inputs which must be string. If all of the remaining inputs appears in the first input as a substring, then the first input will be placed in the output, otherwise a null output will result.

List handling functions.

down_interp
The `down_interp' function is used to perform vertical interpolation in the case where the value of the vertical coordinate being used increases downward. This function takes three inputs. All inputs must have a base type of float, the first two must be list, and the last one may be a scalar float or numeric constant. The first input is the vertical coordinate list, the second is the list of values to interpolate to a level, and the last input is the specific vertical coordinate value to interpolate to. The resulting item is a scalar float, and the interpolation is linear versus the log of the vertical coordinate. By default, it is possible to interpolate data from within a gap 2 levels wide (one missing level). The size of this gap can be specified optionally as an input to a `parameters' keyword.

up_interp
The `up_interp' function is used to perform vertical interpolation in the case where the value of the vertical coordinate being used increases upward. This function takes three inputs. All inputs must have a base type of float, the first two must be list, and the last one may be a scalar float or numeric constant. The first input is the vertical coordinate list, the second is the list of values to interpolate to a level, and the last input is the specific vertical coordinate value to interpolate to. The resulting item is a scalar float, and the interpolation is linear versus the vertical coordinate. By default, it is possible to interpolate data from within a gap 2 levels wide (one missing level). The size of this gap can be specified optionally as an input to a `parameters' keyword.

down_sample
The `down_sample' function is used to perform vertical sampling in the case where the value of the vertical coordinate being used increases downward. Sampling means directly using the value from the nearest level rather than interpolating. This function takes three inputs. All inputs must have a base type of float, the first two must be list, and the last one may be a scalar float or numeric constant. The first input is the vertical coordinate list, the second is the list of values to sample, and the last input is the specific vertical coordinate value that identifies which level to use. The resulting item is a scalar float. By default, it is possible to sample data from within a gap 2 levels wide (one missing level). The size of this gap can be specified optionally as an input to a `parameters' keyword.

up_sample
The `up_sample' function is used to perform vertical sampling in the case where the value of the vertical coordinate being used increases upward. Sampling means directly using the value from the nearest level rather than interpolating. This function takes three inputs. All inputs must have a base type of float, the first two must be list, and the last one may be a scalar float or numeric constant. The first input is the vertical coordinate list, the second is the list of values to sample, and the last input is the specific vertical coordinate value that identifies which level to use. The resulting item is a scalar float. By default, it is possible to sample data from within a gap 2 levels wide (one missing level). The size of this gap can be specified optionally as an input to a `parameters' keyword.

find_first
The `find_first' function takes one input, which must have list dimensionality, but can have any basic type. The result of this function is a scalar int item, which contains the list index of the first non-null item in the list.

find_last
The `find_last' function takes one input, which must have list dimensionality, but can have any basic type. The result of this function is a scalar int item, which contains the list index of the last non-null item in the list.

find_max
The `find_max' function takes one input, which must have list dimensionality, but can have any basic type. The result of this function is a scalar int item, which contains the list index of the first non-null item in the list that tests greater than or equal to all the others.

find_min
The `find_min' function takes one input, which must have list dimensionality, but can have any basic type. The result of this function is a scalar int item, which contains the list index of the first non-null item in the list that tests less than or equal to all the others.

index
The `index' function must have two inputs, the first of which must have list dimensionality, but can have any basic type. The second item must be either a numeric constant or a scalar int, most often from one of the functions `find_first', `find_last', `find_max', or `find_min'. The value in the first input whose list index corresponds to the second input is placed in the output item, which has scalar dimensionality and the same base type as the input.

select
The `select' function requires a lookup table to perform its primary function, supplied with the `function_table' keyword. This function must have one input that has list dimensionality but can have any base type. The table used must have numeric output, and its input must be compatible with the major type of the intput item. The table is used to arbitrarily rank each item in the list. Whichever item in the list has the lowest rank number is selected and placed in the output item, which has scalar dimensionality and the same base type as the input.

catenate
The `catenate' function can take one or more string inputs of any dimensionality, and is one of only two functions that can take inputs items with list dimensionality and different list lengths. Each individual string, whether it is part of a list or scalar, it catenated to the eventual result string. The output item is a scalar string type. The default delimiter between each string is an empty string. One can optionally supply a different delimiter as an argument to a `parameters' keyword.

group
The `group' function can take two or more inputs that all have the same base type but any dimensionality, and is one of only two functions that can take inputs items with list dimensionality and different list lengths. Each individual input value, whether it is part of a list or scalar, it added to the eventual resulting output list. The output item has list dimensionality and the same base type as the inputs.

Appendix III.
Display Methods.

formatted
The `formatted' method operates on single item as input. The item should have scalar dimensionality, but can be of any base type. The main function of this display method is to plot a data value using a C format specifier provided in the `format' keyword. The positioning of the plotted element responds to the nine standard position options available as an argument to the `placement' keyword, or to the coordinates supplied as arguments to `location' if the placement is `free'. If numeric, the value undergoes a units conversion based on the arguments to `multiplier' and `offset', and will only be plottable if the converted value is within the bounds set by the arguments to `min_trans' and `max_trans'. If the value is null or otherwise deemed unplottable, the string supplied as an argument to `undef_string' can be plotted instead of nothing. Some other character set than ascii can be used to plot the resulting string based on the argument to `alt_char_set'. It is possible to strip a specified number of characters off of the front of the formatted string before plotting by supplying an argument to the keyword `trim_count'. The size of the item plotted can be made different from the default size by supplying an argument to the keyword `magnification'. If the item being displayed is numeric, the format can be determined by the output of a string to number lookup table provided the file name of the table is supplied as an argument to the `table_file' keyword.

lookup
The `formatted' method operates on single item as input. The item should have scalar dimensionality, but can be of any base type. The main function of this display method is to plot the result of a simple lookup table translation. The lookup table, the file for which is supplied as an argument to the `table_file' keyword, needs to have string output and input compatible with the major type of the item being plotted. The positioning of the plotted element responds to the nine standard position options available as an argument to the `placement' keyword, or to the coordinates supplied as arguments to `location' if the placement is `free'. Some other character set than ascii can be used to plot the resulting string based on the argument to `alt_char_set'. The size of the item plotted can be made different from the default size by supplying an argument to the keyword `magnification'.

translation
The `formatted' method operates on single item as input, which must be of string scalar type. The output string is generated by performing a recursive translation using a string to string lookup table, supplied as an argument to the `table_file' keyword. Any text that is successfully translated will be displayed by whatever character set is specified as an argument to `alt_char_set'. Untranslated text will be shown in plain ascii. The positioning of the plotted element responds to the nine standard position options available as an argument to the `placement' keyword, or to the coordinates supplied as arguments to `location' if the placement is `free'. The size of the item plotted can be made different from the default size by supplying an argument to the keyword `magnification'.

trend
The `trend' method is a specialized display method used to display the standard coding for the three hour pressure tendency. The inputs must be the change amount, as a float scalar type, and the pressure change character, as a short scalar type. Any desired units conversion for the change amount must be supplied as an argument to the `multiplier' keyword, but the `offset' keyword is ignored. The formating of the change amount can also be affected by the argument to the `format' keyword. The positioning of the plotted element responds to the nine standard position options available as an argument to the `placement' keyword, or to the coordinates supplied as arguments to `location' if the placement is `free'. The size of the item plotted can be made different from the default size by supplying an argument to the keyword `magnification'.

barb
The `barb' method is used to display vector data as wind barbs. The barb method can have from two to four inputs, which must all be scalar float inputs. The inputs represent wind speed, wind direction, gust speed, and gust direction, respectively. When gust information is available, it is plotted as an additional arrow with the gust speed labeled in ascii. The position is predetermined by the method, so one needs to provide the argument `free' to the `placement' keyword. If supplied, the argument to the `max_trans' keyword is the maximum wind speed considered valid. The argument to the `multiplier' keyword is used for units conversion on the speed, but there is no offset applied. The argument to the `min_trans' keyword is the smallest gust speed that will be plotted as such, and the argument to the `offset' keyword is the minimum vector difference that must exist between the wind and the gust before a gust will be plotted. The argument to the `multiplier' keyword is overloaded; its magnitude is the size relative to the default of the barb to be plotted. If it is negative, then calm circles will not be plotted. If the barb as a whole is deemed unplottable, normally nothing will be plotted, but the `undef_string' keyword can be used to specify some ascii string to plot in that case.

arrow
The `arrow' method is used to display vector data as arrows. The arrow method must have two scalar float inputs. The inputs represent wind speed and wind direction, respectively. The length of the arrow is fixed, and the speed is plotted in ascii at the end of the arrow. The position is predetermined by the method, so one needs to provide the argument `free' to the `placement' keyword. If supplied, the argument to the `max_trans' keyword is the maximum wind speed considered valid. The argument to the `multiplier' keyword is used for units conversion on the speed, but there is no offset applied. The argument to the `multiplier' keyword is the size relative to the default of the arrow to be plotted. If the arrow as a whole is deemed unplottable, normally nothing will be plotted, but the `undef_string' keyword can be used to specify some ascii string to plot in that case.

barbuv
The `barbuv' method is used to display vector data as wind barbs. The barb method can have from two to four inputs, which must all be float inputs. If there are two inputs they must have scalar dimensionality, otherwise they must all have list dimensionality. The inputs represent u component, v component, height, and qc parameters, respectively. When just two inputs are present, just a single barb is plotted. If there are three inputs present, then a stack of wind barbs is plotted. If there are four inputs present, then wind barbs in the stack are colored according to the speed and the qc information. The position is predetermined by the method, so one needs to provide the argument `free' to the `placement' keyword. If supplied, the argument to the `max_trans' keyword is the maximum wind speed considered valid. The argument to the `multiplier' keyword is used for units conversion on the speed, but there is no offset applied. The argument to the `multiplier' keyword is overloaded; its magnitude is the size relative to the default of the barb to be plotted. If it is negative, then calm circles will not be plotted. If the barb as a whole is deemed unplottable, normally nothing will be plotted, but the `undef_string' keyword can be used to specify some ascii string to plot in that case.

arrowuv
The `arrowuv' method is used to display vector data as arrows. The arrow method can have from two to four inputs, which must all be float inputs. If there are two inputs they must have scalar dimensionality, otherwise they must all have list dimensionality. The inputs represent u component, v component, height, and qc parameters, respectively. The length of the arrow is fixed, and the speed is plotted in ascii at the end of the arrow. When just two inputs are present, just a single arrow is plotted. If there are three inputs present, then a stack of wind arrows is plotted. If there are four inputs present, then wind arrows in the stack are colored according to the speed and the qc information. The position is predetermined by the method, so one needs to provide the argument `free' to the `placement' keyword. If supplied, the argument to the `max_trans' keyword is the maximum wind speed considered valid. The argument to the `multiplier' keyword is used for units conversion on the speed, but there is no offset applied. The argument to the `multiplier' keyword is overloaded; its magnitude is the size relative to the default of the arrow to be plotted. If it is negative, then calm circles will not be plotted. If the arrow as a whole is deemed unplottable, normally nothing will be plotted, but the `undef_string' keyword can be used to specify some ascii string to plot in that case.

Author: Jim Ramer
Last update: 9 Feb 2000