Super resolution radar data.

Updated May 31 2007

http://www-sdd.fsl.noaa.gov/~ramer/noaa/superRes/superRes.html
During the OB8.2 time frame new higher resolution base data radar products will come on line; the most notable feature of these products is that they will have half degree azimuth resolution. These products also introduce for the first time 250m gate spacing for reflectivity, 8 bit data for spectrum width, and 300km range for velocity and spectrum width.

This writeup will concentrate on, but not be strictly limited to, the new super-res products; there will be some other miscellaneous OB8.2 radar changes discussed as well. We will start by discussing the changes to the radar menus for OB8.2, and follow that by discussing the updates to the displays. Next, the changes made to the radar ingest to support the addition of super-res products will be described, followed by material that will support a code review of those changes. We will conclude with a summary of all files changed in OB8.2 meant to support updates to the radar subsystem.

We will not discuss here any of changes to the radar applications made in OB8.2; the link that follows contains material about these changes:

Application updates.

Changes to Radar Menus.

The current set of default selectors for reflectivity and velocity on the main menu for each radar actually point to multiple products for a given tilt and base data type. If 8 bit data (the current one degree azimuthm variety) is available, then that is displayed, if not an attempt will be made to display 4 bit data, and if that is not available then an attempt will be made to display 3 bit data.

The means by which the new super resolution products will be made available to the user will be to put them first in the list of data sets associated with the default selectors for reflectivity, velocity, 8 bit SRM, and spectrum width. The super resolution products will also become the prefered data to use for any combos or four panels containing such data.

This arrangement of the default main menus means that when the super resolution data is present for a given tilt and scan, the current one degree azimuth resolution 8 bit base data products are not viewable. To deal with this, we are adding a set of of simple non-cascading selectors that only load the current standard resolution 8 bit base data products. Here we show the menu path to the standard resolution selectors for 8 bit reflectivity:

Note that these are on the 4 Bit Prods pull-right, which has been renamed to 4Bit/Legacy Prods. Also note the new section called Legacy Graphics has been added to the 4Bit/Legacy Prods pull-right, and that the legacy SWA products have been further deemphasized by removing the SWA four panels. There are no combos or four-panels constructed specfically from the standard resolution 8 bit data, but one could always construct these manually and put them in a procedure if this was desired. Other modifications to the main menu are the removal of the CZE product and the consolidation of the clutter products into a Data Quality selector. The screen shot that follows show the Precip, Derived, Graphics and Data Quality menus torn away:

The hybrid scan products have been moved from the Derived menu to the Precip menu, the Layer Average Reflectivity products have been removed from the Derived, the MRU and M products have been moved from the Graphics menu over to the Legacy Graphics area (shown previously), the NEXRAD Unit Status has been moved to the top of the Graphics menu with a separator, and the CFC has been moved from the Derived menu to the new Data Quality menu. Note that the CFC product now has separate selectors for each segment, which for the CFC sort of corresponds to a layer.

New Displays, Display Updates.

Since this is our first foray into 8 bit spectrum width data, new color tables were needed. Last month we received for the first time some sample data based on a real data set. Here is a screen shot of a super-res spectrum width product, showing the new GSD color table for spectrum width:

Note that the color scheme is very similar to the existing GSD color table for spectrum width; the range folded color has been changed to the same color as range folding on the GSD velocity color tables, ramping has been added to each existing color bin, and one additional ramped color has been added to account for the greater dynamic range of the super res product. If spectrum width is chosen and only the existing 3 bit version is available, it will display exactly as previously other than the color change for range folded data. The hot pink below zero only get used for the case where the 8 bit spectrum width value is exactly 0; the 0 to 4 range for 3 bit data still gets the lightest gray as before. Very similar changes were made to the OSF spectrum width color table.

Also note the addition to the resolution information under the color bar; the string 0.5dAz indicates this is a super res product. Current prototypes have no indication of azimuth resolution for standard azimuth resolution products.

The following are screen shots for the OB8.2 versions of the Nexrad Unit Status Message product and the CFC:

Note that in the the Nexrad Unit Status Message there is an 'S' next to the tilts that have super resolution data available. In the CFC, the notation in the upper left of the surveilance versus doppler mode is gone, and the segment number appears in the regular legend as well.

Other than resolution of the data, the displays of super resolution reflectivity and velocity will not look very different than the existing displays, but here are some screen shots for completness.

Changes to Ingest.

The basic functioning of the AWIPS radar ingest has always been as follows:


- Decode the radar id, Nexrad message code, and any applicable tilt angle
  (or layer) from the RPG message.
- Look up the entry from radarInfo.txt based on the message code, and extract
  the product type (e.g. base reflectivity), resolution (gate spacing), 
  and number of data levels (bit depth) from the radar info.
- Match the radar id, product type, resolution, data levels and tilt angle
  to entries in the data keys to identify the data key to store the
  data on behalf of.
- Copy the RPG message file to the directory specified in the data key.

With the introduction of the super res velocity product, we now have two kinds of products that would be identified as 8 bit base velocity with gate spacing of 0.25km. From a pure data modeling approach, what distinguishes these products is the azimuthal resolution. Thus, one way this could have been dealt with was to add azimuthal resolution to our ingest data model. This approach has two drawbacks; it adds more complexity to the ingest data model, and it locks us into yet another rigid data model that could potentially have to change again in the future.

What we have done to solve this is to allow the option of identifying an ingest data set with the current data model, or to identify it solely on the basis of the Nexrad message code and tilt angle. Here are two example generic data key entries; each is for the 0.5 degree tilt angle. 41805 is the key for the existing 8 bit base velocity with standard azimuth resolution; this entry is actually unchanged. 21805 is the key for the new super res base velocity:


41805|@@@@/@@@@Rad230km|1|0.5|0.25|256|0|radar/@@@@/V/elev0_5/res0_25/level256| | |
21805|@@@@/@@@@Rad30Okm|1|0.5|0.25|256|0,154|radar/@@@@/V/elev0_5/res0_25/az0_5/level256| | |

Note that the only differences are the key itself, the directory path, and that the super res entry (21805) has an extra ,154 in the seventh vertical bar delimited field. 154 is the Nexrad message code for super res velocity, and the presence of this field tells the software to perform the ingest data key matching based on the message code and tilt angle. The existing entry for key 41805, not having a Nexrad message code present, will use the same ingest data model as it always has.

No existing directory names will change, but we will add the az0_5/ to the directory path for any data set that is half degree azimuth resolution. The az0_5/ is being added one level up from the ultimate final storage directory so that none of the storage directories get a mix of directory and data files.

Review of Code Changes.

At the end of this document is a list of all files changed against the OB8.2 radar DCS (DCS3421). This code review will only cover changes made in the radar ingest code. Furthermore, only changes that allow the determination of the radar storage directory directly from the RPG message code will be discussed; we will not discuss ingest changes that support treating the CFC as a layer product. What follows is a list of the files which will be discussed here:


D-2D/src/localization/scripts/genRadarDataKeys.ksh
D-2D/src/dataMgmt/DM_DataAccessInfo.H
D-2D/src/dataMgmt/DM_DataAccessInfo.C
D-2D/src/dataMgmt/dmDataAccessFns.C
D-2D/src/dataMgmt/dmDataAccessFns.H
D-2D/src/dataMgmt/RadarAccessKey.C
D-2D/src/dataMgmt/RadarAccessKey.H

In order to be able to use the RPG message code directly for directory assignment, it is useful to be able to place the message code directly into the data access key entry. In order that RPG message codes placed in radarDataKeys.template would get propagated to the specific data key entries, genRadarDataKeys.ksh had to be changed. Also, clients of the data access key info need a way to obtain the message code; for this we changed the DM_DataAccessInfo class.Here are the changed areas in these files:

genRadarDataKeys.ksh:

23:  cut '-d|' -f1-6 temp2 > temp1
24:  cut '-d|' -f7-11 temp2 | sed 's/^ *0 *//g' > temp3
25:  $pasteUtil temp1 "-|$index" temp3 >> radarDataKeys.txt

DM_DataAccessInfo.H:
71:        MessageCode messageCode();

DM_DataAccessInfo.C:
428:  MessageCode DM_DataAccessInfo::messageCode()
429:      {
430:      return (int)_integer6_1;
431:      }

The module RadarAccessKey is where clients can obtain radar data keys on behalf of the individual items in our radar data model. A new alternate lookup method was added that allows the RPG message code to be substituted for the resolution, bit depth, and AWIPS product type in the data model. For OB8.2, some logic was also changed to streamline the identification of which radars are TDWR radars, but that will not be discussed here. Here are the pertinent changed areas in these files:

RadarAccessKey.H:

50:        DataAccessKey dataKey(MessageCode & msgCode, RadarElevation& elev, 
51:          RadarSite& hashSite, RadarSite& keySite);

RadarAccessKey.C:
183:      if (info.messageCode()==0)
184:          {
185:          DM_GetRadarAccessKeyHashKey infoKey(info.getRadarProduct(), 
186:                                              info.getRadarElevation(),
187:                                              info.getRadarLevel(),
188:                                              info.getRadarResolution(),
189:                                              0);
190:          dataAccessKey &= 0x4000FFFF;
191:          _dict.add(infoKey, dataAccessKey);
192:          }
193:      else
194:          {
195:          DM_GetRadarAccessKeyHashKey infoKey(info.messageCode(), 
196:                                              info.getRadarElevation(),
197:                                              0);
198:          dataAccessKey &= 0x4000FFFF;
199:          _dict.add(infoKey, dataAccessKey);
200:          }
             :
             :
241:// Use as a shortcut to verify if a key is valid.
242:static StaticTextDict * dataKeyImpl = 0;
243:
244:// -- public ----------------------------------------------------------------
245:// RadarAccessKey::dataKey(RadarProduct& prod, RadarElevation& elev, 
246://   RadarLevel& level, RadarResolution& res, RadarSite& hashSite)
247://
248:// Returns a valid DataAccessKey for the specified product, or returns 0
249:// --------------------------------------------------------------------------
250:DataAccessKey RadarAccessKey::dataKey(
251:        MessageCode & msgCode, RadarElevation& elev, 
252:        RadarSite& hashSite, RadarSite& keySite)
253:    {
254:    DM_GetRadarAccessKeyHashKey hashKey(msgCode, elev, hashSite);
255:    DataAccessKey dataAccessKey = 0;
256:
257:    if (_dict.map(hashKey, dataAccessKey))
258:        {
259:        if (!dataKeyImpl)
260:            dataKeyImpl = DM_DataAccessInfo::allDataKeys();
261:        dataAccessKey |= (keySite<<16);
262:        if (!dataKeyImpl->keyValid(dataAccessKey))
263:            dataAccessKey = 0;
264:        }
265:
266:    return dataAccessKey;
267:    } 
268:
269:// -- public ----------------------------------------------------------------
270:// RadarAccessKey::dataKey(RadarProduct& prod, RadarElevation& elev, 
271://   RadarLevel& level, RadarResolution& res, RadarSite& hashSite)
272://
273:// Returns a valid DataAccessKey for the specified product, or returns 0
274:// --------------------------------------------------------------------------
275:DataAccessKey RadarAccessKey::dataKey(
276:        RadarProduct& prod, RadarElevation& elev, 
277:        RadarLevel& level, RadarResolution& res,
278:        RadarSite& hashSite, RadarSite& keySite)
279:    {
280:    DM_GetRadarAccessKeyHashKey hashKey(prod, elev, level, res, hashSite);
281:    DataAccessKey dataAccessKey = 0;
282:
283:    if (_dict.map(hashKey, dataAccessKey))
284:        {
285:        if (!dataKeyImpl)
286:            dataKeyImpl = DM_DataAccessInfo::allDataKeys();
287:        dataAccessKey |= (keySite<<16);
288:        if (!dataKeyImpl->keyValid(dataAccessKey))
289:            dataAccessKey = 0;
290:        }
291:
292:    return dataAccessKey;
293:    }

In the module dmDataAccessFns, the method dmGetRadarAccessKey was changed so that first a key lookup will be tried based on the new RPG message code model, and if that failed, it would revert to the old resolution, bit depth, product model. The parsing of the file tiltAngleGroups.txt was moved out of dmGetRadarAccessKey, both to make it simpler and to make the information from that table availble to outside clients; the outside client interface logic will not be covered here as it is not important to the radar ingest.

dmDataAccessFns.H:

93:DataAccessKey dmGetRadarAccessKey(RadarProductName, MessageCode, 
94:                                  RadarElevation, RadarLevel, RadarResolution, 
95:                                  RadarSiteName);
             :
             :
151:        DM_GetRadarAccessKeyHashKey(MessageCode msgCode, RadarElevation elev,
152:                                    RadarSite site)
153:            : _product(-(RadarProduct)msgCode), _elev(elev), _level(0), _res(0),
154:              _site(site) {}

dmDataAccessFns.C:
619:// Shared global data structures that have info parsed from tiltAngleGroups.txt
620:static float * angleLookup = 0;
621:static long * spaceOffsetLookup = 0;
622:static float * tiltOfSpaceOffset = 0;
623:static int nSpaceOffsets = 0;
624:
625:// -- static -----------------------------------------------------------------
626:// void dmParseTiltAngleGroups()
627:// 
628:// Parse information out of tiltAngleGroups.txt.  Very hardwired to deal with
629:// a range of tilts from -0.9 to 90.0 in 0.1 degree steps.
630:// ---------------------------------------------------------------------------
631:// ---------------------------------------------------------------------------
632:static void dmParseTiltAngleGroups()
633:    {
634:    if (angleLookup) return;
635:
636:    // Set up a default state.
637:    int a, t;
638:    float p, v1, v2;
639:    angleLookup = new float[910];
640:    angleLookup += 9;
641:    // Table set up to go from -0.9 to 90.0
642:    for (a=-9; a<=900; a++) angleLookup[a] = a*0.1;
643:    spaceOffsetLookup = new long[910];
644:    spaceOffsetLookup += 9;
645:    for (a=-9; a<=900; a++) spaceOffsetLookup[a] = 0;
646:
647:    // See if we have tilt angle groups available.
648:    TextBuffer tbag("tiltAngleGroups.txt");
649:    for (t=0; t<tbag.buffer().length(); t++)
650:        {
651:        a = sscanf(tbag.buffer()[t].stringPtr(),"%f %f %f",&p,&v1,&v2);
652:        if (a!=3 || v1<-0.9 || v2>90 || p<v1 || p>v2) continue;
653:        a = v1<0 ? -(int)(0.5-10*v1) : (int)(0.5+10*v1);
654:        v2 = v2*10+0.5;
655:        // Because in other parts of the radar ingest system we use 
656:        // zero for the null tilt case, and angle of 0.0 must ALWAYS 
657:        // map to 0.0.
658:        if (a<=0 && v2>=0) p = 0.0;
659:        nSpaceOffsets++;
660:        for ( ; a<v2; a++)
661:            {
662:            angleLookup[a] = p;
663:            spaceOffsetLookup[a] = nSpaceOffsets;
664:            }
665:        }
666:    if (nSpaceOffsets==0)
667:        {
668:        logProblem << "No useable entries in tiltAngleGroups.txt" << std::endl;
669:        return;
670:        }
671:    if (nSpaceOffsets<10)
672:        logProblem << "Only " << nSpaceOffsets
673:                   << " entries in tiltAngleGroups.txt" << std::endl;
674:    tiltOfSpaceOffset = new float[nSpaceOffsets];
675:    tiltOfSpaceOffset--;
676:    for (a=-9; a<=900; a++)
677:        if (spaceOffsetLookup[a]!=0)
678:            tiltOfSpaceOffset[spaceOffsetLookup[a]] = angleLookup[a];
679:    }
680:
       :
       :
726:
727:// -- global -----------------------------------------------------------------
728:// dmGetRadarAccessKey()
729:// Given information identifying a radar product
730:// return the corresponding DataAccessKey (or zero if none exists).
731://
732:// ---------------------------------------------------------------------------
733:// ---------------------------------------------------------------------------
734:DataAccessKey dmGetRadarAccessKey(RadarProductName product,
735:                                  MessageCode msgCode,
736:                                  RadarElevation elevation,
737:                                  RadarLevel level,
738:                                  RadarResolution resolution,
739:                                  RadarSiteName site)
740:    {
741:    static bool initialized = false;
742:    static Dict dict;
743:    static RadarSite knatSite = -1;
744:
745:    knatSite = cvtRadarSite("knat");
746:    RadarAccessKey radarKey(knatSite);
747:
748:    // Initialize a dictionary, the first time through
749:    if (!initialized)
750:        {
751:        AbsTime start = AbsTime::current();
752:        dmParseTiltAngleGroups();
          :
          :
891:    DataAccessKey dKey = radarKey.dataKey(msgCode, elevation,
892:                                          hashSite, keySite);
893:    if (dKey==0)
894:        dKey = radarKey.dataKey(prod, elevation, level, resolution,
895:                                hashSite, keySite);
896:    return dKey;
897:    }

In the course of preparing the material for this review, an issue has surfaced. As it turns out, some of the code that was modified here was previously replicated in the file D-2D/src/dm/scan/dmGetRadarAccKey_SCAN.C and subsequently modified. The author of dmGetRadarAccKey_SCAN.C should be contacted so that it will be possible to discuss whether the ability to respond to RPG message code based data key entries will have to be added there.

List of files changed for OB8.2 radar work.


Add new RPG products, allow specification of azimuth resolution:
D-2D/src/dataMgmt/DM_RadarInfo.C
D-2D/src/dataMgmt/radarInfo.txt

New color tables for spectrum width:
D-2D/src/localization/scripts/genRadarDepictKeys.ksh
D-2D/src/dataMgmt/colorMaps.mark

Application updates:
D-2D/src/applications/radar/common/prod-mgmt.tcl
D-2D/src/applications/radar/otr/prods.conf
D-2D/src/applications/radar/rmr/prods.conf
D-2D/src/applications/radar/rps/prods.conf
D-2D/src/ipc/radar/sendOTR.cfc.sh
D-2D/src/ipc/radar/sendOtrs.C
D-2D/src/ipc/radar/sendOtrs.sh
D-2D/src/ipc/radar/ProductRequestEntry.C

Assignment of storage directory uses message code directly:
D-2D/src/localization/scripts/genRadarDataKeys.ksh
D-2D/src/dataMgmt/DM_DataAccessInfo.C
D-2D/src/dataMgmt/DM_DataAccessInfo.H
D-2D/src/dataMgmt/dmDataAccessFns.C
D-2D/src/dataMgmt/dmDataAccessFns.H
D-2D/src/dataMgmt/RadarAccessKey.C
D-2D/src/dataMgmt/RadarAccessKey.H

Treat CFC product as being layer based:
D-2D/src/dm/radar/RadarFileName.C
D-2D/src/dm/radar/RadarStorageController.C

Display changes for super-res.
D-2D/src/depict/ImageDepictable.C
D-2D/src/depict/RadarPVImageDepict.C
D-2D/src/depict/RadarPVImageDepict.H
D-2D/src/dm/radar/decodeRadar.h

Display change for CFC.
D-2D/src/dm/radar/prodDepParmsRoutines.c

Now have products with 300km coverage.
D-2D/src/dm/radar/Radar.H
D-2D/src/localization/scripts/makeRadarSups.csh

Show super-res tilts on Unit Status display
D-2D/src/dm/nexrad/graphics/Radar_fg.C
D-2D/src/dm/nexrad/nonprod/RadarGsmMsg.C
D-2D/src/dm/nexrad/nonprod/RadarGsmMsg.H

Add new data sets, displays, and menus.
D-2D/src/localization/nationalData/radarDataKeys.template
D-2D/src/localization/nationalData/radarDataMenus.template
D-2D/src/localization/nationalData/radarDepictKeys.template
D-2D/src/localization/nationalData/radarGenericImageStyle.txt
D-2D/src/localization/nationalData/radarImageStyleInfo.template
D-2D/src/localization/nationalData/radarMultiLoadInfo.template
D-2D/src/localization/nationalData/radarProductButtonInfo.template