MADIS Plugin Report

by Jim Fluke


  1. Used the MADIS web page to get a sample of XML formatted MADIS data. Selected the standard surface variables. In Firefox I had to select "View/Page Source" to see the data. Here is a small sample of the data.
  2. Installed both EDEX and ADE on my development workstation following Joanne's instructions. And insured that I could build and install using ant.
  3. Downloaded the TO8 version of the EDEX Plugin tool from Cliff Wong's 3/13/08 email message, and installed it in my <ADE>/eclipse/plugins directory.
  4. Ran the Plugin Tool from Eclipse by selecting "Window/Show View/Other.../Other/Plugin Creator".  Chose gov.noaa as the Organization Domain and madis as the Plugin Name. Did not select IPersistable or "Create Separator". Turned on the "Data URI" checkbox for all of the data fields. Chose AWIPSEdex/extensions as the output directory, and let the tool create the plugin-madis subdirectory. There was no indication that the plugin had been created other than the "Generate Plugin" button appearing depressed for a short time. In the "Package" tree, selected the "extensions" label under "AWIPSEdex" with the 3rd mouse button. This highlighted the label and brought up the 3rd button menu. Selecting "Refresh" caused "plugin-madis" to appear under "extensions".
  5. Implemented MadisDecoder.java, the helper class file MadisParser.java and package-info.java under "AWIPSEdex/extensions/plugin-madis/src/gov/noaa/edex/plugin/madis".
  6. In eclipse, added the plugin to the Build Path by highlighting the extensions/plugin-madis/src directory and selecting to add it to the build path through the 3rd mouse button menu. This causes the directory to be displayed directly under AWIPSEdex as a build path selector. Then saw an "Unexpected Package" error even though the package was correct. Fixed it by shutting down eclipse and starting it back up again.
  7. Added plugin-madis to build/deployments/deployment.properties.
  8. Tried to build, outside of eclipse using ant. Got jibx build errors about gov.noaa.edex.madis. Corrected them by changing extensions/plugin-madis/res/conf/plugin.xml and extensions/plugin-madis/res/binding.xml. Both had gov.noaa.edex.madis instead of gov.noaa.edex.plugin.madis.
  9. Retried the build. Got a Calendar serialize/deserialize error from jibx. Borrowed the Calendar serializer/deserializer attributes from the pirep plugin binding.xml for the madis binding.xml.
  10. Retried the build. It worked.
  11. Tried to run it after deploying it to my local EDEX install area. Mule crashes. Here is the error log.
  12. Remove edex/mule/lib/user/plugins/plugin-madis.jar and edex/opt/esb/conf/madis-ingest.xml from my EDEX working area. Mule runs just fine, but of course the madis plugin is not included.
  13. Created a tar file of my plugin from the projects/AWIPSEdex directory. This old file is no longer available. Use the new one from step 26.
  14. Changed to fully qualified variable type names madis.hbm.xml. Mule now starts up successfully.
  15. Copied a small madis data file (the sample above) to <EDEX>/edex/opt/data/sbn/madis. This gives an xml parsing error about: "...<columnType><UNKNOWN TYPE>...".
  16. Found the error in madis.db.xml and fixed it. No longer see the xml parsing error. Now see an error about not being able to create the MadisDecoder. Can't get its "properties."
  17. Added Name and Plugin properties elements to plugin.xml to make it look like the pirep plugin.xml. This fixed the previous error, but now there is new one complaining about "Could not create PluginDecoderProxy".
  18. Added logger.info() statements to PluginDecoderProxy.java to try to find the problem. The problem is in my madis java code.
  19. Added logger.info() statements to MadisDecoder.java and MadisParser.java. MadisParser get to the "elev" field and then stops - can't parse it?
  20. Added a try/catch block around the code in MadisParser.java that populates the records. Got a NumberFormatException from Integer.parseInt() for "elev". Ok, so "elev" does have a floating point format - it always ends in ".00".
  21. Changed the elev parsing to use Float.parseFloat() and then convert to an int. Now the exception is: Error creating data URI for MADIS product:
        at com.raytheon.edex.db.objects.PluginDataObject.constructDataURI(PluginDataObject.java:192)
  22. Added logger.info() statements to PluginDataObject.constructDataURI() to find out which madis field is causing the problem. Its ObTime, but why? constructDataURI() looks like it deals with the Calendar type. Added more
    logging statements. Didn't help. Added a try/catch block around the PropertyUtils.getProperty() call in constructDataURI(). Got:
    ERROR 2008-05-16 22:30:38,239 [Awips.Edex.Service.IngestSrv-madis.2] MadisRecord: java.lang.NoSuchMethodException: Unknown property ObTime'
    With no trace back. Strange.
  23. Looked at the obs plugin per Tom Kretz' suggestion. It does not use any Calendar type data members for the dataURI, and it does not look as though any of the other Raytheon plugins do either. Changed madis.db.xml so that ObTime is not part of the dataURI.
  24. It works! At least mule says the records are being "Successfully persisted to database".
  25. Brought up pgadmin3. Had to get Leigh's help to find the data, but we did find it: Created a server with "awips" as the password and myself as the user (since this is the edex I installed as myself). Looked under databases/metadata/Schemas/awips/Tables/madis*. Used a 3rd button click on eachmadis label to "View Data".
  26. Created a tar file of my plugin from the projects/AWIPSEdex directory. You can get it here. Again, you should be able to extract it into your ADE working area with out any trouble as long as you don't have an extensions/plugin-madis directory. This tar file does not include build/deployments/deployment.properties. You will need to add plugin-madis to the end of this file.
Last modified 20 May 08.