At the USDA Forest Service’s Western Wildland Environmental Threat Assessment Center (WWETAC), we are exploring webcrawlers to facilitate wildland threat assessments. The Threat Center was established by Congress in 2005 to facilitate the development of tools and methods for the assessment of multiple interacting threats (wildfire, insects, disease, invasive species, climate change, land use change)
However, they also note that
much of the online data is stored in large institutional data warehouses (Natureserve, Geodata.gov, etc.) that have their own catalog and searching systems and are not open to webcrawlers like ours. In fact, most federal land management agencies do not allow services to their data, but allow downloading and in-house viewers (i.e. FHTET 2006). This policy does not simplify the problem of integrated threat assessments for federal land management agencies.
The group is now developing a more powerful webcrawler. You can find and search the database for geospatial data and map here
. Still a long way to go it seems, but a very important first step!