Batch Processing on Windows — InVEST 3.10.2.post0+gc978f9e5.d20220207 documentation (2024)

»
Tutorial: Batch Processing on Windows
Edit on GitHub

Introduction¶

These are the steps that will need to be taken in order to use the batchscripting framework for InVEST models available in the natcap.invest pythonpackage.

For those wishing to do batch-processing with InVEST without setting up aPython scripting environment, see The InVEST CLI for examples of how to runInVEST models from the command-line.

Setting up your Python environment¶

Install Python 3.6 or later.
Python can be downloaded from here. When installing,be sure to allow python.exe to be added to the path in the installation options.
Put pip on the PATH.
The pip utility for installing python packages is already included withPython 2.7.9 and later. Be sure to add C:\Python37\Scripts (or your custominstall location) to the Windows PATH environment variable so that pip canbe called from the command line without needing to use its full path.
After this is done (and you’ve opened a new command-line window), you will beable to use pip at the command-line to install packages like so:
```
> pip install <packagename>
```

Install packages needed to run InVEST.

Most (maybe even all) of these packages can be downloaded as precompiledwheels from Christoph Gohlke’s build page.Others should be able to be installed via pip install <packagename>.

## Any lines with "# pip-only" at the end will be processed by# scripts/convert-requirements-to-conda-yml.py as though it can only be found# on pip.GDAL>=3.1.2,!=3.3.0 # 3.3.0 had a bug that broke our windows builds: https://github.com/OSGeo/gdal/issues/3898Pyro4==4.77 # pip-onlypandas>=1.2.1numpy>=1.11.0,!=1.16.0Rtree>=0.8.2,!=0.9.1Shapely>=1.7.1,<2.0.0scipy>=1.6.0 # pip-onlypygeoprocessing>=2.3.2 # pip-onlytaskgraph[niced_processes]>=0.11.0 # pip-onlypsutil>=5.6.6chardet>=3.0.4openpyxlpintBabel

Creating Sample Python Scripts¶

Launch InVEST Model
Once an InVEST model is selected for scripting,launch that model from the Windows Start menu. This example in this guidefollows the NDR model.
Fill in InVEST Model Input Parameters
Once the user interface loads, populate the inputs in the model likely tobe used in the Python script. For testing purposes the default InVEST’sdata is appropriate. However, if a user wishes to write a batch for severalInVEST runs, it would be reasonable to populate the user interface withdata for the first run.
Generate a sample Python Script from the User Interface
Open the Development menu at the top of the user interface and select“Save to python script…” and save the file to a known location.

Execute the script in the InVEST Python Environment
Launch a Windows PowerShell from the Start menu (type “powershell” in thesearch box), then invoke the Python interpreter on the InVEST Pythonscript from that shell. In this example the Python interpreter isinstalled in C:\Python37\python.exe and the script was saved inC:\Users\rpsharp\Desktop\ndr.py, thus the command to invoke theinterpreter is:
```
> C:\Python37\python.exe C:\Users\rpsharp\Desktop\ndr.py
```

Output Results
As the model executes, status information will be printed to the console.Once complete, model results can be found in the workspace folder selectedduring the initial configuration.

Modifying a Python Script¶

InVEST Python scripts consist of two sections:

The argument dictionary that represents the model’s user interface inputboxes and parameters.
The call to the InVEST model itself.

For reference, consider the following script generated by the Nutrient model with default data inputs:

"""This is a saved model run from natcap.invest.ndr.ndr.Generated: Mon 16 May 2016 03:52:59 PMInVEST version: 3.3.0"""import natcap.invest.ndr.ndrargs = { u'k_param': u'2', u'runoff_proxy_uri': u'C:\InVEST_3.3.0_x86\Base_Data\Freshwater\precip', u'subsurface_critical_length_n': u'150', u'subsurface_critical_length_p': u'150', u'subsurface_eff_n': u'0.8', u'subsurface_eff_p': u'0.8', u'threshold_flow_accumulation': u'1000', u'biophysical_table_uri': u'C:\InVEST_3.3.0_x86\WP_Nutrient_Retention\Input\water_biophysical_table.csv', u'calc_n': True, u'calc_p': True, u'suffix': '', u'dem_uri': u'C:\InVEST_3.3.0_x86\Base_Data\Freshwater\dem', u'lulc_uri': u'C:\InVEST_3.3.0_x86\Base_Data\Freshwater\landuse_90', u'watersheds_uri': u'C:\InVEST_3.3.0_x86\Base_Data\Freshwater\watersheds.shp', u'workspace_dir': u'C:\InVEST_3.3.0_x86\ndr_workspace',}if __name__ == '__main__': natcap.invest.ndr.ndr.execute(args)

Elements to note:

Parameter Python Dictionary: Key elements include the ‘args’ dictionary. Note the similarities between the key values such as ‘workspace_dir’ and the equivalent “Workspace” input parameter in the user interface. Every key in the ‘args’ dictionary has a corresponding reference in the user interface.

In the example below we’ll modify the script to execute the nutrient model for a parameter study of ‘threshold_flow_accumulation’.

Execution of the InVEST model: The InVEST API invokes models with a consistent syntax where the module name that contains the InVEST model is listed first and is followed by a function called ‘execute’ that takes a single parameter called ‘args’. This parameter is the dictionary of input parameters discussed above. In this example, the line

natcap.invest.ndr.ndr.execute(args)

executes the nutrient model end-to-end. If the user wishes to make batch calls to InVEST, this line will likely be placed inside a loop.

Example: Threshold Flow Accumulation Parameter Study¶

This example executes the InVEST NDR model on 10 values of thresholdaccumulation stepping from 500 to 1000 pixels in steps of 50. To modify thescript above, replace the execution call with the following loop:

if __name__ == '__main__': #Loops through the values 500, 550, 600, ... 1000 for threshold_flow_accumulation in range(500, 1001, 50): #set the accumulation threshold to the current value in the loop args['threshold_flow_accumulation'] = threshold_flow_accumulation #set the suffix to be accum### for the current threshold_flow_accumulation args['suffix'] = 'accum' + str(threshold_flow_accumulation) natcap.invest.ndr.ndr.execute(args)

This loop executes the InVEST nutrient model 10 times for accumulation values500, 550, 600, ... 1000 and adds a suffix to the output files so resultscan be distinguished.

Example: Invoke NDR Model on a directory of Land Cover Maps¶

In this case we invoke the InVEST nutrient model on a directory of land coverdata located at C:UserRichDesktoplandcover_data. As in the previousexample, replace the last line in the UI generated Python script with:

import oslandcover_dir = r'C:\User\Rich\Desktop\landcover_data'if __name__ == '__main__': #Loop over all the filenames in the landcover dir for landcover_file in os.listdir(landcover_dir): #Point the landuse uri parameter at the directory+filename args['lulc_uri'] = os.path.join(landcover_dir, landcover_file) #Make a useful suffix so we can differentiate the results args['suffix'] = 'landmap' + os.path.splitext(landcover_file)[0] #call the nutrient model natcap.invest.ndr.ndr.execute(args)

This loop covers all the files located inC:\User\Rich\Desktop\landcover_dataand updates the relevant lulc_uri key in the args dictionary to eachof those files during execution as well as making a useful suffix so outputfiles can be distinguished from each other.

Example: Saving model log messages to a file¶

There are many cases where you may want or need to capture all of the logmessages generated by the model. When we run models through the InVEST userinterface application, the UI captures all of this logging and saves it to alogfile. We can replicate this behavior through the python logging package,by adding the following code just after the import statements in theexample script.

import loggingimport pygeoprocessing# Write all NDR log messages to logfile.txtMODEL_LOGGER = logging.getLogger('natcap.invest.ndr')handler = logging.FileHandler('logfile.txt')MODEL_LOGGER.addHandler(handler)# log pygeoprocessing messages to the same logfilePYGEO_LOGGER = logging.getLogger('pygeoprocessing')PYGEO_LOGGER.addHandler(handler)

This will capture all logging generated by the ndr model and bypygeoprocessing, writing all messages to logfile.txt. Whilethis is a common use case, the logging package provides functionalityfor many more complex logging features. For moreadvanced use of the python logging module, refer to the Python project’sLogging Cookbook

Example: Enabling Parallel Processing¶

Note

This is an in-development feature and should be used with caution.

Most InVEST models accept an optional entry in the args dictionaryrepresenting the number of parallel workers. Acceptable values for thisnumber are:

-1, representing synchronous execution (this is the default acrossInVEST)
0 representing threaded task management
Any other positive integer represents the number of processes that will becreated to handle tasks. 2*multiprocessing.cpu_count() is usually a goodnumber.

Warning

If you use this feature, you must wrap your script in aif __name__ == '__main__': condition. Failure to do so will resultin a fork bomb (https://en.wikipedia.org/wiki/Fork_bomb).

Using the parameter study example, this might look like:

if __name__ == '__main__': args['n_workers'] = 4 # Use 4 processes #Loops through the values 500, 550, 600, ... 1000 for threshold_flow_accumulation in range(500, 1001, 50): #set the accumulation threshold to the current value in the loop args['threshold_flow_accumulation'] = threshold_flow_accumulation #set the suffix to be accum### for the current threshold_flow_accumulation args['suffix'] = 'accum' + str(threshold_flow_accumulation) natcap.invest.ndr.ndr.execute(args)

Internationalization¶

If you use the InVEST python API to access model names, ARGS_SPEC``s, or validation messages, you can translate those strings using ``gettext:

import gettextimport pkg_resources# replace with your desired ISO 639-1 language code# see https://en.wikipedia.org/wiki/List_of_ISO_639-1_codeslanguage_code = 'en'# locate the internationalization data included with the invest package# for languages supported by invest# or alternatively, provide your ownlocale_dir = pkg_resources.resource_filename('natcap.invest', 'internationalization/locales/')language = gettext.translation( 'messages', languages=[language_code], localedir=locale_dir)language.install()import natcap.invest.carbon...

Note that natcap.invest defines the _(...) function globally in builtins. When you import anything from natcap.invest, _ is first defined to return the default English. This way everything works whether or not the client sets up gettext.

See the GNU gettext manual and the Python gettext documentation for more information.

Summary¶

The InVEST scripting framework was designed to assist InVEST users in automating batch runs or adding custom functionality to the existing InVEST software suite. Support questions can be directed to the NatCap support forums at http://community.naturalcapitalproject.org.