Reference

The source code is in a folder named src. Inside that are files for the top level application (main.py), a module that serves as the interface to the server (op.py), and a folder named gui that has definitions of the components in graphical user interface.

src
├── gui
│   ├── app.py
│   ├── budgets.py
│   ├── infobox.py
│   ├── output.py
│   ├── regionbox.py
│   ├── styles.py
│   ├── targetbox.py
│   └── tgmap.py
├── main.py
└── op.py

main.py

This file has the main entry point called when the program is launched from the command line. It uses argparse to get command line options, initializes the OptiPass interface, creates the Panel application, and starts the application.

make_app

Instantiate the top level widget.

Returns:
  • a TideGatesApp object

Source code in src/main.py
60
61
62
63
64
65
66
67
68
69
70
def make_app():
    """
    Instantiate the top level widget.

    Returns:
        a TideGatesApp object
    """
    return TideGatesApp(
        title='Tide Gate Optimization', 
        sidebar_width=450,
    )

start_app

Launch the Bokeh server.

Source code in src/main.py
72
73
74
75
76
77
78
79
80
81
82
83
def start_app(port):
    """
    Launch the Bokeh server.
    """
    pn.extension(design='native')
    pn.serve( 
        {'tidegates': make_app},
        port = port,
        verbose = True,
        autoreload = True,
        websocket_origin= '*',
    )

op.py

A class named OP provides an abstract interface to the data for the current project. For example, the "widgets" in the GUI call OP methods to get the list of region names or descriptions of restoration targets.

The module is essentially a "singleton object". It defines a collection of static methods. When the top level application starts it calls a method named setup, which initializes all the data; after that the GUI objects call methods to get values of the data.

MetaOP

The OP class is actually constructed at runtime by a "metaclass". When the application is started, OP has a few methods but no data. The setup method fetches the data from the server and adds it to the OP class.

Here's a concrete example of how this is done, using target data, which is stored in a Pandas dataframe. The metaclass, which is named MetaOP, defines a method named target_frame:

    @property
    def target_frame(cls):
        return cls._target_frame

That definition defines the name, but that name doesn't refer to any acutal table of values at this point.

The setup method includes these lines to fetch the target descriptions from the server, convert those descriptions into a Pandas dataframe, and save the frame:

        req = f'{server}/targets/{project}'
        resp = requests.get(req)
        if resp.status_code != 200:
            raise OPServerError(resp)
        dct = resp.json()
        buf = StringIO(dct['targets'])
        cls._target_frame = pd.read_csv(buf).set_index('abbrev')

The first line creates the URL of the REST request to send to the remote server, the second line sends the request. When the response comes back, the third line makes sure the request succeeded. Lines 4 and 5 create the data frame, and the last line saves it in an internal variable named _target_frame.

From this point on, methods in the GUI can access the data by using the expression OP.target_frame. That will call the target_frame method shown above, which accesses the value in the internal variable and returns it to the GUI.

Bases: type

This metaclass creates the API for the OP class. It defines read-only attributes that can be accessed but not written from outside the OP module. The values of the attributes can only be set when the setup method is called. Note: one attribute (region_names) is writeable.

setup(server, project, tab)

Initialize the connection to the OptiPass server. Connect to the server, get the barrier file and other data for the project, save it in read-only class variables.

Raises an exception if the server is not accessible or if the project name is not known to the server.

Parameters:
  • server (str) –

    the URL of an OptiPass REST server

  • project (str) –

    the name of a data set on the server

  • tab (int) –

    the tab to show when starting the app

Source code in src/op.py
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
def setup(cls, server: str, project: str, tab: int):
    '''
    Initialize the connection to the OptiPass server.  Connect to the
    server, get the barrier file and other data for the project, save
    it in read-only class variables.

    Raises an exception if the server is not accessible or if the project
    name is not known to the server.

    Arguments:
      server: the URL of an OptiPass REST server
      project: the name of a data set on the server
      tab: the tab to show when starting the app
    '''
    logging.info(f'Connecting to {server}')

    req = f'{server}/projects'
    logging.info(f'request: {req}')
    resp = requests.get(req)
    if resp.status_code == 502:
        raise requests.exceptions.ConnectionError()
    elif resp.status_code != 200:
        raise OPServerError(resp)
    elif project not in resp.json():
        raise ValueError(f'unknown project: {project}')
    cls._server_url = server
    cls._project_name = project

    req = f'{server}/targets/{project}'
    resp = requests.get(req)
    if resp.status_code != 200:
        raise OPServerError(resp)
    dct = resp.json()
    buf = StringIO(dct['targets'])
    cls._target_frame = pd.read_csv(buf).set_index('abbrev')
    cls._target_layout = dct['layout'].strip().split('\n')

    req = f'{server}/colnames/{project}'
    resp = requests.get(req)
    if resp.status_code != 200:
        raise OPServerError(resp)
    dct = resp.json()
    cls._mapping_name = dct['name']
    cls._target_columns = dct['files']

    req = f'{server}/mapinfo/{project}'
    resp = requests.get(req)
    if resp.status_code != 200:
        raise OPServerError(resp)
    cls._mapinfo = json.loads(resp.json()['mapinfo'])

    req = f'{server}/barriers/{project}'
    resp = requests.get(req)
    if resp.status_code != 200:
        raise OPServerError(resp)
    buf = StringIO(resp.json()['barriers'])
    cls._barrier_frame = pd.read_csv(buf).set_index('ID')

    total_cost = cls._barrier_frame[['region','cost']].groupby('region').sum()
    cls._region_names = sorted(list(total_cost.index))
    cls._total_cost = { r[0]: r[1].cost for r in total_cost.iterrows() }

    cls._initial_tab = tab

    logging.info('setup complete')

OP

Interface to an OptiPass server. The module consists of a set of static methods that manage a single connection (i.e. it's basically a singleton object).

url_for_figure(fn) staticmethod

Return the URL to use to fetch an image from the server.

Parameters:
  • fn (str) –

    the file name of the image

Returns:
  • str

    the URL

Source code in src/op.py
152
153
154
155
156
157
158
159
160
161
162
163
@staticmethod
def url_for_figure(fn: str) -> str:
    '''
    Return the URL to use to fetch an image from the server.

    Arguments:
      fn: the file name of the image

    Returns:
      the URL
    '''
    return f'{OP.server_url}/static/images/'

format_budgets(cols) staticmethod

Create a dictionary that maps budget values to abbreviated dollar amounts.

Parameters:
  • cols (list[int]) –

    the list of budget amounts

Returns:
  • dict[int, str]

    dictionary

Source code in src/op.py
165
166
167
168
169
170
171
172
173
174
175
176
177
@staticmethod
def format_budgets(cols: list[int]) -> dict[int,str]:
    '''
    Create a dictionary that maps budget values to abbreviated
    dollar amounts.

    Arguments:
      cols: the list of budget amounts

    Returns:
      dictionary 
    '''
    return { n: OP.format_budget_amount(n) for n in cols }

format_budget_amount(n) staticmethod

Convert an integer dollar amount into an abbreviation, e.g. 100000 becomes "$100K" and 2500000 becomes "$2.5M".

Parameters:
  • n (int) –

    the amount to convert

Returns:
  • str

    the abbreviation

Source code in src/op.py
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
@staticmethod
def format_budget_amount(n: int) -> str:
    '''
    Convert an integer dollar amount into an abbreviation, e.g.
    100000 becomes "$100K" and 2500000 becomes "$2.5M".

    Arguments:
      n: the amount to convert

    Returns:
      the abbreviation
    '''
    divisor, suffix = OP.dollar_format['mil'] if n >= 1000000 else OP.dollar_format['thou']
    s = '${:}'.format(n/divisor)
    if s.endswith('.0'):
        s = s[:-2]
    return s+suffix

fetch_html_file(fn) staticmethod

Fetch an HTML file from the server.

Parameters:
  • fn (str) –

    the name of the file

Returns:
  • str

    the contents of the file, as a single string

Source code in src/op.py
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
@staticmethod
def fetch_html_file(fn: str) -> str:
    '''
    Fetch an HTML file from the server.

    Arguments:
      fn:  the name of the file

    Returns:
      the contents of the file, as a single string
    '''
    req = f'{OP.server_url}/html/{OP.project_name}/{fn}'
    resp = requests.get(req)
    if resp.status_code != 200:
        raise OPServerError(resp)
    return resp.json()

run_optimizer(regions, budgets, targets, weights, mapping) staticmethod

Send a request to the op-server to run OptiPass using settings from the widgets.

Parameters:
  • regions (list[str]) –

    a list of geographic regions (river names) to use

  • budgets (tuple[str, str]) –

    a tuple with budget settings (start, increment, count)

  • targets (list[str]) –

    a list of IDs of targets

  • weights (list[int]) –

    a list of target weights

  • mapping (str | None) –

    the name of a column mapping file for targets

Returns:
  • summary

    data frame with one row per budget level

  • matrix

    data frame with one row per barrier

Source code in src/op.py
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
@staticmethod
def run_optimizer(
    regions: list[str],
    budgets: tuple[str,str],
    targets: list[str],
    weights: list[int],
    mapping: str | None,       
    ):
    '''
    Send a request to the op-server to run OptiPass using settings
    from the widgets.

    Args:
        regions: a list of geographic regions (river names) to use
        budgets: a tuple with budget settings (start, increment, count)
        targets: a list of IDs of targets
        weights: a list of target weights
        mapping: the name of a column mapping file for targets

    Returns:
        summary: data frame with one row per budget level
        matrix:  data frame with one row per barrier
    '''
    req = f'{OP.server_url}/optipass/{OP.project_name}'
    args = {
        'regions': regions,
        'budgets': budgets,
        'targets': targets,
        'weights': weights or None,
        'mapping': [OP.mapping_name,mapping],
    }
    if token := DevOP.results_dir():
        args['tempdir'] = token

    resp = requests.get(req, args)
    if resp.status_code != 200:
        raise OPServerError(resp)

    dct = resp.json()
    buf = StringIO(dct['summary'])
    summary = pd.read_csv(buf)
    buf = StringIO(dct['matrix'])
    matrix = pd.read_csv(buf).set_index('ID')

    return summary, matrix

OPResult

The run_optimizer method creates an instance of this class each time the server returns a set of results from an optimization run.

Pass the constructor the dictionaries returned by the server and the widget settings (region names, budget levels, target selection) that were passed to run_optimizer.

The code that creates the output tab calls methods of this class to make figures and tables displayed in the GUI.

Source code in src/op.py
278
279
280
281
282
283
284
285
286
287
288
289
290
def __init__(self, regions, budgets, targets, weights, mapping, summary, matrix):
    self.summary = pd.DataFrame(summary)
    self.matrix = pd.DataFrame(matrix)
    self.regions = regions
    self.bmin, self.binc, self.bcount = budgets
    self.targets = targets
    self.weights = weights
    self.mapping = mapping
    self.display_figures = []
    self.download_figures = []

    # The 'gates' column is a string, need to convert it to a list
    self.summary.gates = summary.gates.map(lambda s: json.loads(s.replace("'",'"')))

make_roi_curves()

Generate ROI plots based on computed benefits.

Source code in src/op.py
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
def make_roi_curves(self):
    """
    Generate ROI plots based on computed benefits.
    """

    climate = None

    subtitle = 'Region: ' if len(self.regions) == 1 else 'Regions: '
    subtitle +=  ', '.join(self.regions)

    for i, t in enumerate(self.targets):
        target = OP.target_frame.loc[t]
        title = target.long
        if target.infra:
            title += f' ({self.mapping} {OP.mapping_name})'
        f = self.bokeh_figure(self.summary.budget, self.summary[target.name], title, subtitle, target.label)
        self.display_figures.append((target.short, f))
        f = self.pyplot_figure(self.summary.budget, self.summary[target.name], title, subtitle, target.label)
        self.download_figures.append((target.short, f))

    if len(self.targets) > 1:
        title = 'Combined Potential Benefit'
        if climate:
            title += f' ({OP.mapping} {OP.mapping_name})'
        if self.weights:
            subtitle += '\nTargets:'
            for i, t in enumerate(self.targets):
                target = OP.target_frame.loc[t]
                subtitle += f' {target.short}{int(self.weights[i])}'
        f = self.bokeh_figure(self.summary.budget, self.summary.netgain, title, subtitle, 'Weighted Net Gain')
        self.display_figures.insert(0, ('Net', f))
        f = self.pyplot_figure(self.summary.budget, self.summary.netgain, title, subtitle, 'Weighted Net Gain')
        self.download_figures.insert(0, ('Net', f))

budget_table()

Make a table that has one column for each budget level, showing which barriers were included in the solution for that level.

Source code in src/op.py
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
def budget_table(self):
    """
    Make a table that has one column for each budget level, showing
    which barriers were included in the solution for that level. 
    """
    df = self.summary[['budget', 'gates']]
    colnames = ['Budget', 'gates']
    df = pd.concat([
        df,
        pd.Series(self.summary.gates.apply(len))
    ], axis=1)
    colnames.append('# Barriers')
    for i, t in enumerate(self.targets):
        target = OP.target_frame.loc[t]
        if target.name in self.summary.columns:
            df = pd.concat([df, self.summary[target.name]], axis=1)
            col = target.short
            # if self.weights:
            #     col += f'⨉{self.weights[i]}'
            colnames.append(col)
    df = pd.concat([
        df,
        self.summary[['wph','netgain']]
    ], axis=1)
    colnames += ['WPH','Net Gain']
    df.columns = colnames
    return df

gate_table()

Make a table that has one row per gate with columns that are relevant to the output display

Source code in src/op.py
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
def gate_table(self):
    """
    Make a table that has one row per gate with columns that are relevant
    to the output display
    """
    filtered = OP.barrier_frame[OP.barrier_frame.region.isin(self.regions)]

    col1 = [c for c in ['region','cost','DSID','type'] if c in filtered.columns]
    budget_cols = [c for c in self.matrix.columns if c.isnumeric() and c > '0']
    target_cols = [c for c in self.matrix.columns if len(c) == 2]
    col2 = [c for c in ['primary','dominant','X','Y'] if c in filtered.columns]

    df = pd.concat([
        filtered[col1],
        self.matrix[budget_cols],
        self.matrix['count'],
        self.matrix[target_cols],
        filtered[col2]
    ], axis=1)

    df = df[df['count'] > 0].sort_values(by='count', ascending=False).fillna('-')
    df.columns = [s.capitalize() if s in ['region','cost','type','primary','dominant'] else s for s in df.columns]

    return df

DevOP

A collection of utility functions for developers. If there is an environment variable that defines a value for a widget the code that builds the GUI will put that value in the widget when it is created, e.g. if OPREGIONS is set to "Coos Umpqua Coquille" those three regions will be selected in the region box.

If OPTMPDIR is defined it should be the path to a directory on the server that has outputs from a previous optimization. The path will be included in the request URL that runs OptiPass. When the server see this it will return the previous results instead of running OptiPass again -- very useful for testing on macOS (which can't run OptiPass).

default_list(varname) staticmethod

Split the value of an environment variable into a list

Source code in src/op.py
441
442
443
444
445
446
447
448
@staticmethod
def default_list(varname):
    '''
    Split the value of an environment variable into a list
    '''
    if lst := os.getenv(varname):
        return lst.split(':')
    return []

default_regions() staticmethod

Return the value of OPREGIONS

Source code in src/op.py
450
451
452
453
454
455
@staticmethod
def default_regions():
    '''
    Return the value of OPREGIONS
    '''
    return DevOP.default_list('OPREGIONS')

default_budget() staticmethod

Return the value of OPBUDGET

Source code in src/op.py
457
458
459
460
461
462
@staticmethod
def default_budget():
    '''
    Return the value of OPBUDGET
    '''
    return int(os.getenv('OPBUDGET','0'))

default_targets() staticmethod

Return the value of OPTARGETS

Source code in src/op.py
464
465
466
467
468
469
@staticmethod
def default_targets():
    '''
    Return the value of OPTARGETS
    '''
    return DevOP.default_list('OPTARGETS')

results_dir() staticmethod

Return the value of OPTMPDIR

Source code in src/op.py
471
472
473
474
475
476
@staticmethod
def results_dir():
    '''
    Return the value of OPTMPDIR
    '''
    return os.getenv('OPTMPDIR')