Parameter Grids

Parameter grids!

This lets you do basic grid expansion and grid arithmetic.

g = Grid([
    ('a', [1, 2]),
    ('b', [1, 2]),
], name='train')

# append two configurations
g = g + LiteralGrid([{'a': 5, 'b': 5}, {'a': 10, 'b': 10}])

# create a bigger grid from the product of another grid
g = g * Grid([
    ('c', [5, 6])
], name='dataset')

assert list(g) == [
    {'a': 1, 'b': 1, 'c': 5},
    {'a': 1, 'b': 1, 'c': 6},
    {'a': 1, 'b': 2, 'c': 5},
    {'a': 1, 'b': 2, 'c': 6},
    {'a': 2, 'b': 1, 'c': 5},
    {'a': 2, 'b': 1, 'c': 6},
    {'a': 2, 'b': 2, 'c': 5},
    {'a': 2, 'b': 2, 'c': 6},
    {'a': 5, 'b': 5, 'c': 5},
    {'a': 5, 'b': 5, 'c': 6},
    {'a': 10, 'b': 10, 'c': 5},
    {'a': 10, 'b': 10, 'c': 6},
]

class slurmjobs.grid.Grid(_Grid__grid, name=None, **constants)

A parameter grid! To get all combinations from the grid, just do list(Grid(...)).

Parameters

• grid (list, dict) –

  The parameter grid. Should be either a dict or a list of key values, where the values are a list of values to use in the grid. Examples of valid inputs:

  # simple grid
  Grid([ ('a', [1, 2]), ('b', [1, 2]) ])
  Grid({ 'a': [1, 2], 'b': [1, 2] })
  
  # paired parameters
  Grid([
      ('a', [1, 2]),
      (('b', 'c'), ([1, 2], [1, 2]))
  ])
  Grid([
      ('a', [1, 2]),
      [{'b': 1, 'c': 1}, {'b': 2, 'c': 2}],
  ])
  
  # any of these are valid grid specs
  g = slurmjobs.Grid([
      # basic
      ('a', [1, 2]),
      # paired
      (('b', 'c'), ([1, 1, 2, 2], [1, 2, 1, 2])),
  
      # literal list of dicts
      [{'d': i} for i in [1, 2]],
      # dict generator
      ({'e': i} for i in [1, 2]),
      # function
      lambda: [{'f': i} for i in [1, 2]],
      # function that returns a generator
      lambda: ({'g': i} for i in [1, 2]),
  
      # basic generator
      ('h', (x for x in [1, 2])),
      # basic function
      ('i', lambda: [x for x in [1, 2]]),
  ])
  keys = 'abcdefghi'
  assert list(g) == [
      dict(zip(keys, vals)) for vals in
      itertools.product(*([ [1, 2] ]*len(keys)))
  ]
  

• name (str) – The name of this grid. Can be used to search for the parameters from this grid.

• **constants – Extra parameters to add to the grid that don’t vary. These will not be included in the job_id name.

Just a heads up, there is nothing stopping you from passing an infinite generator, meaning that you can make some fancy sampling grid generators, but slurmjobs will take that and not know when to stop. If you want to use an infinite generator, just wrap it in itertools.islice which will let you provide a limit.

Obviously, slurmjobs doesn’t operate anywhere near the memory scale where you’d need to even use generators in the first place, but I figured why limit the implementation if it can be used for other things too.

__len__()

Get the number of iterations in the grid.

Note that any use of generators or functions without a length will cause this to raise a TypeError.

__getitem__(index)

Get the series for a variable name.

__iter__()

Yield all combinations from the parameter grid.

class slurmjobs.grid.LiteralGrid(_LiteralGrid__grid, name=None, **constants)

A parameter grid, specified as a flattened list. This doesn’t do any grid expansion, it lets you specify the grid as you want.

Parameters

• grid (list, dict) – The parameter list. Should be a list of dicts, each corresponding to a parameter config.

• name (str) – The name of this grid. Can be used to search for the parameters from this grid.

g = LiteralGrid([
    {'a': 1, 'b': 1},
    {'a': 1, 'b': 2},
    {'a': 2, 'b': 2},
])
assert list(g) == [
    {'a': 1, 'b': 1},
    {'a': 1, 'b': 2},
    {'a': 2, 'b': 2},
]

__len__()

Get the number of iterations in the grid.

Note that any use of generators or functions without a length will cause this to raise a TypeError.

__iter__()

Yield all combinations from the parameter grid.

Grid Operations

class slurmjobs.grid.GridCombo(*grids, name=None)

This handles the multiplication of two grids (combinations). It will create a grid as a product of all provided grids.

You can create this doing grid_a * grid_b. The only reason to use this directly is if you want to give it a name or if you want to make a grid product of 3 or more grids.

a = Grid(('a', [1, 2]), ('b', [1, 2]))
b = Grid(('c', [1, 2]))

# functionally equivalent
c = a * b
c = GridCombo(a, b, name='my-a-b-combo-grid')
c_items = [
    dict(da, **db)
    for da, db in itertools.product(a, b)
]

__len__()

Get the number of iterations in the grid.

Note that any use of generators or functions without a length will cause this to raise a TypeError.

__iter__()

Yield all combinations from the parameter grid.

class slurmjobs.grid.GridChain(*grids, name=None)

This handles the addition of two grids (one after the other).

You can create this doing grid_a + grid_b. The only reason to use this directly is if you want to give it a name.

a = Grid(('a', [1, 2]), ('b', [1, 2]))
b = Grid(('c', [1, 2]))

# functionally equivalent
c = a + b
c = GridChain(a, b, name='my-a-then-b-grid')
c_items = list(a) + list(b)

__len__()

Get the number of iterations in the grid.

Note that any use of generators or functions without a length will cause this to raise a TypeError.

__iter__()

Yield all combinations from the parameter grid.

class slurmjobs.grid.GridOmission(grid, omission, name=None)

This handles the subtraction of two grids (combinations). It will yield only dicts from grid_a that don’t appear in grid_b.

You can create this doing grid_a - grid_b. The only reason to use this directly is if you want to give it a name.

a = Grid(('a', [1, 2]), ('b', [1, 2]))
b = Grid(('a', [2]), ('b', [1]))

# functionally equivalent
c = a - b
c = GridOmission(a, b, name='my-a-minus-b-grid')
omit = list(b)
c_items = [da for da in a if da not in omit]

__len__()

Get the number of iterations in the grid.

Note that any use of generators or functions without a length will cause this to raise a TypeError.

__iter__()

Yield all combinations from the parameter grid.

Extending Grids

class slurmjobs.grid.BaseGrid(name=None, ignore_job_id_keys=None, **constants)

The base class for all grids. Use this if you want to extend another grid.

You just need to implement:

• __iter__: This should yield all items generated by the grid

• __len__: This should tell you the number of items in the grid. If you cannot determine the length of a grid, then raise a TypeError (as the other grids do).

• __repr__: A nice string representation of the grid

__repr__()

A nice string representation of the grid.

__len__()

Get the number of iterations in the grid.

Note that any use of generators or functions without a length will cause this to raise a TypeError.

__iter__()

Yield all combinations from the parameter grid.

__add__(other)

Combine two parameter grids sequentially.

__mul__(other)

Create a grid as the combination of two grids.

classmethod as_grid(grid)

Ensure that a value is a grid.