Collection reference

The full Collection API, after a task index. The guide shows the common methods in use.

Task	Method
Keep matching elements	filter()
Group under one or several keys	groupby(), multi_groupby()
Apply lookup thresholds	filter_above_threshold(), filter_any_above_threshold()
Preserve metadata while replacing elements	clone_with_new_elements()
Write record-shaped elements to a table	to_dataframe(), to_csv()
Save or reload supported objects	to_json(), inherited from_json()

Table conversion expects elements that the serialization helper can turn into field dictionaries. Strings, bytes, numbers and None are the exception: each becomes a row with a single value column. to_csv() leaves out the row numbers unless called with index=True.

Collection

Bases: Serializable

Collection base class with grouping and filtering functions.

source property

source

Returns the single source name for a variant collection if it is unique, otherwise raises an error.

path property

path

Alternative name for source, maintained for backward compatibility.

filenames property

filenames

Sorted base names of the non-empty sources, assuming they are file paths (for example VCF or MAF files). A source without a base name, such as a directory path ending in a separator, is left out.

filename property

filename

First entry of filenames (the only one when there is a single source), kept for backward compatibility.

Raises ValueError when filenames is empty.

__init__

__init__(elements, distinct=False, sort_key=None, sources=None)

Parameters:

• elements (iterable) –

  Elements of any class which is compatible with the sort key. They are copied into a new list, so an iterator or generator is accepted and later changes to the argument don't affect this collection.

• distinct (bool, default: False ) –

  Keep only the first occurrence of each element (elements must be hashable) or allow duplicates.

• sort_key (fn, default: None ) –

  Function which maps each element to a sorting criterion.

• sources (set of str, str, path or None, default: None ) –

  Files from which this collection was generated. They are copied into a new set, so collections never share one; a single string or path is one source and None means no sources.

to_dict

to_dict()

Return the constructor arguments needed to rebuild this collection.

This also serves subclasses whose constructor takes the same arguments, such as one that inherits Collection.__init__, so they can be pickled, copied and saved as JSON without their own to_dict.

Returns:

• dict –

  elements, distinct, sort_key and sources.

Raises:

• ValueError –

  If this object's class has a constructor with other arguments. Such a subclass must define its own to_dict.

to_json

to_json()

Returns a string containing a JSON representation of this collection.

Raises:

• ValueError –

  If sort_key is a lambda or a function defined inside another function. JSON stores a function as its module and name, so only one which can be looked up that way can be loaded again.

to_dataframe

to_dataframe()

Build a table with one row per element.

Strings, bytes, numbers and None each become a row with a single value column. Any other element is converted with serializable.to_dict, so it must be a dict, an object with a to_dict method (such as a Serializable instance), or an object whose attributes match its constructor arguments.

Returns:

• DataFrame –

to_csv

to_csv(path, index=False)

Write to_dataframe() to a CSV file.

Parameters:

• path (str or path - like) –

  Destination file; an existing file is overwritten.

• index (bool, default: False ) –

  Also write the row numbers as an unnamed first column.

clone_with_new_elements

clone_with_new_elements(new_elements, drop_keywords=(), rename_dict=None, extra_kwargs=None)

Create another Collection of the same class and with same state but possibly different entries. Extra parameters to control which keyword arguments get passed to the initializer are necessary since derived classes have different constructors than the base class.

A subclass constructor which differs only by leaving out distinct, sort_key or sources, or by giving its first argument a name other than elements, needs none of the extra parameters: the arguments it doesn't take are left out and the elements are passed by position. The elements are only passed by position to a first argument which is required; one with a default value, or *args, is not assumed to hold the elements.

Parameters:

• new_elements (list) –

  Elements of the new collection.

• drop_keywords (set of str, default: () ) –

  Base-class constructor arguments (distinct, sort_key, sources) to leave out.

• rename_dict (dict, default: None ) –

  Maps base-class argument names to the names the subclass constructor uses.

• extra_kwargs (dict, default: None ) –

  Additional constructor arguments, applied last.

Returns:

• Collection –

  A new instance of self.__class__.

Raises:

• TypeError –

  If the constructor of self.__class__ can't be called with the resulting arguments, for example because it requires others. Such a subclass must pass extra_kwargs or override this method.

short_string

short_string()

Compact string representation which doesn't print any of the collection elements. Sources are listed in sorted order.

to_string

to_string(limit=None)

Create a string representation of this collection, showing up to limit items (all of them when limit is None).

Raises ValueError when limit is negative.

__eq__

__eq__(other)

Collections are equal when they have the same class, the same elements in the same order and the same sources.

filter

filter(filter_fn)

Keep the elements for which filter_fn returns a true value.

Parameters:

• filter_fn (callable) –

  Takes one element and returns a truthy or falsy value.

Returns:

• Collection –

  A collection of the same class, built with clone_with_new_elements.

groupby

groupby(key_fn)

Group elements by the key key_fn returns for each.

Parameters:

• key_fn (callable) –

  Takes one element and returns a hashable grouping key.

Returns:

• dict –

  Maps each key to a collection of the same class, built with clone_with_new_elements from that key's elements in iteration order (so distinct and sort_key still apply).

multi_groupby

multi_groupby(key_fn)

Like groupby, but key_fn returns several keys for each element, and the element is added to the group for every one of them.

Parameters:

• key_fn (callable) –

  Takes one element and returns an iterable of hashable keys.

Returns:

• dict –

  Maps each key to a collection of the same class.

filter_above_threshold

filter_above_threshold(key_fn, value_dict, threshold, default_value=0.0)

Keep elements whose looked-up value is above a threshold.

For example, keep variant effects whose gene expression is above a cutoff.

Parameters:

• key_fn (callable) –

  Given an element of this collection, returns a key into value_dict

• value_dict (dict) –

  Dict from keys returned by key_fn to float values

• threshold (float) –

  Only keep elements whose value in value_dict is above this threshold.

• default_value (float, default: 0.0 ) –

  Value to use for elements whose key is not in value_dict

Returns:

• Collection –

  A collection of the same class, as filter returns.

filter_any_above_threshold

filter_any_above_threshold(multi_key_fn, value_dict, threshold, default_value=0.0)

Like filter_above_threshold but multi_key_fn returns multiple keys and the element is kept if any of them have a value above the given threshold.

Parameters:

• multi_key_fn (callable) –

  Given an element of this collection, returns multiple keys into value_dict

• value_dict (dict) –

  Dict from keys returned by multi_key_fn to float values

• threshold (float) –

  Only keep elements whose value in value_dict is above this threshold.

• default_value (float, default: 0.0 ) –

  Value to use for elements whose key is not in value_dict

Returns:

• Collection –

  A collection of the same class, as filter returns.