caosadvancedtools.cfood module#

Defines how something that shall be inserted into LinkAhead is treated.

LinkAhead can automatically be filled with Records based on some structure, a file structure, a table or similar.

The Crawler will iterate over the respective items and test for each item whether a CFood class exists that matches the file path, i.e. whether CFood class wants to treat that pariticular item. If one does, it is instanciated to treat the match. This occurs in basically three steps:

Create a list of identifiables, i.e. unique representation of LinkAhead Records (such as an experiment belonging to a project and a date/time).
The identifiables are either found in LinkAhead or they are created.
The identifiables are update based on the date in the file structure.

class caosadvancedtools.cfood.AbstractCFood(item)#

Bases: object

Abstract base class for Crawler food (CFood).

attach(item)#

collect_information()#

The CFood collects information for further processing.

Often CFoods need information from files or even from the database in order to make processing decision. It is intended that this function is called after match. Thus match can be used without connecting to the database.

To be overwritten by subclasses

abstractmethod create_identifiables()#: should set the instance variable Container with the identifiables

looking_for(item)#

returns True if item can be added to this CFood.

Typically a CFood exists for a file and defines how to deal with the file. However, sometimes additional files “belong” to a CFood. E.g. an experiment CFood might match against a README file but labnotes.txt also shall be treated by the cfood (and not a special cfood created for labnotes.txt) This function can be used to define what files shall be ‘attached’.

To be overwritten by subclasses

classmethod match_item(item)#

Matches an item found by the crawler against this class. Returns True if the item shall be treated by this class, i.e. if this class matches the item.

Parameters:

item (object) – iterated by the crawler
subclasses! (To be overwritten by)

static remove_property(entity, prop)#

static set_parents(entity, names)#

static set_property(entity, prop, value, datatype=None)#

abstractmethod update_identifiables()#: Changes the identifiables as needed and adds changed identifiables to self.to_be_updated

class caosadvancedtools.cfood.AbstractFileCFood(crawled_path, *args, **kwargs)#

Bases: AbstractCFood

property crawled_file#

classmethod get_re()#

Returns the regular expression used to identify files that shall be processed

This function shall be implemented by subclasses.

looking_for(crawled_file)#

returns True if crawled_file can be added to this CFood.

Typically a CFood exists for a file and defines how to deal with the file. However, sometimes additional files “belong” to a CFood. E.g. an experiment CFood might match against a README file but labnotes.txt also shall be treated by the cfood (and not a special cfood created for labnotes.txt) This function can be used to define what files shall be ‘attached’.

classmethod match_item(path)#

Matches the regular expression of this class against file names

Parameters:: path (str) – The path of the file that shall be matched.

static re_from_extensions(extensions)#

Return a regular expression which matches the given file extensions.

Useful for inheriting classes.

Parameters:: extensions (iterable<str>) – An iterable with the allowed extensions.
Returns:: out – The regular expression, starting with .*\. and ending with the EOL dollar character. The actual extension will be accessible in the pattern group name ext.
Return type:: str

class caosadvancedtools.cfood.CMeal#

Bases: object

CMeal groups equivalent items and allow their collected insertion.

Sometimes there is no one item that can be used to trigger the creation of some Record. E.g. if a collection of image files shall be referenced from one Record that groups them, it is unclear which image should trigger the creation of the Record.

CMeals are grouped based on the groups in the used regular expression. If, in the above example, all the images reside in one folder, all groups of the filename match except that for the file name should match. The groups that shall match need to be listed in the matching_groups class property. Subclasses will overwrite this property.

This allows to use has_suitable_cfood in the match_item function of a CFood to check whether the necessary CFood was already created. In order to allow this all instances of a CFood class are tracked in the existing_instances class member.

Subclasses must have a cls.get_re function and a match member variable (see AbstractFileCFood)

classmethod all_groups_equal(m1, m2)#

belongs_to_meal(item)#

existing_instances = []#

static get_re()#

classmethod has_suitable_cfood(item)#

checks whether the required cfood object already exists.

item : the crawled item

matching_groups = []#

class caosadvancedtools.cfood.FileGuide#

Bases: object

access(path)#

should be replaced by a function that adds a prefix to paths to allow to access LinkAhead files locally

This default just returns the unchanged path.

class caosadvancedtools.cfood.RowCFood(item, unique_cols, recordtype, **kwargs)#

Bases: AbstractCFood

create_identifiables()#: should set the instance variable Container with the identifiables

update_identifiables()#: Changes the identifiables as needed and adds changed identifiables to self.to_be_updated

caosadvancedtools.cfood.add_files(filemap)#: add to the file cache

caosadvancedtools.cfood.assure_has_description(entity, description, to_be_updated=None, force=False)#

Checks whether entity has the description that is passed.

If this is the case this function ends. Otherwise the entity is assigned a new description. The list to_be_updated is supplied, the entity is added to the list in order to indicate, that the entity entity should be updated. Otherwise it is directly updated

caosadvancedtools.cfood.assure_has_parent(entity, parent, to_be_updated=None, force=False, unique=True)#

Checks whether entity has a parent with name parent.

If this is the case this function ends. Otherwise the entity is assigned a new parent. The list to_be_updated is supplied, the entity is added to the list in order to indicate, that the entity entity should be updated. Otherwise it is directly updated

caosadvancedtools.cfood.assure_has_property(entity, name, value, to_be_updated=None, datatype=None, setproperty=False)#

Checks whether entity has a property name with the value value.

If this is the case this function ends. Otherwise the entity is assigned a new parent.

Note that property matching occurs based on names.

If the list to_be_updated is supplied, the entity is added to the list in order to indicate, that the entity entity should be updated. Otherwise it is directly updated

setproperty: boolean, if True, overwrite existing properties.

caosadvancedtools.cfood.assure_name_is(entity, name, to_be_updated=None, force=False)#

Checks whether entity has the name that is passed.

If this is the case this function ends. Otherwise the entity is assigned a new name. The list to_be_updated is supplied, the entity is added to the list in order to indicate, that the entity entity should be updated. Otherwise it is directly updated

caosadvancedtools.cfood.assure_object_is_in_list(obj, containing_object, property_name, to_be_updated=None, datatype=None)#

Checks whether obj is one of the values in the list property property_name of the supplied entity containing_object.

If this is the case this function returns. Otherwise the entity is added to the property property_name and the entity containing_object is added to the supplied list to_be_updated in order to indicate, that the entity containing_object should be updated. If none is submitted the update will be conducted in-place.

If the property is missing, it is added first and then the entity is added/updated.

If obj is a list, every element is added

caosadvancedtools.cfood.assure_parents_are(entity, parents, to_be_updated=None, force=False, unique=True)#

Checks whether entity has the provided parents (and only those).

If this is the case this function ends. Otherwise the entity is assigned the new parents and the old ones are discarded.

Note that parent matching occurs based on names. If a parent does not have a name, a ValueError is raised.

If the list to_be_updated is supplied, the entity is added to the list in order to indicate, that the entity entity should be updated. Otherwise it is directly updated

parents: single string or list of strings

caosadvancedtools.cfood.assure_property_is(entity, name, value, datatype=None, to_be_updated=None, force=False)#

Checks whether entity has a Property name with the given value.

If this is the case this function ends. Otherwise the entity is assigned a new property or an existing one is updated.

If the list to_be_updated is supplied, the entity is added to the list in order to indicate, that the entity entity should be updated. Otherwise it is directly updated

caosadvancedtools.cfood.assure_special_is(entity, value, kind, to_be_updated=None, force=False)#

Checks whether entity has the name or description that is passed.

If this is the case this function ends. Otherwise the entity is assigned a new name. The list to_be_updated is supplied, the entity is added to the list in order to indicate, that the entity entity should be updated. Otherwise it is directly updated

caosadvancedtools.cfood.get_entity(name)#

Returns the entity with a given name, preferably from a local cache.

If the local cache does not contain the entity, retrieve it from LinkAhead.

caosadvancedtools.cfood.get_entity_for_path(path)#

caosadvancedtools.cfood.get_ids_for_entities_with_names(entities)#

caosadvancedtools.cfood.get_property(name)#

Returns the property with a given name, preferably from a local cache.

If the local cache does not contain the record type, try to retrieve it from LinkAhead. If it does not exist, see whether it could be a record type used as a property.

caosadvancedtools.cfood.get_record(name)#

Returns the record with a given name, preferably from a local cache.

If the local cache does not contain the record, try to retrieve it from LinkAhead.

caosadvancedtools.cfood.get_recordtype(name)#

Returns the record type with a given name, preferably from a local cache.

If the local cache does not contain the record type, try to retrieve it from LinkAhead. If it does not exist, add it to the data model problems

caosadvancedtools.cfood.insert_id_based_on_name(entity)#