Manual

parsley.aspect package¶

Submodules¶

parsley.aspect.abstractaspect module¶

Aspects control the behavior of parsley.syncengine.sync.Sync configurations.

class parsley.aspect.abstractaspect.Aspect¶

Bases: object

Abstract base class for aspect implementations that build the actual behavior of a parsley.syncengine.sync.Sync synchronization implementation.

For implementing custom synchronization behavior, implement a subclass of this class and write some aspect hook methods. An aspect hook method is just a method that follows a certain signature and is registered by means of parsley.aspect.hook (hook()). A minimalist aspect looks like this:

@verbatim class DoSomething(parsley.aspect.abstractaspect.Aspect):

def __init__(self):
parsley.aspect.abstractaspect.Aspect.__init__(self)

@parsley.aspect.hook(“”, “”, “”, event=parsley.syncengine.common.SyncEvent.UpdateDir_Prepare) def sleepwhilebeginupdatedir(self, ctx, filesystem): # this signature must be followed!

do_something()

@endverbatim

A sync configuration can be enhanced by this functionality by adding this aspect to one or more filesystems or globally (which enhances all filesystems).

A hook method is called on top of a certain filesystem whenever a certain event occurs or a certain state in the sync controller is reached. These are actually two different things, even if it sounds similar. It essentially means different modes of the parsley.aspect.hook function (read there for more).

Each call to an aspect hook method gets this parameters:

ctx: a parsley.syncengine.syncruntime.SyncEventRuntime, which holds many flow control information and intermediate data for the execution of one event. It also has some control flow methods.
filesystem: the parsley.filesystem.abstractfilesystem.Filesystem the method is executed on top (it should execute whatever needs to be done for particul filesystem).

Find some function decorators in this module for usage with aspect hook methods, e.g. for some control flow.

parsley.aspect.abstractaspect.execute_only_for_master_fs()¶

Used on aspect hook methods for executing the function body only for the master filesystem.

Creates a function decorator for usage with methods of parsley.aspect.abstractaspect.Aspect subclasses.

Return type: Callable

parsley.aspect.abstractaspect.execute_only_for_master_fs_filetype(*types)¶

Used on aspect hook methods for executing the function body only if the file has certain types in the master filesystem.

Creates a function decorator for usage with methods of parsley.aspect.abstractaspect.Aspect subclasses.

Parameters: types (str) – All arguments are parsley.syncengine.common.EntryType.
Return type: Callable

parsley.aspect.abstractaspect.execute_only_for_non_master_fs()¶

Used on aspect hook methods for executing the function body only for non-master filesystems.

Creates a function decorator for usage with methods of parsley.aspect.abstractaspect.Aspect subclasses.

Return type: Callable

parsley.aspect.abstractaspect.execute_only_for_slave_fs_filetype(*types)¶

Used on aspect hook methods for executing the function body only if the file has certain types in the argument filesystem.

Creates a function decorator for usage with methods of parsley.aspect.abstractaspect.Aspect subclasses.

Parameters: types (str) – All arguments are parsley.syncengine.common.EntryType.
Return type: Callable

parsley.aspect.abstractaspect.execute_only_if_not_already_maximally_elected()¶

Used on aspect hook methods for executing the function body only if there is not already an election with maximum key.

Creates a function decorator for usage with methods of parsley.aspect.abstractaspect.Aspect subclasses.

Return type: Callable

parsley.aspect.abstractaspect.execute_only_if_not_update_set_skipped()¶

Used on aspect hook methods for executing the function body only if the update is not marked as skipped.

Creates a function decorator for usage with methods of parsley.aspect.abstractaspect.Aspect subclasses.

Return type: Callable

parsley.aspect.abstractaspect.hook(after, provides, before, event)¶

Used for registering a function as parsley.syncengine.sync.Sync hook for implementating parts of Parsley synchronization behavior (those are sometimes called ‘aspect hook methods’).

Creates a function decorator for usage with methods of parsley.aspect.abstractaspect.Aspect subclasses.

The parameters control the alignment (i.e. the ordering of all aspect hook methods) and to which event to assign the method.

after, provides and before control the alignment. Please read parsley.syncengine.sync.Sync for details about ordering.

event specifies the event for which this method is to be hooked.

See parsley.syncengine.common.SyncEvent for all events. Read parsley.syncengine.sync.Sync about how all the hook methods are actually executed. Read the user manual about how to directly add custom code into your Parsley configuration files.

Parameters

after (str) –
provides (str) –
before (str) –
event (str) –

Return type

Callable

parsley.aspect.applypathacceptor module¶

class parsley.aspect.applypathacceptor.ApplyPathAcceptor(function)¶

Bases: parsley.aspect.abstractaspect.Aspect

Filters out files/dirs that should be excluded from syncing. It works by providing a path acceptor function.

Parameters: function – The acceptor function. it gets the path as parameter and must return True for accepting it. Warning: This parameter is a string that will be evaluated as Python expression (so usage from config files is easy)! The path argument is available as path. Example: path[2]==’a’. The filesystem is available as fs.

applypathacceptor_applypathacceptor(ctx, filesystem)¶: Skips this entry if it is rejected by the path acceptor.

parsley.aspect.baseinfrastructure module¶

class parsley.aspect.baseinfrastructure.BaseInfrastructure¶

Bases: parsley.aspect.abstractaspect.Aspect

Provides very basic infrastructure functionality like the workflow described in parsley.syncengine.sync.Sync. It is implicitly available and must never be added explicitely to a configuration.

baseinfrastructure_update_directory(ctx, filesystem)¶: Triggers the directory synchronization if the item is a directory.

parsley.aspect.collectinformation module¶

class parsley.aspect.collectinformation.CollectInformation¶

Bases: parsley.aspect.abstractaspect.Aspect

Used for populating the ._filelists_curr lists.

collectinformation_collectinfo(ctx, filesystem)¶: Collects information (type, size, …) for this entry from the filesystem and stores it to _filelists_curr.

parsley.aspect.conflicts module¶

class parsley.aspect.conflicts.DetectTypeConflicts¶

Bases: parsley.aspect.abstractaspect.Aspect

Detects type conflicts between two filesystems (file vs directory, for example). Part of parsley.aspect.defaults.DefaultSync.

detect_type_conflict(ctx, filesystem)¶: Detects a type conflict to master.

class parsley.aspect.conflicts.ResolveConflictsByHint¶

Bases: parsley.aspect.abstractaspect.Aspect

Resolve conflicts together with TrackConflicts. Part of parsley.aspect.defaults.DefaultSync.

resolveconflicts_byhint(ctx, filesystem)¶: Resolves a type conflict by a conflict resolution hint stored in the control files.

class parsley.aspect.conflicts.TrackConflicts¶

Bases: parsley.aspect.abstractaspect.Aspect

Tracks conflict for handling outside of parsley (interactively in a gui for example).

For a conflict in file p/f in a sync task t, a control file conflicts/t/p/f is created. The conflict can be resolved outside of parsley in an arbitrary way by modifying this file in a certain way (you find examples somewhere in the parsley sources).

Part of parsley.aspect.defaults.DefaultSync.

cleanup_conflicts_dir(ctx, filesystem)¶: Cleans up old entries.

prepare_trackconflicts(ctx, filesystem)¶: Prepares stuff.

trackconflicts_trackskipped_skipconflict(ctx, filesystem)¶: Stores information about the conflict for resolving.

parsley.aspect.defaultiteration module¶

class parsley.aspect.defaultiteration.DefaultIteration¶

Bases: parsley.aspect.abstractaspect.Aspect

Lists child elements via the filesystem. Part of parsley.aspect.defaults.DefaultBase.

defaultiteration_listdir(ctx, filesystem)¶: Adds all the child entries to the list (as returned by the underlying filesystem).

parsley.aspect.defaults module¶

class parsley.aspect.defaults.DefaultBase¶

Bases: parsley.aspect.defaults.DefaultBaseBare, parsley.aspect.electmaster.ElectMasterFileByMtime, parsley.aspect.electmaster.ElectMasterLinkByTargetHistory

Mid-basic part of default behavior for a plain synchronization. Part of parsley.aspect.defaults.DefaultSync.

Parameters: function – The acceptor function. it gets the path as parameter and must return True for accepting it.

Warning: This parameter is a string that will be evaluated as Python expression (so usage from config files is easy)! The path argument is available as path. Example: path[2]==’a’. The filesystem is available as fs.

class parsley.aspect.defaults.DefaultBaseBare¶

Bases: parsley.aspect.defaultiteration.DefaultIteration, parsley.aspect.collectinformation.CollectInformation, parsley.aspect.applypathacceptor.ApplyPathAcceptor, parsley.aspect.readmefile.ReadmeFile, parsley.aspect.volumepathbreadcrumb.VolumePathBreadcrumb, parsley.aspect.volumesyncreport.VolumeSyncReport

Very basic part of default behavior for a plain synchronization. Part of parsley.aspect.defaults.DefaultBase.

Parameters: function – The acceptor function. it gets the path as parameter and must return True for accepting it.

Warning: This parameter is a string that will be evaluated as Python expression (so usage from config files is easy)! The path argument is available as path. Example: path[2]==’a’. The filesystem is available as fs.

class parsley.aspect.defaults.DefaultSync¶

Bases: parsley.aspect.defaults.DefaultBase, parsley.aspect.update.DefaultUpdateItems, parsley.aspect.conflicts.DetectTypeConflicts, parsley.aspect.conflicts.TrackConflicts, parsley.aspect.conflicts.ResolveConflictsByHint, parsley.aspect.remove.DetectRemoval, parsley.aspect.remove.DefaultRemoveDirs

Default behavior for a plain synchronization.

Parameters: function – The acceptor function. it gets the path as parameter and must return True for accepting it.

Warning: This parameter is a string that will be evaluated as Python expression (so usage from config files is easy)! The path argument is available as path. Example: path[2]==’a’. The filesystem is available as fs.

class parsley.aspect.defaults.PullAndPurgeSyncSink¶

Bases: parsley.aspect.defaults.DefaultBaseBare, parsley.aspect.update.DefaultUpdateItems, parsley.aspect.remove.CleanupTrashBin

Default behavior for a sink filesystem in a pull-and-purge configuration.

Parameters: function – The acceptor function. it gets the path as parameter and must return True for accepting it.

Warning: This parameter is a string that will be evaluated as Python expression (so usage from config files is easy)! The path argument is available as path. Example: path[2]==’a’. The filesystem is available as fs.

ppsyncsink_electifnowhereelse(ctx, filesystem)¶: Elects the sink filesystem if the item exists here … - by a minimal key, if it is a file, so it gets elected if there are no others - by a maximum key, if it is a directory, so the sync can go deeper

ppsyncsink_renameexistingtonew(ctx, filesystem)¶: Renames an already existing entry to some new name.

class parsley.aspect.defaults.PullAndPurgeSyncSource¶

Bases: parsley.aspect.defaults.DefaultBase

Default behavior for a source filesystem in a pull-and-purge configuration.

Parameters: function – The acceptor function. it gets the path as parameter and must return True for accepting it.

Warning: This parameter is a string that will be evaluated as Python expression (so usage from config files is easy)! The path argument is available as path. Example: path[2]==’a’. The filesystem is available as fs.

ppsyncsource_removefromsource(ctx, filesystem)¶: Removes the file in the source filesystem.

parsley.aspect.electmaster module¶

class parsley.aspect.electmaster.ElectMasterFileByMtime¶

Bases: parsley.aspect.abstractaspect.Aspect

Elects a master file by mtime. Default master election strategy for files. Part of parsley.aspect.defaults.DefaultBase.

electfilebymtime(ctx, filesystem)¶: Returns the mtime of this entry as election key.

class parsley.aspect.electmaster.ElectMasterLinkByTargetHistory¶

Bases: parsley.aspect.abstractaspect.Aspect

Elects a master link by changes in history. Default master election strategy for links. Part of parsley.aspect.defaults.DefaultBase.

electlinkbyparam(ctx, filesystem)¶: Returns 0 if the link remained unchanged since last sync, or 1 otherwise as election key.

parsley.aspect.highlevelcustomization module¶

class parsley.aspect.highlevelcustomization.HighLevelCustomization¶

Bases: parsley.aspect.abstractaspect.Aspect

Executes high level customization handlers. They are loaded directly from the sync tree, whenever a .parsley.custom.py file exists.

class BeforeUpdateEvent(controller, isnew, isdeleted, ischanged, scriptsource)¶

Bases: object

CONTINUE_TOUCHED = 'CONTINUE_TOUCHED'¶

CONTINUE_UNTOUCHED = 'CONTINUE_UNTOUCHED'¶

continue_touched()¶

continue_untouched()¶

exception BeforeUpdateHandlerTerminatedWithoutDecision¶: Bases: parsley.exceptions.ParsleyError

callonbeforeupdate(ctx, filesystem)¶

init(ctx, filesystem)¶

loadcustomizations(ctx, filesystem)¶

unloadcustomizations(ctx, filesystem)¶

parsley.aspect.logging module¶

class parsley.aspect.logging.Logging(*, logcreate='1', logremove='1', logupdate='1', logproblem='1')¶

Bases: parsley.aspect.abstractaspect.Aspect

Switchable logging by situation.

_logcreate(ctx, filesystem)¶: Logs an entry creation (only hooked if this is activated in aspect configuration).

_logproblem(ctx, filesystem)¶: Logs a problem (only hooked if this is activated in aspect configuration).

_logremove(ctx, filesystem)¶: Logs an entry removal (only hooked if this is activated in aspect configuration).

_logupdate(ctx, filesystem)¶: Logs an entry update (only hooked if this is activated in aspect configuration).

parsley.aspect.metadata module¶

class parsley.aspect.metadata.MetadataSynchronization¶

Bases: parsley.aspect.abstractaspect.Aspect

Synchronizes metadata without a shadow storage (typical for the workstations).

metadata_checkagainstshadow(ctx, filesystem)¶: Checks if this entry has a metadata update against the shadow storage. If so, it updates the version numbers (in the in-memory data structure).

metadata_propagatetofilesystem_file(ctx, filesystem)¶: Checks if metadata of the entry differ from the filesystem information and, if so, updates the entry metadata in the filesystem (with exactly what it is now stored in-memory).

class parsley.aspect.metadata.MetadataSynchronizationWithShadow¶

Bases: parsley.aspect.metadata.MetadataSynchronization

Synchronizes metadata with a shadow storage (typical for the file server).

cleanup_shadow(ctx, filesystem)¶: Cleans up orphaned files in shadow metadata storage.

metadata_findhighestshadow(ctx, filesystem)¶: Updates latestmd as stored in ctx if this filesystem has a higher version in shadow.

metadata_init(ctx, filesystem)¶: Some initialization.

metadata_propagatetoshadow_file(ctx, filesystem)¶: Checks if metadata of the entry differ from the shadow storage information and, if so, updates the entry metadata in the shadow storage (with exactly what it is now stored in-memory).

exception parsley.aspect.metadata.SyncMetadataAspectError¶: Bases: parsley.exceptions.ParsleyError

class parsley.aspect.metadata._Metadata(version=0)¶

Bases: object

static get_metadata_from_file(filesystem, path, ignore_removed=False)¶

static get_metadata_from_shadow(filesystem, shadowfilesystem, path)¶

getdata(k, d=None)¶

getkeys()¶

static metadata_differs(md1, md2)¶

putdata(k, v)¶

static remove_shadow(shadowfilesystem, path, isdir)¶

static set_metadata_to_file(filesystem, path, md, ignore_removed=False)¶

static set_metadata_to_shadow(filesystem, shadowfilesystem, path, md)¶

static set_metadataversion_to_file(filesystem, path, md, ignore_removed=False)¶

class parsley.aspect.metadata._MetadataStruct¶: Bases: object

parsley.aspect.monitorfilechanges module¶

class parsley.aspect.monitorfilechanges.MonitorFileChanges¶

Bases: parsley.aspect.abstractaspect.Aspect

Tries to establish a proxy on the other end of an ssh filesystem connection that monitors file changes. If changes appear, a sync is triggered for the next possible moment. It only works on parsley.filesystem.sshfs.SshfsFilesystem’s.

_beforeelectmaster(ctx, filesystem)¶: Remembers which files have been seen, so we can catch just notifications for them (the other ones are not interesting).

_beginsync(ctx, filesystem)¶: Establishes the proxy.

_closesync(ctx, filesystem)¶: Stops the filesseenwriter thread (and more).

filesseenwriterthread(sync, filesystem, notifyonly_storage, filesseen, filesseen_lock)¶

static monitorproxy(sshtarget, port, idfile, proxypath, rvolpath, lastsuccfile, notifyonlylist, asynclogpath, options)¶

parsley.aspect.permissions module¶

class parsley.aspect.permissions.ApplyPermissions(user, group, fileaddperms='', filesubtractperms='', diraddperms='', dirsubtractperms='')¶

Bases: parsley.aspect.abstractaspect.Aspect

Applies the specified file system permissions to all entries processed by parsley. Warning: Do not use this feature in security relevant contexts. It’s not guaranteed that all aspects follow this.

Parameters

user – Either “#uid” or user name of the new owner user.
group – Either “#gid” or group name of the new owner group.
fileaddperms – Numerical mask of permissions that will be added to each synchronized file.
filesubtractperms – Numerical mask of permissions that will be subtracted from each synchronized file.
diraddperms – Numerical mask of permissions that will be added to each synchronized directory.
dirsubtractperms – Numerical mask of permissions that will be subtracted from each synchronized directory.

static _setperms(path, uid, gid, addperms, subtractperms)¶

applypermissions_setdirperms(ctx, filesystem)¶: Sets the file permissions of a dir entry to what is configured in the aspect.

applypermissions_setfileperms(ctx, filesystem)¶: Sets the file permissions of a file entry to what is configured in the aspect.

parsley.aspect.readmefile module¶

class parsley.aspect.readmefile.ReadmeFile¶

Bases: parsley.aspect.abstractaspect.Aspect

Writes a readme file into the on-volume parsley control directory.

readmefile_writereadme(ctx, filesystem)¶: Writes the readme.

parsley.aspect.remove module¶

class parsley.aspect.remove.CleanupTrashBin(trashdelay='7d')¶

Bases: parsley.aspect.abstractaspect.Aspect

Cleans up the trash bin. Part of parsley.aspect.remove.DefaultRemove and parsley.aspect.remove.TrashRemove.

cleanup_init(ctx, filesystem)¶: Some initialization.

cleanup_trashbin(ctx, filesystem)¶: Cleans up files in the trashbin control directory that fulfill certain conditions.

class parsley.aspect.remove.DefaultRemove¶

Bases: parsley.aspect.remove.RemoveOrphanedDirs, parsley.aspect.remove.CleanupTrashBin

Default removal strategy without a trashbin.

defaultremove_file_link(ctx, filesystem)¶: Removes the file or link from the filesystem.

class parsley.aspect.remove.DefaultRemoveDirs¶

Bases: parsley.aspect.abstractaspect.Aspect

Removes directories via the filesystem. Part of parsley.aspect.defaults.DefaultSync.

defaultremovedirs_removedir(ctx, filesystem)¶: Removes a directory (recursively or safely) in the filesystem.

class parsley.aspect.remove.DetectRemoval¶

Bases: parsley.aspect.abstractaspect.Aspect

Detects removed entries. Part of parsley.aspect.defaults.DefaultSync.

static _mtimes_equal(t1, t2, precise)¶

detectremoval_elect(ctx, filesystem)¶: Detects if an entry is to be removed and if so, returns the current time as election key.

class parsley.aspect.remove.RemoveOrphanedDirs¶

Bases: parsley.aspect.abstractaspect.Aspect

Removes empty orphaned directories. Part of parsley.aspect.remove.DefaultRemove and parsley.aspect.remove.TrashRemove.

check_orphaned_dirs(ctx, filesystem)¶

remove_orphaned_dirs(ctx, filesystem)¶: Removes a directory in the filesystem if it is empty and was removed in another filesystem.

class parsley.aspect.remove.TrashRemove(trashdelay='7d')¶

Bases: parsley.aspect.remove.RemoveOrphanedDirs, parsley.aspect.remove.CleanupTrashBin

Removal strategy with a trashbin.

trashremove_file_link(ctx, filesystem)¶: Moves a file or link to the trash bin.

parsley.aspect.revisiontracking module¶

class parsley.aspect.revisiontracking.RevisionTracking(number_unarchived_revisions=3, number_revisions_per_archive=20)¶

Bases: parsley.aspect.abstractaspect.Aspect

Keeps all versions of all files that parsley has seen into a revision storage in the control directory.

Parameters

number_unarchived_revisions – Number of revisions per file to be stored directly, not inside an archive file.
number_revisions_per_archive – Number of revisions per file to be stored in one archive, before the next archive file begins.

static _updatefile_helper(ctx, filesystem, path, number_unarchived_revisions, number_revisions_per_archive)¶

revisiontracking_init(ctx, filesystem)¶: Some initialization.

revisiontracking_store1(ctx, filesystem)¶: Checks if the version with the current mtime of this entry is already stored in the version history, and if not, do so.

revisiontracking_store2(ctx, filesystem)¶: Checks if the version with the current mtime of this entry is already stored in the version history, and if not, do so.

parsley.aspect.update module¶

class parsley.aspect.update.DefaultUpdateItems¶

Bases: parsley.aspect.abstractaspect.Aspect

Default handling for updating non-directory items (i.e. files, links). Part of parsley.aspect.defaults.DefaultSync.

defaultupdateitem_detectandskipupdateconflict(ctx, filesystem)¶: Checks if an entry stays in an update conflict (by mtime/param comparisons) and if so, mark and skip the entry.

defaultupdateitem_skipidentical(ctx, filesystem)¶: Skips an entry if they have identical mtime (or link target, for links) in both filesystems.

defaultupdateitem_skipidentical_aux(ctx, filesystem)¶

defaultupdateitem_update(ctx, filesystem)¶: Updates the entry in the non-master filesystems.

resolveconflicts_cleanupbeforenewdir(ctx, filesystem)¶: Forcefully remove files/links for paths that begin a directory sync (so they must be directories).

parsley.aspect.volumepathbreadcrumb module¶

class parsley.aspect.volumepathbreadcrumb.VolumePathBreadcrumb¶

Bases: parsley.aspect.abstractaspect.Aspect

Writes a file to the data directory that helps finding the sync volumes after runtime.

volumepathbreadcrumb_write(ctx, filesystem)¶: Writes the breadcrumb.

parsley.aspect.volumesyncreport module¶

class parsley.aspect.volumesyncreport.VolumeSyncReport¶

Bases: parsley.aspect.abstractaspect.Aspect

Writes some report data to the volume.

syncvolumereport_write(ctx, filesystem)¶: Writes the report.

Module contents¶

Aspects control the behavior of parsley.syncengine.sync.Sync configurations. Each feature should be encapsulated as one subclass of parsley.aspect.abstractaspect.Aspect that hooks some event handlers into the processing chain (so-called aspect hook methods).

This module also contains some function decorators for usage with aspect hook methods.

☰

⇗