parsley.syncengine package

Submodules

parsley.syncengine.benchmark module

Time measurement of sync runs.

class parsley.syncengine.benchmark.BenchmarkRun(syncruntime)

Bases: object

Collects benchmark data for a sync run.

This collection is populated with data by the engine during execution and stored afterwards, so external tools can inspect this data and get detailed ideas about the runtime performance.

Parameters

syncruntime (parsley.runtime.runtime.RuntimeData) –

beginsync()
Return type

None

beginsyncevent(syncevent, path)
Parameters

  • syncevent (str) –

  • path (str) –

Return type

None

beginsynceventhandler(handler, filesystem)
Parameters
Return type

None

endsync()
Return type

None

endsyncevent()
Return type

None

endsynceventhandler()
Return type

None

get_report()
version = 1

parsley.syncengine.common module

Boring common data structures.

class parsley.syncengine.common.EntryType

Bases: object

A type of an entry (sometimes the word ‘file’ is wrongly used instead of ‘entry’) in the filesystem.

Directory = 'D'

Directory

File = 'F'

Regular file

Symlink

parsley.syncengine.common.SKIP = <object object>

A flag that means skipping parts of execution. It is a general purpose signalling object. It might be interpreted from different parties at different places in a different way. Used e.g. in master election as a special master for skipping the synchronization process (read parsley.syncengine.syncruntime.SyncEventRuntime.skip_master_filesystem_promotion) and in the update process for signalling skipping due to lately found conflicts (read parsley.aspect.execute_only_if_not_update_set_skipped).

exception parsley.syncengine.common.SyncConfigurationError

Bases: parsley.syncengine.common.SyncError

Errors while processing the sync configuration.

exception parsley.syncengine.common.SyncError

Bases: parsley.exceptions.ParsleyError

Errors happening in the sync engine.

class parsley.syncengine.common.SyncEvent

Bases: object

Names of well known synchronization events.

BeginSync = 'BeginSync'

Occurs once at the very beginning of a synchronization task execution.

CloseSync = 'CloseSync'

Occurs after EndSync, even when a fatal error occurred, i.e. typically an unhandled exception. This event is not used for typical stuff but only for infrastructures.

EndSync = 'EndSync'

Occurs once at the very end of a synchronization task execution.

LogCreate = 'LogCreate'

Logging.

LogProblem = 'LogProblem'

Logging.

LogRemove = 'LogRemove'

Logging.

LogUpdate = 'LogUpdate'

Logging.

UpdateDir_AfterUpdate = 'UpdateDir_AfterUpdate'

Occurs in updating directories, when the directory was updated completely. Use this state for executing code after a directory was synchronized, e.g. for bookkeeping.

UpdateDir_ListDir = 'UpdateDir_ListDir'

Occurs in updating directories, when the list of entry names must be fetched. Use this state for listing the directory. The code must call ctx.entrylist.update with a list of entry names in order to participate to listing the directory.

UpdateDir_Prepare = 'UpdateDir_Prepare'

Occurs in updating directories, when the preparation phase takes places, but before the directory is listed. Use this state for preparing directory synchronization or for some bookkeeping.

UpdateItem_AfterUpdate = 'UpdateItem_AfterUpdate'

Occurs in updating items, when the actual update phase is done. Use this state for executing code after the update took place, e.g. for some bookkeeping.

UpdateItem_BeforeElectMaster = 'UpdateItem_BeforeElectMaster'

Occurs in updating items, when very early initialization can happen, before the actual master election phase begins. Use this state for executing code at the very beginning of item synchronization, even before the master election takes place, e.g. for some bookkeeping.

UpdateItem_CheckConflicts = 'UpdateItem_CheckConflicts'

Occurs in updating items, when a master filesystem is just elected, for checking for conflicts. Hook methods here which checks for conflicts between the master filesystem and the filesystems it is hooked for. Signal conflicts by calling ctx.add_conflict.

UpdateItem_ElectMaster = 'UpdateItem_ElectMaster'

Occurs in updating items, when the master election phase began. Elect a master filesystem here in order to control which filesystems gets updated from which one. Use parsley.syncengine.syncruntime.SyncEventRuntime.promote_master_filesystem on ctx in order to elect a master filesystem or parsley.syncengine.syncruntime.SyncEventRuntime.skip_master_filesystem_promotion for ultimatively vote for skipping the synchronization of this entry completely. In all subsequent steps, the elected master filesystem is available in ctx.masterfs.

UpdateItem_ResolveConflicts = 'UpdateItem_ResolveConflicts'

Occurs in updating items, when conflicts occurred, in order to solve them. If this phase cannot resolve the conflicts, the usual update is skipped. Use this state for executing code that can resolve conflicts detected in SyncEvent.UpdateItem_CheckConflicts. For signalling that a conflict is resolved, call ctx.remove_conflict. It may decide to set a new ctx.masterfs for a new master filesystem.

UpdateItem_SkippedDueConflicts = 'UpdateItem_SkippedDueConflicts'

Occurs in updating items, when conflicts were not resolved and the item is skipped due to that. Use this state for executing code that handles unresolved conflicts that led to skipping this item.

UpdateItem_Update_ExistsInMaster = 'UpdateItem_Update_ExistsInMaster'

Occurs in updating items, when the actual update must take place and there exists an item in the master filesystem (typical update scenario). Use this state for actually transfer content from the master to the filesystem on which the hook runs.

UpdateItem_Update_NotExistsInMaster = 'UpdateItem_Update_NotExistsInMaster'

Occurs in updating items, when the actual update must take place and there does not exist an item in the master filesystem (typical removal scenario). Use this state for actually remove content from the filesystem on which the hook runs.

UpdateItem_Update_Prepare = 'UpdateItem_Update_Prepare'

Occurs after conflict resolution took place and the actual update phase is just before beginning.

exception parsley.syncengine.common.SyncExecutionError

Bases: parsley.syncengine.common.SyncError

Errors while executing the synchronization task.

parsley.syncengine.core module

The engine which abstractly executes arbitrary synchronization tasks.

class parsley.syncengine.core.CmdLine(args)

Bases: object

Parser for command line arguments.

Parameters

args (List[str]) –

configfile_or_default()
Return type

str

datadir_or_default()
Return type

str

class parsley.syncengine.core.Core

Bases: object

The Parsley engine. It is the lowest (or higest, if you want) layer, which coordinates all the synchronization work.

class CoreRuntime(sync)

Bases: object

Parameters

sync (parsley.syncengine.sync.Sync) –

class _Myclass

Bases: object

static execute(syncs, datadir, syncname=None, logging=None, throwexceptions=False)

Executes the selected synchronization tasks.

Parameters

  • syncs (List[parsley.syncengine.sync.Sync]) – a list of synchronization tasks existing in your configuration.

  • datadir (str) – which directory to use for persistent control data (file lists, …).

  • syncname (Optional[str]) – which sync task shall be executed (None: all).

  • logging (Optional[List[parsley.logger.Logger]]) – list of loggers of logging output.

  • throwexceptions (bool) – If exceptions should be raised in error case.

Return type

int

parsley.syncengine.core._createemptyconfigifneeded(cmdline)
Parameters

cmdline (parsley.syncengine.core.CmdLine) –

Return type

None

parsley.syncengine.core.main(cmdline=None, callexit=True, environment=None)

Main method for execution from command line.

Parameters

  • cmdline (Optional[List[str]]) –

  • callexit (bool) –

  • environment (Optional[Dict[str, str]]) –

Return type

int

parsley.syncengine.entrylist module

Internal in-memory list of entries for one filesystem with (de-)serialization features.

class parsley.syncengine.entrylist.CombinedEntryList(first, second)

Bases: object

Parameters
addtag(path, tag)
exists(path)
foundfile(path, ftype, size, mtime, param, last_synced_mtime=None)
getftype(path)
getlastsyncedmtime(path)
getmtime(path)
getparam(path)
getsize(path)
gettags(path)
notfoundfile(path)
removefile(path)
setlastsyncedmtime(path, mtime)
updatefile(path, ftype, size, mtime, param)
class parsley.syncengine.entrylist.EntryList

Bases: object

addtag(path, tag)
Parameters

  • path (str) –

  • tag (str) –

Return type

None

exists(path)
Parameters

path (str) –

Return type

bool

foundfile(path, ftype, size, mtime, param, last_synced_mtime=None)
Parameters

  • path (str) –

  • ftype (str) –

  • size (int) –

  • mtime (datetime.datetime) –

  • param (Optional[str]) –

  • last_synced_mtime (Optional[datetime.datetime]) –

Return type

None

getftype(path)
Parameters

path (str) –

Return type

str

getlastsyncedmtime(path)
Parameters

path (str) –

Return type

Optional[datetime.datetime]

getmtime(path)
Parameters

path (str) –

Return type

datetime.datetime

getparam(path)
Parameters

path (str) –

Return type

Optional[str]

getsize(path)
Parameters

path (str) –

Return type

int

gettags(path)
Parameters

path (str) –

Return type

List[str]

notfoundfile(path)
Parameters

path (str) –

Return type

None

read(cnt)
Parameters

cnt (bytes) –

Return type

None

removefile(path)
Parameters

path (str) –

Return type

None

save()
Return type

bytes

setlastsyncedmtime(path, mtime)
Parameters

  • path (str) –

  • mtime (datetime.datetime) –

Return type

None

updatefile(path, ftype, size, mtime, param)
Parameters

  • path (str) –

  • ftype (str) –

  • size (int) –

  • mtime (datetime.datetime) –

  • param (Optional[str]) –

Return type

None

parsley.syncengine.sync module

Find the synchronization implementation in Sync and many other stuff in submodules.

class parsley.syncengine.sync.Sync(*objlist, name, interval='0s', warn_after='7d', warn_interval='7d', benchmark_data_size=30)

Bases: object

The synchronization implementation.

Parsley configuration files typically create some instances of this class.

The entire synchronization work takes place by all hook methods, hooked by all specified aspects. Read Sync.executeevent() for more details.

Parameters
class EventHandler(event, func, funcargs)

Bases: object

A data structure that represents one hooked instance of a hook method (i.e. bound to a certain funcargs, including the filesystem).

Parameters

  • event (str) –

  • func (Callable) –

  • funcargs (Dict[str, Optional[Any]]) –

_execute_helper(runtime, preparations)
Parameters
Return type

None

_get_eventhandlers(event)

Returns a list of event handlers for an event.

Parameters

event (str) –

Return type

List[parsley.syncengine.sync.Sync.EventHandler]

_hook_aspect(aspect, fss)

Request to hook program logic into certain events. Used internally. See parsley.aspect.abstractaspect.hook about how to hook custom synchronization logic.

Parameters
Return type

None

execute(runtime, force=False)

Used by the parsley engine for executing the sync run.

Parameters
Return type

bool

executeevent(event, eventcontext, **eventparams)

Executes an event by name. This means executing the aspect hooks and enforcing the ordering rules.

The technical realization of the workflow bases on an event-based execution engine. The events (in parsley.syncengine.common.SyncEvent) define the workflow stages of the engine. The engine essentially executes aspect hook methods (see parsley.aspect.abstractaspect.Aspect) on filesystems (see parsley.filesystem.abstractfilesystem.Filesystem) for each event.

The entire synchronization behavior is implemented by a horde of builtin aspects and optionally also custom ones. To be exact: The configuration for a synchronization task specifies the synchronization behavior by specifying a list of aspect instances. The list of aspects can be specified and configured for each filesystem individually and globally (which is equal to assigning it to each filesystem).

Whenever an aspect is assigned to one filesystem by that means, it hooks some methods into some events for that filesystem.

The order of the aspect hook executions is controlled by the symbol list after, provides and before. Each is a list of string symbols, e.g. [“foo”, “bar”, “baz”] (you can often also write one comma-separated string like “foo,bar,baz”). The order of execution is chosen in a way it strictly follows this rules:

  • A hook method is never executed when it contains one symbol in after that also exists in a provides list of any hook that is not yet executed (use it for after-dependencies; the typical ones).

  • A hook method is never executed when it contains one symbol in provides that also exists in a before list of any hook that is not yet executed (use it for before-dependencies).

Example: A hook with after=””,provides=”foo”,before=”” will be executed before a one with after=”foo”,provides=”bar”,before=”” and after a one with after=””,provides=”baz”,before=”foo”.

Specifying those three lists for each aspect hook method implies the execution order that must be used whenever certain hook methods depend on results or actions of other hook methods.

@note For each hook, the function name itself is always implicitly added to provides by the engine.

Read the manual for a high-level description of the workflow.

For advanced usage in special situations:

  • A symbol in after and before can begin with a *. If so, it does not raise a validation error if this symbol is not available at all.

Parameters

  • event (str) – The event to execute. See parsley.syncengine.common.SyncEvent.

  • eventcontext (parsley.syncengine.syncruntime.SyncEventRuntime) – The parsley.syncengine.syncruntime.SyncEventRuntime control interface for this execution.

  • eventparams (str) – Additional keyword parameters are added to eventcontext (convenience).

Return type

None

sync_directory(path)

Executes the sub workflow for synchronizing one directory.

Parameters

path (str) –

Return type

None

parsley.syncengine.syncruntime module

Runtime additions for a sync event execution.

class parsley.syncengine.syncruntime.SyncEventRuntime(path)

Bases: object

Control interface for hook methods in parsley.aspect.abstractaspect.Aspect implementations.

Also holds all kinds of data for one run of parsley.syncengine.sync.Sync.sync_directory.

Dynamically extends parsley.syncengine.syncruntime.SyncRuntime.

Parameters

path (str) –

add_conflict(fs, comment=None)

Adds a new conflict.

Parameters
Return type

None

get_conflicts()

Returns the list of all recognized conflicts.

Return type

List[Tuple[parsley.filesystem.abstractfilesystem.Filesystem, Optional[str]]]

had_conflicts()

Returns if there are or were any conflicts.

Return type

bool

is_update_marked_bad()

Returns if the update is marked as bad.

Return type

bool

is_update_set_skipped()

Returns if the update process is markes to skip (by skip_update).

Return type

bool

mark_update_bad()

Marks the current update as bad.

Return type

None

promote_master_filesystem(fs, key)

Elects a master filesystem.

Parameters
Return type

None

remove_conflict(i)

Removes a conflict by index.

Parameters

i (int) –

Return type

None

skip_master_filesystem_promotion()

Elects ‘SKIP’ filesystem with highest key. This will lead to skipping the current item.

Return type

None

skip_update()

Marks the update process to skip.

Return type

None

class parsley.syncengine.syncruntime.SyncRuntime

Bases: object

Control interface for sync executions.

Dynamically extends parsley.runtime.runtime.RuntimeData.

_read_filelist(fs)
Parameters

fs (parsley.filesystem.abstractfilesystem.Filesystem) –

Return type

None

_syncruntime_init()
Return type

None

_verify_filelists_cookies()
Return type

None

create_directory(fs, path)

Creates a directory in a filesystem. This is a high-level function that automatically handles the parsley bookkeeping and have some safeguards.

Parameters
Return type

None

Creates a link in a filesystem. This is a high-level function that automatically handles the parsley bookkeeping and have some safeguards.

Parameters
Return type

None

delete_directory(fs, path, *, recursive=False)

Deletes a directory in a filesystem. This is a high-level function that automatically handles the parsley bookkeeping and have some safeguards.

Parameters
Return type

None

delete_file(fs, path)

Deletes a file in a filesystem. This is a high-level function that automatically handles the parsley bookkeeping and have some safeguards.

Parameters
Return type

None

delete_item(fs, path)

Deletes an item (file, directory, …) in a filesystem. This is a high-level function that automatically handles the parsley bookkeeping and have some safeguards.

Parameters
Return type

None

Deletes a link in a filesystem. This is a high-level function that automatically handles the parsley bookkeeping and have some safeguards.

Parameters
Return type

None

getfscookie(fs)

Gets the cookie of a filesystem (for checking if it was completely purged/replaced between sync runs)

Parameters

fs (parsley.filesystem.abstractfilesystem.Filesystem) –

Return type

str

getinfo_current_exists(fs, path)

Gets the info from the parsley bookkeeping, if an entry with given path current exists.

Parameters
Return type

bool

getinfo_current_ftype(fs, path)

Gets the info from the parsley bookkeeping, which type an entry with given path has currently. The type is one of parsley.syncengine.common.EntryType.

Parameters
Return type

str

getinfo_current_gettags(fs, path)

Gets the tags this entry got in the last run. See also setinfo_current_addtag().

Parameters
Return type

List[str]

getinfo_current_mtime(fs, path)

Gets the info from the parsley bookkeeping, which modification time an entry with given path has currently.

Parameters
Return type

datetime.datetime

getinfo_current_param(fs, path)

Gets the info from the parsley bookkeeping, which param an entry with given path has currently. The param is used for storing the target path of a symlink, but is designed to be a general purpose field.

Parameters
Return type

str

getinfo_current_size(fs, path)

Gets the info from the parsley bookkeeping, which size an entry with given path has currently.

Parameters
Return type

int

getinfo_lastrun_exists(fs, path)

Gets the info from the parsley bookkeeping, if an entry with given path existed after last run.

Parameters
Return type

bool

getinfo_lastrun_ftype(fs, path)

Gets the info from the parsley bookkeeping, which type an entry with given path had after last run. The type is one of parsley.syncengine.common.EntryType.

Parameters
Return type

str

getinfo_lastrun_gettags(fs, path)

Gets the tags this entry currently got. See also setinfo_current_addtag().

Parameters
Return type

List[str]

getinfo_lastrun_mtime(fs, path)

Gets the info from the parsley bookkeeping, which modification time an entry with given path had after last run.

Parameters
Return type

datetime.datetime

getinfo_lastrun_param(fs, path)

Gets the info from the parsley bookkeeping, which param an entry with given path had after last run. The param is used for storing the target path of a symlink, but is designed to be a general purpose field.

Parameters
Return type

str

getinfo_lastrun_size(fs, path)

Gets the info from the parsley bookkeeping, which size an entry with given path had after last run.

Parameters
Return type

int

logcreate(fs, subject, verb='created', comment='', symbol='+', severity=3)

Logs events around item creation.

Parameters
Return type

None

logproblem(fs, subject, verb='has problems', comment='', symbol='E', severity=6)

Logs events around problems.

Parameters
Return type

None

logremove(fs, subject, verb='removed', comment='', symbol='-', severity=5)

Logs events around item removal.

Parameters
Return type

None

logupdate(fs, subject, verb='updated', comment='', symbol='*', severity=4)

Logs events around updates of exiting items.

Parameters
Return type

None

setinfo_current_addtag(fs, path, tag)

Adds a tag to this entry. Tags are strings that can be set and retrieved from different places in order to exchange state information. For example, the engine sets the tag ‘S+’ to an entry after successful synchronization.

Parameters
Return type

None

setinfo_removefile(fs, path)

Mark a file as removed for a given path in the parsley bookkeeping.

Parameters
Return type

None

setinfo_updatefile(fs, path, ftype, size=None, mtime=None, param=None)

Updates the info stored for a given path in the parsley bookkeeping.

Parameters
Return type

None

write_file(fs, source, path, *, verifier=<function SyncRuntime.<lambda>>, forcedupdate=False)

Creates a file in a filesystem with some content. This is a high-level function that automatically handles the parsley bookkeeping and have some safeguards.

Parameters
Return type

Tuple[datetime.datetime, int]

Module contents

Find the synchronization implementation in parsley.syncengine.sync.Sync and many other stuff in submodules.