pyndn.sync package

Subpackages

• pyndn.sync.detail package
  • Submodules
  • pyndn.sync.detail.invertible_bloom_lookup_table module
  • pyndn.sync.detail.psync_segment_publisher module
  • pyndn.sync.detail.psync_state module
  • pyndn.sync.detail.psync_user_prefixes module
  • Module contents

Submodules

pyndn.sync.chrono_sync2013 module

This module defines the ChronoSync2013 class which implements the NDN ChronoSync protocol as described in the 2013 paper “Let’s ChronoSync: Decentralized Dataset State Synchronization in Named Data Networking”. http://named-data.net/publications/chronosync . Note: The support for ChronoSync is experimental and the API is not finalized. See the API docs for more detail at http://named-data.net/doc/ndn-ccl-api/chrono-sync2013.html .

class pyndn.sync.chrono_sync2013.ChronoSync2013(onReceivedSyncState, onInitialized, applicationDataPrefix, applicationBroadcastPrefix, sessionNo, face, keyChain, certificateName, syncLifetime, onRegisterFailed)

Bases: object

Create a new ChronoSync2013 to communicate using the given face. Initialize the digest log with a digest of “00” and and empty content. Register the applicationBroadcastPrefix to receive interests for sync state messages and express an interest for the initial root digest “00”. Note: Your application must call processEvents. Since processEvents modifies the internal ChronoSync data structures, your application should make sure that it calls processEvents in the same thread as this constructor (which also modifies the data structures).

Parameters:

• onReceivedSyncState (function object) – When ChronoSync receives a sync state message, this calls onReceivedSyncState(syncStates, isRecovery) where syncStates is the list of SyncState messages and isRecovery is true if this is the initial list of SyncState messages or from a recovery interest. (For example, if isRecovery is true, a chat application would not want to re-display all the associated chat messages.) The callback should send interests to fetch the application data for the sequence numbers in the sync state. NOTE: The library will log any exceptions raised by this callback, but for better error handling the callback should catch and properly handle any exceptions.
• onInitialized (function object) – This calls onInitialized() when the first sync data is received (or the interest times out because there are no other publishers yet). NOTE: The library will log any exceptions raised by this callback, but for better error handling the callback should catch and properly handle any exceptions.
• applicationDataPrefix (Name) – The prefix used by this application instance for application data. For example, “/my/local/prefix/ndnchat4/0K4wChff2v”. This is used when sending a sync message for a new sequence number. In the sync message, this uses applicationDataPrefix.toUri().
• applicationBroadcastPrefix (Name) – The broadcast name prefix including the application name. For example, “/ndn/broadcast/ChronoChat-0.3/ndnchat1”. This makes a copy of the name.
• sessionNo (int) – The session number used with the applicationDataPrefix in sync state messages.
• face (Face) – The Face for calling registerPrefix and expressInterest. The Face object must remain valid for the life of this ChronoSync2013 object.
• keyChain (KeyChain) – To sign a data packet containing a sync state message, this calls keyChain.sign(data, certificateName).
• certificateName (Name) – The certificate name of the key to use for signing a data packet containing a sync state message.
• syncLifetime (float) – The interest lifetime in milliseconds for sending sync interests.
• onRegisterFailed (function object) – If failed to register the prefix to receive interests for the applicationBroadcastPrefix, this calls onRegisterFailed(applicationBroadcastPrefix). NOTE: The library will log any exceptions raised by this callback, but for better error handling the callback should catch and properly handle any exceptions.

class PrefixAndSessionNo(dataPrefixUri, sessionNo)

Bases: object

A PrefixAndSessionNo holds a user’s data prefix and session number (used to return a list from getProducerPrefixes).

getDataPrefix()

Get the application data prefix.

Returns:The application data prefix as a Name URI string. Return type:str

getSessionNo()

Get the session number associated with the application data prefix.

Returns:The session number. Return type:int

class SyncState(dataPrefixUri, sessionNo, sequenceNo, applicationInfo)

Bases: object

A SyncState holds the values of a sync state message which is passed to the onReceivedSyncState callback which was given to the ChronoSyn2013 constructor. Note: this has the same info as the Protobuf class sync_state_pb2.SyncState, but we make a separate class so that we don’t need the Protobuf definition in the ChronoSync API.

getApplicationInfo()

Get the application info which was included when the sender published the next sequence number.

Returns:The applicationInfo Blob. If the sender did not provide any, return an isNull Blob. Return type:Blob

getDataPrefix()

Get the application data prefix for this sync state message.

Returns:The application data prefix as a Name URI string. Return type:str

getSequenceNo()

Get the sequence number for this sync state message.

Returns:The sequence number. Return type:int

getSessionNo()

Get the session number associated with the application data prefix for this sync state message.

Returns:The session number. Return type:int

getProducerPrefixes()

Get a copy of the current list of producer data prefixes, and the associated session number. You can use these in getProducerSequenceNo(). This includes the prefix for this user.

Returns:A copy of the list of each producer prefix and session number. Return type:array of ChronoSync2013.PrefixAndSessionNo

getProducerSequenceNo(dataPrefix, sessionNo)

Get the current sequence number in the digest tree for the given producer dataPrefix and sessionNo.

Parameters:

• dataPrefix (std) – The producer data prefix as a Name URI string.
• sessionNo (int) – The producer session number.

Returns:

The current producer sequence number, or -1 if the producer namePrefix and sessionNo are not in the digest tree.

Return type:

int

getSequenceNo()

Get the sequence number of the latest data published by this application instance.

Returns:The sequence number. Return type:int

publishNextSequenceNo(applicationInfo=None)

Increment the sequence number, create a sync message with the new sequence number and publish a data packet where the name is the applicationBroadcastPrefix + the root digest of the current digest tree. Then add the sync message to the digest tree and digest log which creates a new root digest. Finally, express an interest for the next sync update with the name applicationBroadcastPrefix + the new root digest. After this, your application should publish the content for the new sequence number. You can get the new sequence number with getSequenceNo(). Note: Your application must call processEvents. Since processEvents modifies the internal ChronoSync data structures, your application should make sure that it calls processEvents in the same thread as publishNextSequenceNo() (which also modifies the data structures).

Parameters:applicationInfo (Blob) – (optional) This appends applicationInfo to the content of the sync messages. This same info is provided to the receiving application in the SyncState state object provided to the onReceivedSyncState callback.

shutdown()

Unregister callbacks so that this does not respond to interests anymore. If you will discard this ChronoSync2013 object while your application is still running, you should call shutdown() first. After calling this, you should not call publishNextSequenceNo() again since the behavior will be undefined. Note: Because this modifies internal ChronoSync data structures, your application should make sure that it calls processEvents in the same thread as shutdown() (which also modifies the data structures).

pyndn.sync.digest_tree module

This module defines the DigestTree class which maintains a digest tree for ChronoSync.

class pyndn.sync.digest_tree.DigestTree

Bases: object

class Node(dataPrefix, sessionNo, sequenceNo)

Bases: object

Create a new DigestTree.Node with the given fields and compute the digest.

Parameters:

• dataPrefix (str) – The data prefix. In Python3, this is encoded as UTF-8 to digest.
• sessionNo (int) – The session number.
• sequenceNo (int) – The sequence number.

getDataPrefix()

Get the data prefix.

Returns:The data prefix. Return type:str

getDigest()

Get the digest.

Returns:The digest as a hex string. Return type:str

getSequenceNo()

Get the sequence number.

Returns:The sequence number. Return type:int

getSessionNo()

Get the session number.

Returns:The session number. Return type:int

lessThan(node2)

Compare this Node with node2 first comparing _dataPrefix then _sessionNo.

Parameters:node2 (DigestTree.Node) – The other Node to compare. Returns:True if this node is less than node2. Return type:bool

setSequenceNo(sequenceNo)

Set the sequence number and recompute the digest.

Parameters:sequenceNo (int) – The new sequence number.

find(dataPrefix, sessionNo)

get(i)

getRoot()

Get the root digest.

Returns:The root digest as a hex string. Return type:str

size()

update(dataPrefix, sessionNo, sequenceNo)

Update the digest tree and recompute the root digest. If the combination of dataPrefix and sessionNo already exists in the tree then update its sequenceNo (only if the given sequenceNo is newer), otherwise add a new node.

Parameters:

• dataPrefix (str) – The data prefix. In Python3, this is encoded as UTF-8 to digest.
• sequenceNo (int) – The sequence number.
• sessionNo (int) – The session number.

Returns:

True if the digest tree is updated, False if not (because the given sequenceNo is not newer than the existing sequence number).

Return type:

bool

pyndn.sync.full_psync2017 module

This modules defines the FullPSync2017 class which extends PSyncProducerBase to implement the full sync logic of PSync to synchronize with other nodes, where all nodes want to sync all the names. The application should call publishName whenever it wants to let consumers know that a new name is available. Currently, fetching and publishing the data given by the announced name needs to be handled by the application. The Full PSync protocol is described in Section G “Full-Data Synchronization” of: https://named-data.net/wp-content/uploads/2017/05/scalable_name-based_data_synchronization.pdf (Note: In the PSync library, this class is called FullProducerArbitrary. But because the class actually handles both producing and consuming, we omit “producer” in the name to avoid confusion.)

class pyndn.sync.full_psync2017.FullPSync2017(expectedNEntries, face, syncPrefix, onNamesUpdate, keyChain, syncInterestLifetime=1000.0, syncReplyFreshnessPeriod=1000.0, signingInfo=<pyndn.security.signing_info.SigningInfo object>, canAddToSyncData=None, canAddReceivedName=None)

Bases: pyndn.sync.psync_producer_base.PSyncProducerBase

DEFAULT_SYNC_INTEREST_LIFETIME = 1000.0

DEFAULT_SYNC_REPLY_FRESHNESS_PERIOD = 1000.0

Create a FullPSync2017.

Parameters:

• expectedNEntries (int) – The expected number of entries in the IBLT.
• face (Face) – The application’s Face.
• syncPrefix (Name) – The prefix Name of the sync group, which is copied.
• onNamesUpdate (function object) – When there are new names, this calls onNamesUpdate(names) where names is a list of Names. However, see the canAddReceivedName callback which can control which names are added. NOTE: The library will log any exceptions thrown by this callback, but for better error handling the callback should catch and properly handle any exceptions.
• keyChain (KeyChain) – The KeyChain for signing Data packets.
• syncInterestLifetime (float) – (optional) The Interest lifetime for the sync Interests, in milliseconds. If omitted or None, use DEFAULT_SYNC_INTEREST_LIFETIME.
• syncReplyFreshnessPeriod (float) – (optional) The freshness period of the sync Data packet, in milliseconds. If omitted or None, use DEFAULT_SYNC_REPLY_FRESHNESS_PERIOD.
• signingInfo (SigningInfo) – (optional) The SigningInfo for signing Data packets, which is copied. If omitted or None, use the default SigningInfo().
• canAddToSyncData (function object) – (optional) When a new IBLT is received in a sync Interest, this calls canAddToSyncData(name, negative) where Name is the candidate Name to add to the response Data packet of Names, and negative is the set of names that the other’s user’s Name set, but not in our own Name set. If the callback returns False, then this does not report the Name to the other user. However, if canAddToSyncData is omitted or None, then each name is reported.
• canAddReceivedName (function object) – (optional) When new names are received, this calls canAddReceivedName(name) for each name. If the callback returns False, then this does not add to the IBLT or report to the application with onNamesUpdate. However, if canAddReceivedName is omitted or None, then each name is added.

publishName(name)

Publish the Name to inform the others. However, if the Name has already been published, do nothing.

Parameters:name (Name) – The Name to publish.

removeName(name)

Remove the Name from the IBLT so that it won’t be announced to other users.

Parameters:name (Name) – The Name to remove.

pyndn.sync.full_psync2017_with_users module

This module defines the FullPSync2017WithUsers class which uses FullPSync2017 to implement the full sync logic of PSync to synchronize with other nodes, where all nodes want to sync the sequence number of all users based on their user prefix. The application should call publishName whenever it wants to let consumers know that new data with a new sequence number is available for the user prefix. Multiple user prefixes can be added by using addUserNode. Currently, fetching and publishing the data (named by the user prefix plus the sequence number) needs to be handled by the application. See FullPSync2017 for details on the Full PSync protocol. The Full PSync protocol is described in Section G “Full-Data Synchronization” of: https://named-data.net/wp-content/uploads/2017/05/scalable_name-based_data_synchronization.pdf (Note: In the PSync library, this class is called FullProducer. But because the class actually handles both producing and consuming, we omit “producer” in the name to avoid confusion.)

class pyndn.sync.full_psync2017_with_users.FullPSync2017WithUsers(expectedNEntries, face, syncPrefix, userPrefix, onUpdate, keyChain, syncInterestLifetime=1000.0, syncReplyFreshnessPeriod=1000.0, signingInfo=<pyndn.security.signing_info.SigningInfo object>)

Bases: object

Create a FullPSync2017WithUsers.

Parameters:

• expectedNEntries (int) – The expected number of entries in the IBLT.
• face (Face) – The application’s Face.
• syncPrefix (Name) – The prefix Name of the sync group, which is copied.
• userPrefix (Name) – The prefix Name of the first user in the group, which is copied. However, if this Name is None or empty, it is not added and you must call addUserNode.
• onUpdate (function object) – When there is new data, this calls onUdate(updates) where updates is a list of PSyncMissingDataInfo. NOTE: The library will log any exceptions thrown by this callback, but for better error handling the callback should catch and properly handle any exceptions.
• keyChain (KeyChain) – The KeyChain for signing Data packets.
• syncInterestLifetime (float) – (optional) The Interest lifetime for the sync Interests, in milliseconds. If omitted or None, use FullPSync2017.DEFAULT_SYNC_INTEREST_LIFETIME.
• syncReplyFreshnessPeriod (float) – (optional) The freshness period of the sync Data packet, in milliseconds. If omitted or None, use FullPSync2017.DEFAULT_SYNC_REPLY_FRESHNESS_PERIOD.
• signingInfo (SigningInfo) – (optional) The SigningInfo for signing Data packets, which is copied. If omitted or None, use the default SigningInfo().

addUserNode(prefix)

Add a user node for synchronization based on the prefix Name, and initialize the sequence number to zero. However, if the prefix Name already exists, then do nothing and return false. This does not add sequence number zero to the IBLT because, if a large number of user nodes are added, then decoding the difference between our own IBLT and the other IBLT will not be possible.

Parameters:prefix (Name) – The prefix Name of the user node to be added. Returns:True if the user node with the prefix Name was added, False if the prefix Name already exists.

getSequenceNo(prefix)

Return the current sequence number of the given user prefix.

Parameters:prefix (Name) – The user prefix for the sequence number. Returns:The sequence number for the user prefix, or -1 if not found. Return type:int

publishName(prefix, sequenceNo=-1)

Publish the sequence number for the prefix Name to inform the others. (addUserNode needs to be called before this to add the prefix, if it was not already added via the constructor.)

Parameters:

• prefix (Name) – the prefix Name to be updated.
• sequenceNo (int) – (optional) The sequence number of the user prefix to be set in the IBLT. However, if sequenceNo is omitted or -1, then the existing sequence number is incremented by 1.

removeUserNode(prefix)

Remove the user node from the synchronization. This erases the prefix from the IBLT and other tables.

Parameters:prefix (Name) – The prefix Name of the user node to be removed. If there is no user node with this prefix, do nothing.

pyndn.sync.psync_missing_data_info module

class pyndn.sync.psync_missing_data_info.PSyncMissingDataInfo(prefix, lowSequenceNo, highSequenceNo)

Bases: object

pyndn.sync.psync_producer_base module

This module defines the PSyncProducerBase class which is a base class for PsyncPartialProducer and FullPSync2017.

class pyndn.sync.psync_producer_base.PSyncProducerBase(expectedNEntries, syncPrefix, syncReplyFreshnessPeriod)

Bases: object

Create a PSyncProducerBase.

Parameters:expectedNEntries (int) – The expected number of entries in the IBLT.

:param Name syncPrefix The prefix Name of the sync group, which is copied. :param float syncReplyFreshnessPeriod: The freshness period of the sync Data

packet, in milliseconds.

insertIntoIblt(name)

Insert the URI of the name into the _iblt, and update _nameToHash and _hashToName.

Parameters:name (Name) – The Name to insert.

static onRegisterFailed(prefix)

This is called when registerPrefix fails to log an error message.

removeFromIblt(name)

If the Name is in _nameToHash, then remove the hash from the _iblt, _nameToHash and _hashToName. However, if the Name is not in _nameToHash then do nothing.

Parameters:name (Name) – The Name to remove.

pyndn.sync.sync_state_pb2 module

Module contents