13. Module Reference¶
This chapter contains all the documentation auto-generated from the source code. It is probably not terribly useful for reading through, but may be useful as a searchable reference.
13.1. piwheels.master¶
Defines the PiWheelsMaster
class. An instance of this is the
entry-point for the piw-master script.
- class piwheels.master.PiWheelsMaster[source]¶
This is the main class for the piw-master script. It spawns various worker threads, then spends its time communicating with any attached monitor applications (see piw-monitor) and build slaves (see piw-slave).
- broadcast_status()[source]¶
Publish messages from the internal status queue to the external status queue, in case any monitors are attached.
- static configure_parser()[source]¶
Construct the command line parser for piw-master with its many options (this method only exists to simplify the main method).
- do_hello()[source]¶
Handler for the HELLO message; this indicates a new monitor has been attached and would like the master’s HELLO message and all the build slave’s HELLO messages replayed to it.
- do_kill(slave_id)[source]¶
Handler for the KILL message; this tells the specified build slave (or all slaves and the master if slave_id is
None
) to terminate.
- do_skip(slave_id)[source]¶
Handler for the SKIP message; this tells the specified build slave to skip its current build next time it contacts the master.
- do_sleep(slave_id)[source]¶
Handler for the SLEEP message; this tells the specified build slave (or all slaves and the master if slave_id is
None
) to pause their operations.
- do_wake(slave_id)[source]¶
Handler for the WAKE message; this tells the specified build slave (or all slaves and the master if slave_id is
None
) to resume their operations.
- main_loop(systemd)[source]¶
This is the main loop of the piw-master script. It receives messages from the internal status queue and forwards them onto the external status queue (for any piw-monitor scripts that are attached). It also retrieves any messages sent to the control queue and dispatches them to a handler.
- piwheels.master.sig_term(signo, stack_frame)[source]¶
Handler for the SIGTERM signal; raises
SystemExit
which will cause thePiWheelsMaster.main_loop()
method to terminate.
13.2. piwheels.master.db¶
This module defines the low level database API, Database
. This is a
simple core SQLAlchemy affair which runs trivial queries against the PostgreSQL
database. All the serious logic is defined within views in the database
itself.
- class piwheels.master.db.Database(dsn)[source]¶
PiWheels database connection class
- add_new_package(package, skip='', description='')[source]¶
Insert a new package record into the database. Returns True if the row was inserted successfully, or False if a key violation occurred.
- add_new_package_version(package, version, released=None, skip='')[source]¶
Insert a new package version record into the database. Returns True if the row was inserted successfully, or False if a key violation occurred.
- add_package_name(package, name, seen)[source]¶
Add a new package alias or update the last seen timestamp.
- delete_build(package, version)[source]¶
Remove all builds for the specified package and version, along with all files records.
- delete_version(package, version)[source]¶
Remove the specified version of the specified package, along with all builds and files.
- get_build_abis(exclude_skipped=False)[source]¶
Return a set of ABIs. If exclude_skipped is
False
, return all ABIs from the build_abis table, otherwise return only active ABIs (not skipped).
- get_build_queue(limit=1000)[source]¶
Returns a mapping of ABI tags to an ordered list of up to limit package version tuples which currently need building for that ABI.
- get_package_aliases(package)[source]¶
Retrieve all aliases for package (not including the canonical name itself).
- get_package_files(package)[source]¶
Returns a mapping of filenames to file hashes; this is all the data required to build the simple index.html for the specified package.
- get_project_data(package)[source]¶
Returns all details required to build the project page of the specified package.
- get_search_index()[source]¶
Return a mapping of all packages to their download count for the last month. This is used to construct the searchable package index.
- get_versions_deleted(package)[source]¶
Return any versions of package which have been marked for deletion.
- load_rewrites_pending()[source]¶
Loads the rewrites-pending queue (the internal state of
TheSecretary
) from the database.
- log_download(download)[source]¶
Log a download in the database, including data derived from JSON in pip’s user-agent.
- log_search(search)[source]¶
Log a search in the database, including data derived from JSON in pip’s user-agent.
- save_rewrites_pending(queue)[source]¶
Save the rewrites-pending queue (the internal state of
TheSecretary
) in the database. The queue parameter is expected to be a list ofRewritePendingRow
tuples.
- set_package_description(package, description)[source]¶
Update the description for package in the packages table.
- skip_package(package, reason)[source]¶
Mark a package with a reason to prevent future builds of all versions (and all future versions).
- skip_package_version(package, version, reason)[source]¶
Mark a version of a package with a reason to prevent future build attempts.
- test_package(package)[source]¶
Check whether package already exists in the database. Returns a boolean.
- test_package_version(package, version)[source]¶
Check whether version of package already exists in the database. Returns a boolean.
13.3. piwheels.master.cloud_gazer¶
Defines the CloudGazer
task; see class for more details.
- class piwheels.master.cloud_gazer.CloudGazer(config)[source]¶
This task scrapes PyPI for the list of available packages, and the versions of those packages. This information is written into the backend database for
TheArchitect
to use.
13.4. piwheels.master.the_oracle¶
Defines TheOracle
task and the DbClient
RPC class for talking
to it.
- class piwheels.master.the_oracle.TheOracle(config)[source]¶
This task provides an RPC-like interface to the database; it handles requests such as registering a new package, version, or build, and answering queries about the hashes of files. The primary clients of this class are
SlaveDriver
,TheScribe
, andCloudGazer
.Note that because database requests are notoriously variable in length the client RPC class (
DbClient
) doesn’t directly talk toTheOracle
. Rather, multiple instances ofTheOracle
are spawned andSeraph
sits in front of these acting as a simple load-sharing router for the RPC clients.- close()[source]¶
Close all registered queues. This should be overridden to close any additional queues the task holds which aren’t registered.
- do_allpkgs()[source]¶
Handler for “ALLPKGS” message, sent by
DbClient
to request the set of all packages define known to the database.
- do_allvers()[source]¶
Handler for “ALLVERS” message, sent by
DbClient
to request the set of all (package, version) tuples known to the database.
- do_delbuild(package, version)[source]¶
Handler for “DELBUILD” message, sent by
DbClient
to remove all builds (and files and downloads by cascade) for version of package.
- do_delver(package, version)[source]¶
Handler for “DELVER” message, sent by
DbClient
to delete a specific version of a package.
- do_getabis(exclude_skipped)[source]¶
Handler for “GETABIS” message, sent by
DbClient
to request the list of all ABIs to build for.
- do_getpkgnames(package)[source]¶
Handler for “GETPKGNAMES” message, sent by
DbClient
to retrieve all aliases for package (not including the canonical name itself).
- do_getpypi()[source]¶
Handler for “GETPYPI” message, sent by
DbClient
to request the record of the last serial number from the PyPI changelog.
- do_getsearch()[source]¶
Handler for “GETSEARCH” message, sent by
DbClient
to request the recent download statistics, returned as a mapping of package to (downloads_recent, downloads_all) tuples.
- do_getskip(package, version)[source]¶
Handler for “GETSKIP” message, send by
DbClient
to request the reason for skipping builds of version of package.
- do_getstats()[source]¶
Handler for “GETSTATS” message, sent by
DbClient
to request the latest database statistics, returned as a list of (field, value) tuples.
- do_loadrwp()[source]¶
Handler for “LOADRWP” message, sent by
DbClient
to request the content of therewrites_pending
table.
- do_logbuild(build)[source]¶
Handler for “LOGBUILD” message, sent by
DbClient
to register a new build result.
- do_logdownload(download)[source]¶
Handler for “LOGDOWNLOAD” message, sent by
DbClient
to register a new download.
- do_logjson(json)[source]¶
Handler for “LOGJSON” message, sent by
DbClient
to register a new project JSON download.
- do_logpage(page)[source]¶
Handler for “LOGPAGE” message, sent by
DbClient
to register a new web page hit.
- do_logproject(project)[source]¶
Handler for “LOGPROJECT” message, sent by
DbClient
to register a new project page hit.
- do_logsearch(search)[source]¶
Handler for “LOGSEARCH” message, sent by
DbClient
to register a new search.
- do_newpkg(package, skip, description)[source]¶
Handler for “NEWPKG” message, sent by
DbClient
to register a new package.
- do_newpkgname(package, name, seen)[source]¶
Handler for “NEWPKGNAME” message, sent by
DbClient
to register a new package alias or update the last seen timestamp.
- do_newver(package, version, released, skip)[source]¶
Handler for “NEWVER” message, sent by
DbClient
to register a new (package, version) tuple.
- do_pkgdeleted(package)[source]¶
Handler for “PKGDELETED” message, sent by
DbClient
to request whether or not the specified package has been marked for deletion.
- do_pkgexists(package)[source]¶
Handler for “PKGEXISTS” message, sent by
DbClient
to request whether or not the specified package exists.
- do_pkgfiles(package)[source]¶
Handler for “PKGFILES” message, sent by
DbClient
to request details of all wheels associated with package.
- do_projdata(package)[source]¶
Handler for “PROJDATA” message, sent by
DbClient
to request details of package and all its releases.
- do_saverwp(queue)[source]¶
Handler for “SAVERWP” message, sent by
DbClient
to request that queue is saved to therewrites_pending
table.
- do_setdesc(package, description)[source]¶
Handler for “SETDESC” message, sent by
DbClient
to update a package’s project description.
- do_setpypi(serial)[source]¶
Handler for “SETPYPI” message, sent by
DbClient
to update the last seen serial number from the PyPI changelog.
- do_skippkg(package, reason)[source]¶
Handler for “SKIPPKG” message, sent by
DbClient
to skip building all versions of a package.
- do_skipver(package, version, reason)[source]¶
Handler for “SKIPVER” message, sent by
DbClient
to skip building a specific version of a package.
- do_unyankver(package, version)[source]¶
Handler for “UNYANKVER” message, sent by
DbClient
to mark a specific version of a package as not “yanked”.
- do_verexists(package, version)[source]¶
Handler for “VEREXISTS” message, sent by
DbClient
to request whether or not the specified version of package exists.
- do_verfiles(package, version)[source]¶
Handler for “VERFILES” message, sent by
DbClient
to request the filenames of all wheels associated with version of package.
- do_versdeleted(package)[source]¶
Handler for “VERSDELETED” message, sent by
DbClient
to request any versions for package which have been marked for deletion.
- class piwheels.master.the_oracle.DbClient(config, logger=None)[source]¶
RPC client class for talking to
TheOracle
.- add_package_name(package, name, seen=datetime.datetime(1970, 1, 1, 0, 0, tzinfo=datetime.timezone.utc))[source]¶
- set_package_description(package, description)[source]¶
See
db.Database.update_project_description()
.
13.5. piwheels.master.seraph¶
Defines the Seraph
task; see class for more details.
- class piwheels.master.seraph.Seraph(config)[source]¶
This task is a simple load-sharing router for
TheOracle
tasks.
13.6. piwheels.master.the_architect¶
Defines TheArchitect
task; see class for more details.
- class piwheels.master.the_architect.TheArchitect(config)[source]¶
This task queries the backend database to determine which versions of packages have yet to be built (and aren’t marked to be skipped). It pushes the results to
SlaveDriver
to sort out.- close()[source]¶
Close all registered queues. This should be overridden to close any additional queues the task holds which aren’t registered.
- update_build_queue()[source]¶
The architect simply runs the build queue query repeatedly, with a break of a minute between each execution.
All entries found within this limit are sorted into per-ABI queues and pushed to
SlaveDriver
which queues and dispatches jobs to build ABI-matched slaves as they become available.
13.7. piwheels.master.slave_driver¶
Defines the SlaveDriver
task; see class for more details.
- class piwheels.master.slave_driver.SlaveDriver(config)[source]¶
This task handles interaction with the build slaves using the slave protocol. Interaction is driven by the slaves (i.e. the master doesn’t push jobs, rather the slaves request a job and the master replies with the next (package, version) tuple from the internal “builds” queue).
The task also incidentally interacts with several other queues: the internal “status” queue is sent details of every reply sent to a build slave (the
main_loop()
method passes this information on to any listening monitors). Also, the internal “indexes” queue is informed of any packages that need web page indexes re-building (as a result of a successful build).- close()[source]¶
Close all registered queues. This should be overridden to close any additional queues the task holds which aren’t registered.
- do_built(slave)[source]¶
Handler for the build slave’s “BUILT” message, which is sent after an attempted package build succeeds or fails. The handler logs the result in the database and, if files have been generated by the build, informs the
FileJuggler
task to expect a file transfer before sending “SEND” back to the build slave with the required filename.If no files were generated (e.g. in the case of a failed build, or a degenerate success), “DONE” is returned indicating that the build slave is free to discard all resources generated during the build and return to its idle state.
- do_busy(slave)[source]¶
Handler for the build slave’s “BUSY” message, which is sent periodically during package builds. If the slave fails to respond with a BUSY ping for a duration longer than
SlaveState.busy_timeout
then the master will assume the slave has died and remove it from the internal state mapping (if the slave happens to resurrect itself later the master will simply treat it as a new build slave).In response to “BUSY” the master can respond “CONT” to indicate the build should continue processing, or “DONE” to indicate that the build slave should immediately terminate and discard the build and return to “IDLE” state.
- do_bye(slave)[source]¶
Handler for the build slave’s final “BYE” message upon shutdown. This removes the associated state from the internal
slaves
dict.- Parameters
slave (SlaveState) – The object representing the current status of the build slave.
- do_hello(slave)[source]¶
Handler for the build slave’s initial “HELLO” message. This associates the specified slave state with the slave’s address and returns “HELLO” with the master’s id for the slave (the id communicated back simply for consistency of logging; administrators can correlate master log messages with slave log messages when both have the same id number; we can’t use IP address for this as multiple slaves can run on one machine).
- Parameters
slave (SlaveState) – The object representing the current status of the build slave.
- do_idle(slave)[source]¶
Handler for the build slave’s “IDLE” message (which is effectively the slave requesting work). If the master wants to terminate the slave, it sends back “BYE”. If the build queue (for the slave’s ABI) is empty or the task is currently paused, “SLEEP” is returned indicating the slave should wait a while and then try again.
If a job can be retrieved from the (ABI specific) build queue, then a “BUILD” message is sent back with the required package and version.
- Parameters
slave (SlaveState) – The object representing the current status of the build slave.
- do_sent(slave)[source]¶
Handler for the build slave’s “SENT” message indicating that it’s finished sending the requested file to
FileJuggler
. TheFsClient
RPC mechanism is used to askFileJuggler
to verify the transfer against the stored hash and, if this is successful, a message is sent toTheScribe
to regenerate the package’s index.If further files remain to be transferred, another “SEND” message is returned to the build slave. Otherwise, “DONE” is sent to free all build resources.
If a transfer fails to verify, another “SEND” message with the same filename is returned to the build slave.
- handle_build(queue)[source]¶
Refresh the ABI-specific queues of package versions waiting to be built. The queues are limited to 1000 packages per ABI, and are kept as lists ordered by release date. When a message arrives from
TheArchitect
it refreshes (replaces) all current queues. There is, however, still a duplication possibility asTheArchitect
doesn’t know what packages are actively being built; this method handles filtering out such packages.Even if the active builds fail (because build slaves crash, or the network dies) this doesn’t matter as a future re-run of the build queue query will return these packages again, and if no build slaves are actively working on them at that time they will then be retried.
- handle_control(queue)[source]¶
Handle incoming requests to the internal control queue.
This class understands a couple of extra control messages unique to it, specifically “KILL” to tell a build slave to terminate, “SKIP” to tell a build slave to terminate its current build immediately, and “HELLO” to cause all “HELLO” messages from build slaves to be replayed (for the benefit of a newly attached monitor process).
- handle_delete(queue)[source]¶
Handle package or version deletion requests.
When the PyPI upstream deletes a version or package, the
CloudGazer
task requests that other tasks perform the deletion on its behalf. In the case of this task, this involves cancelling any pending builds of that package (version), and ignoring any builds involving that package (version) in the next queue update fromTheArchitect
.
- handle_slave(queue)[source]¶
Handle requests from build slaves.
See the piw-slave chapter for an overview of the protocol for messages between build slaves and
SlaveDriver
. This method retrieves the message from the build slave, finds the associatedSlaveState
and updates it with the message, then calls the appropriate message handler. The handler will be expected to return a reply (in the usual form of a list of strings) orNone
if no reply should be sent (e.g. for a final “BYE” message).
- kill_slave(slave_id)[source]¶
Additional task control method to trigger a “KILL” message to the internal control queue. See
handle_control()
for more information.
- list_slaves()[source]¶
Additional task control method to trigger a “HELLO” message to the internal control queue. See
quit()
for more information.
- skip_slave(slave_id)[source]¶
Additional task control method to trigger a “SKIP” message to the internal control queue. See
handle_control()
for more information.
- sleep_slave(slave_id)[source]¶
Additional task control method to trigger a “SLEEP” message to the internal control queue. See
handle_control()
for more information.
- wake_slave(slave_id)[source]¶
Additional task control method to trigger a “WAKE” message to the internal control queue. See
handle_control()
for more information.
13.8. piwheels.master.mr_chase¶
Defines the MrChase
task; see class for more details.
- class piwheels.master.mr_chase.MrChase(config)[source]¶
This task handles smuggling packages into the database manually. It is the task that the piw-import script talks to in order to import packages.
Internally, the task is essentially an abbreviated
SlaveDriver
(in as much as it has to perform similar database and file-system interactions) but without having to handle talking to lots of build slaves.- close()[source]¶
Close all registered queues. This should be overridden to close any additional queues the task holds which aren’t registered.
- do_add_package(state)[source]¶
Handler for the remover’s “ADDPKG” message, indicating a request to add a package to the system, or update it.
- do_add_version(state)[source]¶
Handler for the remover’s “ADDVER” message, indicating a request to add a specific version of a package to the system, or update it.
- do_import(state)[source]¶
Handler for the importer’s initial “IMPORT” message. This method checks the information in the state passes some simple tests, then ensures that the requested package and version exist in the database (creating them if necessary).
- do_rebuild(state)[source]¶
Handler for the rebuilder’s “REBUILD” message, indicating a request to rebuild part of the website.
- do_remove_package(state)[source]¶
Handler for the remover’s “REMPKG” message, indicating a request to remove or alter a whole package.
- do_remove_version(state)[source]¶
Handler for the remover’s “REMVER” message, indicating a request to remove or alter a specific package version.
- do_sent(state)[source]¶
Handler for the importer’s “SENT” message indicating that it’s finished sending the requested file to
FileJuggler
. The file is verified (as inSlaveDriver
) and, if this is successful, a mesasge is sent toTheScribe
to regenerate the package’s index.If further files remain to be transferred, another “SEND” message is returned to the build slave. Otherwise, “DONE” is sent to free all build resources.
If a transfer fails to verify, another “SEND” message with the same filename is returned to the build slave.
- handle_import(queue)[source]¶
Handle requests from piw-import instances.
See the piw-import and piw-remove chapters for an overview of the protocol for messages between the importer and
MrChase
.
13.9. piwheels.master.file_juggler¶
Defines the FileJuggler
task and the FsClient
RPC class
for interacting with it.
- exception piwheels.master.file_juggler.TransferError[source]¶
Base class for errors raised during a file transfer.
- exception piwheels.master.file_juggler.TransferIgnoreChunk[source]¶
Exception raised when a build slave sends CHUNK instead of HELLO as the first message (see
FileJuggler.new_transfer()
).
- exception piwheels.master.file_juggler.TransferDone[source]¶
Exception raised when a transfer is complete. It may seem a little odd to use an exception for this, but it is “exceptional” behaviour to terminate the file transfer.
- class piwheels.master.file_juggler.FileJuggler(config)[source]¶
This task handles file transfers from the build slaves. The specifics of the file transfer protocol are best understood from the implementation of the
FileState
class.However, to detail how a file transfer begins: when a build slave has successfully completed a build it informs the master via the
SlaveDriver
task. That task replies with a “SEND” instruction to the slave (including a filename). The slave then initiates the transfer with a “HELLO” message to this task. Once transfers are complete the slave sends a “SENT” message to theSlaveDriver
task which verifies the transfer and either retries it (when verification fails) or sends back “DONE” indicating the slave can wipe the source file.- current_transfer(transfer, msg, *args)[source]¶
Called for messages associated with an existing file transfer.
Usually this is “CHUNK” indicating another chunk of data. Rarely, it can be “HELLO” if the master has fallen silent and dropped tons of packets.
- Parameters
transfer (TransferState) – The object representing the state of the transfer.
msg (str) – The message sent during the transfer.
*args – All additional arguments; for “CHUNK” the first must be the file offset and the second the data to write to that offset.
- do_expect(slave_id, file_state)[source]¶
Message sent by
FsClient
to inform file juggler that a build slave is about to start a file transfer. The message includes the fullFileState
. The state is stored in thepending
map.
- do_verify(slave_id, package)[source]¶
Message sent by
FsClient
to request that juggler verify a file transfer against the expected hash and, if it matches, rename the file into its final location.
- handle_file(queue)[source]¶
Handle incoming file-transfer messages from build slaves.
The file transfer protocol is in some ways very simple (see the chart in the piw-slave chapter for an overview of the message sequence) and in some ways rather complex (read the ZeroMQ guide chapter on file transfers for more detail on why multiple messages must be allowed in flight simultaneously).
The “normal” state for a file transfer is to be requesting and receiving chunks. Anything else, including redundant re-sends, and transfer completion is handled as an exceptional case.
- new_transfer(msg, *args)[source]¶
Called for messages initiating a new file transfer.
The first message must be HELLO along with the id of the slave starting the transfer. The metadata for the transfer will be looked up in the
pending
list (which is written to bydo_expect()
).- Parameters
msg (str) – The message sent to start the transfer (must be “HELLO”)
*args – All additional arguments (expected to be an integer slave id).
13.10. piwheels.master.big_brother¶
Defines the BigBrother
task; see class for more details.
- class piwheels.master.big_brother.BigBrother(config)[source]¶
This task periodically queries the database and output file-system for various statistics like the number of packages known to the system, the number built, the number of packages built in the last hour, the remaining file-system space, etc. These statistics are written to the internal “status” queue which
main_loop()
uses to pass statistics to any listening monitors.
13.11. piwheels.master.the_secretary¶
Defines the TheSecretary
task; see class for more details.
- class piwheels.master.the_secretary.TheSecretary(config)[source]¶
This task buffers requests for the scribe, for the purpose of consolidating multiple consecutive (duplicate) requests.
Requests to write the project page for a package (which is a relatively expensive operation in terms of database accesses) can come in thick and fast, particularly when a new version is being registered with lots of files. There’s little point in writing the project page 5 times in as many seconds, or writing the project page, then the index and project page immediately afterward. This class is used to buffer requests for up to a minute, allowing us to eliminate many of the duplicate requests.
- close()[source]¶
Close all registered queues. This should be overridden to close any additional queues the task holds which aren’t registered.
- handle_input(queue)[source]¶
Handle incoming write requests with buffering and de-dupe.
Some incoming requests (currently “HOME”, “SEARCH”, “LOG”, “DELPKG”, and “DELVER”) are passed directly through to
TheScribe
as these are either sufficiently rare (“HOME”, “SEARCH”) that no benefit is gained by buffering them or sufficiently urgent (“DELPKG”, “DELVER”, “LOG”) that they must be acted on immediately.For other requests (“PROJECT” and “BOTH”), requests can come thick and fast in the case of multiple file registrations picked up by
CloudGazer
. In this case, requests are buffered for a minute and de-duplicated; e.g. if several requests are made to re-write the project page for package “foo” within that period, they will be combined into a single request. After the minute of buffering, the request is passed down toTheScribe
.
13.12. piwheels.master.the_scribe¶
Defines the TheScribe
task; see class for more details.
- class piwheels.master.the_scribe.TheScribe(config)[source]¶
This task is responsible for writing web-page
index.html
files. It reads the names of packages off the internal “indexes” queue and rebuilds theindex.html
for that package and, optionally, the overallindex.html
if the package is one that wasn’t previously present.Note
It is important to note that package names are never pushed into the internal “indexes” queue until all file-transfers associated with the build are complete. Furthermore, while the entire index for a package is re-built, hashes are never re-calculated from the disk files (they are always read from the database).
- close()[source]¶
Close all registered queues. This should be overridden to close any additional queues the task holds which aren’t registered.
- delete_package(package)[source]¶
Attempts to remove the index and project page directories (including all known wheel files) of the specified package.
- Parameters
package (str) – The name of the package to delete.
- delete_version(package, version)[source]¶
Attempts to remove any known wheel files corresponding with deleted versions of the specified package.
- handle_index(queue)[source]¶
Handle incoming requests to (re)build index files. These will be in the form of:
“HOME”, a request to write the homepage with some associated statistics
“BOTH”, a request to write the index and project page for the specified package
“PROJECT”, a request to write just the project page for the specified package
“LOG”, a request to write a build log
Note
In all handlers below, care is taken to ensure clients never see a partially written file and that temporary files are cleaned up in the event of any exceptions.
- once()[source]¶
This method is called once before the task loop starts. It the task needs to do some initialization or setup within the task thread, this is the place to do it.
- setup_output_path()[source]¶
Called on task startup to copy all static resources into the output path (and to make sure the output path exists as a directory).
- write_homepage(statistics)[source]¶
Re-writes the site homepage using the provided statistics in the homepage template (which is effectively a simple Python format string).
- Parameters
statistics (dict) – A dict containing statistics obtained by
BigBrother
.
- write_log(build_id, log)[source]¶
Attempts to write the log of build build_id to the log output directories, splitting the numeric build id into three parts to flatten the output hierarchy. Log data is also gzip compressed.
- write_package_index(package, data)[source]¶
(Re)writes the index of the specified package. The file meta-data (including the hash) is retrieved from the database, never from the file-system.
The data parameter is expected to be the dictionary of package data returned by
db.Database.get_project_data()
. This is expected to have at least the following content in the example case of a package named “foo” with version “1.0” containing a validly built wheel:{ 'releases': { '1.0': { 'files': { 'foo-1.0-py3-none-any.whl': { 'hash': 'abcdef1234567890...', 'requires_python': '>= 3.6', }, }, 'yanked': False, }, }, }
- Parameters
package (str) – The name of the package to write the index page for.
data (dict) – The dictionary of data returned by
db.Database.get_project_data()
which is expected to have at least the structure documented above.
- write_pages(package, *, both=False, exclude=None)[source]¶
(Re)writes the project page and project JSON file (and simple index if both is True) for the specified package.
- Parameters
package (str) – The name of the package to write the pages for
both (bool) – Write both the project page and the simple page if True, otherwise only write the project page. Note project page also includes project JSON.
exclude (set or None) – The set of (deleted) versions to exclude from pages. Defaults to
None
.
- write_project_json(package, data)[source]¶
(Re)writes the project JSON data of the specified package.
The data parameter is expected to be the dictionary of package data returned by
db.Database.get_project_data()
. This is expected to have at least the following content in the example case of a package named “foo” with version “1.0” containing a validly built wheel:{ 'name': 'foo', 'description': 'A foomatic package', 'releases': { '1.0': { 'abis': { 'cp35m': { 'build_id': 1, 'status': 'success', 'skip': '', } } 'files': { 'foo-1.0-py3-none-any.whl': { 'hash': 'abcdef1234567890...', 'size': 123456, 'apt_dependencies': {'libc6'}, }, }, 'released': datetime(2000, 1, 1, 12, 34, 56), 'yanked': False, 'skip': '', }, }, }
- Parameters
package (str) – The name of the package to write the project data for.
data (dict) – The dictionary of data returned by
db.Database.get_project_data()
which is expected to have at least the structure documented above.
- write_project_page(package, data)[source]¶
(Re)writes the project page of the specified package.
The data parameter is expected to be the dictionary of package data returned by
db.Database.get_project_data()
. This is expected to have at least the following content in the example case of a package named “foo” with version “1.0” containing a validly built wheel:{ 'name': 'foo', 'description': 'A foomatic package', 'releases': { '1.0': { 'abis': { 'cp35m': { 'build_id': 1, 'status': 'success', 'skip': '', } } 'files': { 'foo-1.0-py3-none-any.whl': { 'hash': 'abcdef1234567890...', 'size': 123456, 'apt_dependencies': {'libc6'}, }, }, 'released': datetime(2000, 1, 1, 12, 34, 56), 'yanked': False, 'skip': '', }, }, }
- Parameters
package (str) – The name of the package to write the project page for.
data (dict) – The dictionary of data returned by
db.Database.get_project_data()
which is expected to have at least the structure documented above.
- write_search_index(search_index)[source]¶
Re-writes the JSON search index using the provided statistics.
- Parameters
search_index (dict) – A dict mapping package names to their download count obtained by
BigBrother
.
13.13. piwheels.slave¶
Defines the PiWheelsSlave
class. An instance of this is the
entry-point for the piw-slave script.
- class piwheels.slave.PiWheelsSlave[source]¶
This is the main class for the piw-slave script. It connects (over 0MQ sockets) to a master (see piw-master) then loops around the slave protocol (see the piw-slave chapter). It retrieves source packages directly from PyPI, attempts to build a wheel in a sandbox directory and, if successful, transmits the results to the master.
- clean_up_build(timeout=datetime.timedelta(seconds=60))[source]¶
Terminate any existing build and clean up its temporary storage. Raises an exception if termination does not occur in a reasonable time.
- do_ack(new_id, pypi_url)[source]¶
In response to our initial “HELLO” (detailing our various PEP 425 tags), the master is expected to send “ACK” back with an integer identifier and the URL of the PyPI repository to download from. We use the identifier in all future log messages for the ease of the administrator.
We reply with “IDLE” to indicate we’re ready to accept a build job.
- do_build(package, version)[source]¶
Alternatively, in response to “IDLE”, the master may send “BUILD” package version. We should then attempt to build the specified wheel and send back a “BUSY” message with more status info.
- do_cont()[source]¶
Once we’re busy building something the master will periodically ping us with “CONT” if it wishes us to continue building. We wait up to 10 seconds for the build to finish and reply with “BUILT” (and a full status report on the build) if it finishes in that time, or “BUSY” and more status info otherwise.
If the master wishes to terminate a build prior to completion, it’ll send “DONE” instead of “CONT” and we move straight to clean-up.
- do_die()[source]¶
The master may respond with “DIE” at any time indicating we should immediately terminate. We raise
SystemExit
to causemain_loop()
to exit. Clean-up of any extant build is handled by our caller.
- do_done()[source]¶
The master can send “DONE” at any point during a built to terminate it prematurely. Alternately, this is also the standard response after a successful build has finished and all files have been sent (and successfully verified).
In response we must clean-up all resources associated with the build (including terminating an on-going build) and return “IDLE” with the usual stats.
- do_send(filename)[source]¶
If a build succeeds and generates files (detailed in a “BUILT” message), the master will reply with “SEND” filename indicating we should transfer the specified file (this is done on a separate socket with a different protocol; see
builder.Wheel.transfer()
for more details). Once the transfers concludes, reply to the master with “SENT”.
- do_sleep(paused)[source]¶
If, in response to an “IDLE” message we receive “SLEEP” this indicates the master has nothing for us to do currently. Sleep for a little while then try “IDLE” again.
- get_status()[source]¶
Returns the set of statistics periodically required by the master when reporting our status.
- handle_reply(msg, data)[source]¶
Dispatch a message from the master to an appropriate handler method.
- main_loop(queue, master_timeout=datetime.timedelta(seconds=300))[source]¶
The main messaging loop. Sends the initial request, and dispatches replies via
handle_reply()
. Implements a timeout for responses from the master and raisesMasterTimeout
if timeout seconds are exceeded.
- piwheels.slave.duration(s)[source]¶
Convert s, a string representing a duration, into a
datetime.timedelta
.
13.14. piwheels.slave.builder¶
Defines the classes which use pip
to build wheels.
- class piwheels.slave.builder.Wheel(path, dependencies=None)[source]¶
Records the state of a build artifact, i.e. a wheel package. The filename is deconstructed into the fields specified by PEP 425.
- Parameters
path (pathlib.Path) – The path to the wheel on the local filesystem.
dependencies (dict) – A dict mapping tool to dependencies that are required to use these particular wheel files. Defaults to a
None
(no dependencies).
- as_message()[source]¶
Return the state as a list suitable for use in the
BUILT
message of piw-slave.
- transfer(queue, slave_id)[source]¶
Transfer the wheel via the specified queue. This is the client side implementation of the
file_juggler.FileJuggler
protocol.
- property abi_tag¶
Return the ABI part of the wheel’s filename (the penultimate “-” separated element).
- property build_tag¶
Return the optional build part of the wheel’s filename (the third “-” separated element when 6 elements exist in total).
- property dependencies¶
Return the dependencies required by the wheel as a mapping of dependency system (e.g. “apt”, “pip”, etc.) to set of package names for that system.
- property filehash¶
Return an SHA256 digest of the wheel’s contents.
- property filename¶
Return the filename of the wheel as a simple string (with no path components).
- property filesize¶
Return the size of the wheel in bytes.
- property metadata¶
Return the contents of the
METADATA
file inside the wheel.
- property package_canon¶
Return the package part of the wheel’s filename, canonicalized according to PyPI’s rules.
- property package_tag¶
Return the package part of the wheel’s filename (the first “-” separated element).
- property package_version_tag¶
Return the version part of the wheel’s filename (the second “-” separated element).
- property platform_tag¶
Return the platform part of the wheel’s filename (the last “-” separated element).
- property py_version_tag¶
Return the python version part of the wheel’s filename (third from last “-” separated element).
- property requires_python¶
Return the contents of the
Requires-Python
specification from the wheel metadata.
- class piwheels.slave.builder.Builder(package, version, *, timeout=datetime.timedelta(seconds=300), index_url='https://pypi.python.org/simple', extra_index_urls={'https://www.piwheels.org/simple'}, dir=None)[source]¶
Class responsible for building wheels for a given version of a package. Note that this class derives from
Thread
and hence is expected to run in the background after callingstart()
.- Parameters
package (str) – The name of the package to attempt to build wheels for.
version (str) – The version of the package to attempt to build.
timeout (datetime.timedelta) – The number of seconds to wait for
pip
to finish before raisingsubprocess.TimeoutExpired
.index_url (str) – The URL of the PEP 503 compliant repository from which to fetch packages for building.
extra_index_urls (set) – The URLs of any additional PEP 503 compliant repositories from which to fetch packages.
dir (str) – The directory in which to store wheel and log output.
- as_message()[source]¶
Return the state as a list suitable for use in the
BUILT
message of piw-slave.
- build_dependencies(wheel)[source]¶
Calculate the apt dependencies of wheel (which is a
Wheel
instance representing a built wheel).
- build_wheel(log_file)[source]¶
Call pip and attempt to build the wheel; handle killing the subprocess if termination is requested, and watch the clock for a build timeout.
- property duration¶
The
timedelta
indicating how long the actual build took (without any extraneous tasks like dependency calculation). This is an indication of how long a user would spend installing the package without piwheels.
- property extra_index_urls¶
The URLs of any additional indexes from which the builder will also check when retrieving packages. This is intended to be used for fetching compiled platform wheels for specified build dependencies.
- property index_url¶
The URL of primary index from which the builder will attempt to obtain the source to build.
- property output¶
The log output from the build.
- property package¶
The package that the builder will attempt to build.
13.15. piwheels.initdb¶
Contains the functions that make up the piw-initdb script.
- piwheels.initdb.main(args=None)[source]¶
This is the main function for the piw-initdb script. It creates the piwheels database required by the master or, if it already exists, upgrades it to the current version of the application.
- piwheels.initdb.detect_users(conn, test_user)[source]¶
Test that the user for conn is a cluster superuser (so we can drop and create anything we want in the database), and that test_user (which will be granted limited rights to various objects for the purposes of the piw-master script) exists and is not a cluster superuser.
- piwheels.initdb.detect_version(conn)[source]¶
Detect the version of the database. This is typically done by reading the contents of the
configuration
table, but before that was added we can guess a couple of versions based on what tables exist (or don’t). ReturnsNone
if the database appears uninitialized, and raisesRuntimeError
is the version is so ancient we can’t do anything with it.
- piwheels.initdb.get_connection(dsn)[source]¶
Return an SQLAlchemy connection to the specified dsn or raise
RuntimeError
if the database doesn’t exist (the administrator is expected to create the database before running this script).
- piwheels.initdb.get_script(version=None)[source]¶
Generate the script to get the database from version (the result of
detect_version()
) to the current version of the software. If version isNone
, this is simply the contents of thesql/create_piwheels.sql
script. Otherwise, it is a concatenation of various update scripts.
- piwheels.initdb.parse_statements(script)[source]¶
This is an extremely crude statement splitter for PostgreSQL’s dialect of SQL. It understands
--comments
,"quoted identifiers"
,'string literals'
and$delim$ extended strings $delim$
, but notE'\escaped strings'
or/* C-style comments */
. If you start using such things in the update scripts, you’ll need to extend this function to accommodate them.It returns a generator which yields individiual statements from script, delimited by semi-colon terminators.
13.16. piwheels.importer¶
Contains the functions that implement the piw-import script.
- piwheels.importer.main(args=None)[source]¶
This is the main function for the piw-import script. It uses some bits of the piw-slave script to deconstruct the filenames passed to it in order to build all the required information that
MrChase
needs.
- piwheels.importer.print_state(state)[source]¶
Dumps a human-readable description of the state to the log / console.
- Parameters
state (BuildState) – The build state to print the description of.
- piwheels.importer.do_import(config, packages, state)[source]¶
Handles constructing and sending the initial “IMPORT” message to
master.mr_chase.MrChase
. If “SEND” is then received, usesdo_send()
to handle transmitting files.- Parameters
config – The configuration obtained from parsing the command line.
packages (list) – A sequence of
Wheel
objects corresponding to files in the state.state (BuildState) – The object representing the state of the build.
- piwheels.importer.do_send(packages, filename)[source]¶
Handles sending files when requested by
do_import()
.
13.17. piwheels.add¶
Contains the functions that implement the piw-add script.
13.18. piwheels.remove¶
Contains the functions that implement the piw-remove script.
13.19. piwheels.transport¶
This module augments the classes provided by pyzmq (the 0MQ Python bindings) to use CBOR encoding, and voluptuous for message validation. It also tweaks a few minor things like using seconds for timeouts.
- class piwheels.transport.Context[source]¶
Wrapper for 0MQ
zmq.Context
. This extends thesocket()
method to include parameters for the socket’s protocol and logger.
- class piwheels.transport.Socket(socket, protocol=None, logger=None)[source]¶
Wrapper for
zmq.Socket
. This extends 0MQ’s sockets to include a protocol which will be used to validate messages that are sent and received (via a voluptuous schema), and a logger which can be used to debug socket behaviour.- close(linger=None)[source]¶
Closes the socket. If linger is specified, it is the number of seconds to wait for pending messages to be flushed.
- drain()[source]¶
Receives all pending messages in the queue and discards them. This is typically useful during shutdown routines or for testing.
- poll(timeout=None, flags=zmq.POLLIN)[source]¶
Polls the socket for pending data (by default, when flags is POLLIN). If no data is available after timeout seconds, returns False. Otherwise returns True.
If flags is POLLOUT instead, tests whether the socket has available slots for queueing new messages.
- recv_addr_msg(flags=0)[source]¶
Receive a CBOR-encoded message (and associated data) along with the address it came from (represented as a
bytes
string).
- recv_msg(flags=0)[source]¶
Receive a CBOR-encoded message, returning a tuple of the unicode message string and its associated data. This is the primary method used in piwheels for receiving information into a task.
The message, and its associated data, will be validated against the
protocol
associated with the socket on construction.
- recv_multipart(flags=0)[source]¶
Receives a multi-part message, returning its content as a list of
bytes
strings.
- send_addr_msg(addr, msg, data=NoData, flags=0)[source]¶
Send a CBOR-encoded message (and associated data) to addr, a
bytes
string.
- send_msg(msg, data=NoData, flags=0)[source]¶
Send the unicode string msg with its associated data as a CBOR-encoded message. This is the primary method used in piwheels for sending information between tasks.
The message, and its associated data, must validate against the
protocol
associated with the socket on construction.
- send_multipart(msg_parts, flags=0)[source]¶
Send msg_parts, a list of
bytes
strings as a multi-part message which can be received intact withrecv_multipart()
.
- unsubscribe(topic)[source]¶
Unsubscribes SUB type sockets from the specified topic (a string prefix).
- property hwm¶
The high-water mark of the socket, i.e. the number of messages that can be queued before the socket blocks (or drops, depending on the socket type) messages.
- class piwheels.transport.Poller[source]¶
Wrapper for 0MQ
zmq.Poller
. This simply tweaks 0MQ’s poller to use seconds for timeouts, and to return adict
by default frompoll()
.- poll(timeout=None)[source]¶
Poll all registered sockets for the events they were registered with, for timeout seconds. Returns a dictionary mapping sockets to events or an empty dictinoary if the timeout elapsed with no events occurring.
13.20. piwheels.tasks¶
Implements the base classes (Task
and its derivative
PauseableTask
) which form the basis of all the tasks in the piwheels
master.
- exception piwheels.tasks.TaskQuit[source]¶
Exception raised when the “QUIT” message is received by the internal control queue.
- class piwheels.tasks.Task(config, control_protocol=Protocol(recv={'PAUSE': NoData, 'RESUME': NoData, 'QUIT': NoData}, send={}))[source]¶
The
Task
class is aThread
derivative which is the base for all tasks in the piwheels master. Therun()
method is overridden to perform a simple task loop which callspoll()
once a cycle to react to any messages arriving into queues, and to dispatch any periodically executed methods.Queues are associated with handlers via the
register()
method. Periodic methods are associated with an interval via theevery()
method. These should be called during initialization (don’t attempt to register handlers from within the thread itself).Generally this shouldn’t be used as a base-class. Use one of the descendents that implements a pausing mechanism,
NonStopTask
,PauseableTask
, orPausingTask
.- close()[source]¶
Close all registered queues. This should be overridden to close any additional queues the task holds which aren’t registered.
- every(interval, handler)[source]¶
Register handler to be called every interval periodically.
- Parameters
interval (timedelta) – The time interval between each run of handler.
handler – The function or method to call periodically.
- handle_control(queue)[source]¶
Default handler for the internal control queue. In this base class it simply handles the “QUIT” message by raising
TaskQuit
(which therun()
method will catch and use as a signal to end).Messages other than QUIT, PAUSE and RESUME raise
TaskControl
which can be caught in descendents to implement custom control messages.
- once()[source]¶
This method is called once before the task loop starts. It the task needs to do some initialization or setup within the task thread, this is the place to do it.
- pause()[source]¶
Requests that the task pause itself. This is an idempotent method; it’s always safe to call repeatedly and even if the task isn’t pauseable it’ll simply be ignored.
- poll(timeout=1)[source]¶
This method is called once per loop of the task’s
run()
method. It runs all periodic handlers, then polls all registered queues and calls their associated handlers if the poll is successful.
- quit()[source]¶
Requests that the task terminate at its earliest convenience. To wait until the task has actually closed, call
join()
afterwards.
- register(queue, handler, flags=zmq.POLLIN)[source]¶
Register queue to be polled on each cycle of the task. Any messages with the relevant flags (defaults to
POLLIN
) will trigger the specified handler method which is expected to take a single argument which will be queue.- Parameters
queue (transport.Socket) – The queue to poll.
handler – The function or method to call when a message with matching flags arrives in queue.
flags (int) – The flags to match in the queue poller (defaults to
POLLIN
).
- resume()[source]¶
Requests that the task resume itself. This is an idempotent method; it’s safe to call repeatedly and even if the task isn’t pauseable it’ll simply be ignored.
- class piwheels.tasks.PauseableTask(config, control_protocol=Protocol(recv={'PAUSE': NoData, 'RESUME': NoData, 'QUIT': NoData}, send={}))[source]¶
Derivative of
Task
that implements a rudimentary pausing mechanism. When the “PAUSE” message is received on the internal control queue, the task will enter a loop which simply polls the control queue waiting for “RESUME” or “QUIT”. No other work will be done (Task.loop()
andTask.poll()
will not be called) until the task is resumed (or terminated).If you need a more complex pausing implementation which can still do some work while paused (to drain incoming queues for instance), use
PausingTask
instead.- handle_control(queue)[source]¶
Default handler for the internal control queue. In this base class it simply handles the “QUIT” message by raising
TaskQuit
(which therun()
method will catch and use as a signal to end).Messages other than QUIT, PAUSE and RESUME raise
TaskControl
which can be caught in descendents to implement custom control messages.
13.21. piwheels.states¶
This module defines several classes which permit interested tasks to track the
state of build slaves (SlaveState
), file transfers
(TransferState
), build attempts (BuildState
), build
artifacts (FileState
) and various loggers.
- class piwheels.states.FileState(filename, filesize, filehash, package_tag, package_version_tag, py_version_tag, abi_tag, platform_tag, requires_python, dependencies, transferred=False)[source]¶
Represents the state of an individual build artifact (a package file, or wheel) including its
filename
,filesize
, the SHA256filehash
, and various tags extracted from the build. Also tracks whether or not the file has beentransferred
.- Parameters
filename (str) – The original filename of the build artifact.
filesize (int) – The size of the file in bytes.
filehash (str) – The SHA256 hash of the file contents.
package_tag (str) – The package tag extracted from the filename (first “-” separated component).
package_version_tag (str) – The package version tag extracted from the filename (second “-” separated component).
py_version_tag (str) – The python version tag extracted from the filename (third from last “-” separated component).
abi_tag (str) – The python ABI tag extracted from the filename (second from last “-” separated component).
platform_tag (str) – The platform tag extracted from the filename (last “-” separated component).
requires_python (str) – The
Requires-Python
specification for the file.dependencies (set) – The set of dependencies that are required to use this particular wheel.
transferred (bool) –
True
if the file has been transferred from the build slave that generated it to the file server.
- as_message()[source]¶
Convert the
FileState
object into a simpler list for serialization and transport.
- classmethod from_message(value)[source]¶
Convert the output from
as_message()
back into aBuildState
.
- class piwheels.states.BuildState(slave_id, package, version, abi_tag, status, duration, output, files, build_id=None)[source]¶
Represents the state of a package build including the
package
,version
,status
, buildduration
, and all the lines ofoutput
. Thefiles
attribute is a mapping containing details of each successfully built package file.- Parameters
slave_id (int) – The master’s identifier for the build slave.
package (str) – The name of the package to build.
version (str) – The version number of the package to build.
abi_tag (str) – The ABI for which the build was attempted (must not be
'none'
).status (bool) –
True
if the build succeeded,False
if it failed.duration (timedelta) – The amount of time (in seconds) it took to complete the build.
output (str) – The log output of the build.
files (dict) – A mapping of filenames to
FileState
objects for each artifact produced by the build.build_id (int) – The integer identifier generated for the build by the database (
None
until the build has been inserted into the database).
- as_message()[source]¶
Convert the
BuildState
, and its nestedFileState
objects into simpler lists for serialization and transport.
- classmethod from_message(value)[source]¶
Convert the output from
as_message()
back into aBuildState
.
- property next_file¶
Returns the filename of the next file that needs transferring or
None
if all files have been transferred.
- property transfers_done¶
Returns
True
if all files have been transferred.
- class piwheels.states.SlaveState(address, build_timeout, busy_timeout, native_py_version, native_abi, native_platform, label, os_name, os_version, board_revision, board_serial)[source]¶
Tracks the state of a build slave. The master updates this state with each request and reply sent to and received from the slave, and this class in turn manages the associated
BuildState
(accessible frombuild
) andTransferState
(accessible fromtransfer
). The class also tracks the time a request was last seen from the build slave, and includes akill()
method.- Parameters
address (bytes) –
The slave’s ephemeral 0MQ address.
Note
This is not the slave’s IP address; it’s a unique identifier generated on connection to the master’s ROUTER socket. It will be different each time the slave re-connects (due to timeout, reboot, etc).
timeout (int) – The number of seconds after which any build will be considered to have timed out (and the slave will be assumed crashed).
native_py_version (str) – The slave’s native Python version.
native_abi (str) – The slave’s native Python ABI.
native_platform (str) – The slave’s native platform.
label (str) – A label representing the slave.
- class piwheels.states.TransferState(slave_id, file_state)[source]¶
Tracks the state of a file transfer. All file transfers are held in temporary locations until
verify()
indicates the transfer was successful, at which point they are atomically renamed into their final location.The state is intimately tied to the file transfer protocol and includes methods to write a received
chunk()
, and to determine the next chunk tofetch()
, as well as a property to determine when the transfer isdone
.
- class piwheels.states.DownloadState(filename, host, timestamp, arch, distro_name, distro_version, os_name, os_version, py_name, py_version, installer_name, installer_version, setuptools_version)[source]¶
Represents the state of the log entry for a download of a package wheel file, including its
filename
, the user’shost
IP, accesstimestamp
and information about the operating system and installer.- Parameters
filename (str) – The filename of the downloaded wheel file.
host – The hostname or IP address of the user.
timestamp (datetime.datetime) – The timestamp at which the file was downloaded.
arch (str or None) – The architecture of the user’s computer system (usually armv6 or armv7).
distro_name (str or None) – The user’s operating system distribution name (e.g. Raspbian).
distro_version (str or None) – The version of the user’s operating system distribution.
os_name (str or None) – The name of the user’s operating system (e.g. Linux).
os_version (str or None) – The version of the user’s operating system (e.g. Linux kernel version).
py_name (str or None) – The Python implementation used (e.g. CPython).
py_version (str or None) – The Python version used (e.g. 3.7.3).
installer_name (str or None) – The name of the tool used to install the file (e.g. pip).
installer_version (str or None) – The version of the tool (e.g. pip) used to install the file.
setuptools_version (str or None) – The version of setuptools used.
- class piwheels.states.SearchState(package, host, timestamp, arch, distro_name, distro_version, os_name, os_version, py_name, py_version, installer_name, installer_version, setuptools_version)[source]¶
Represents the state of the log entry for an instance of a package search, including the
package
name, user’shost
IP, accesstimestamp
and information about the operating system and installer.- Parameters
package (str) – The name of the package searched for.
host (str) – The hostname or IP address of the user.
timestamp (datetime.datetime) – The timestamp at which the search occurred.
arch (str or None) – The architecture of the user’s computer system (usually armv6 or armv7).
distro_name (str or None) – The user’s operating system distribution name (e.g. Raspbian).
distro_version (str or None) – The version of the user’s operating system distribution.
os_name (str or None) – The name of the user’s operating system (e.g. Linux).
os_version (str or None) – The version of the user’s operating system (e.g. Linux kernel version).
py_name (str or None) – The Python implementation used (e.g. CPython).
py_version (str or None) – The Python version used (e.g. 3.7.3).
installer_name (str or None) – The name of the tool used (e.g. pip).
installer_version (str or None) – The version of the tool (e.g. pip) used.
setuptools_version (str or None) – The version of setuptools used.
- class piwheels.states.ProjectState(package, host, timestamp, user_agent)[source]¶
Represents the state of the log entry for an instance of project page hit, including the
page
name, the user’shost
IP, accesstimestamp
and the user’suser_agent
.- Parameters
package (str) – The name of the package searched for.
host (str) – The hostname or IP address of the user.
timestamp (
datetime.datetime
) – The timestamp at which the page was accessed.user_agent (str) – The user agent of the page request.
- class piwheels.states.JSONState(package, host, timestamp, user_agent)[source]¶
Represents the state of the log entry for an instance of project JSON download, including the
page
name, the user’shost
IP, accesstimestamp
and the user’suser_agent
.- Parameters
package (str) – The name of the package whose JSON file was accessed.
host (str) – The hostname or IP address of the user.
timestamp (datetime.datetime) – The timestamp at which the page was accessed.
user_agent (str) – The user agent of the request.
- class piwheels.states.PageState(page, host, timestamp, user_agent)[source]¶
Represents the state of the log entry for an instance of web page hit, including the
page
name, the user’shost
IP, accesstimestamp
and the user’suser_agent
.- Parameters
page (str) – The name of the page accessed.
host (str) – The IP address of the user.
timestamp (datetime.datetime) – The timestamp at which the page was accessed.
user_agent (str) – The user agent of the page request.
- class piwheels.states.SlaveStats(timestamp, disk_size, disk_free, mem_size, mem_free, swap_size, swap_free, load_average, cpu_temp)[source]¶
- class piwheels.states.MasterStats(timestamp, packages_built, builds_last_hour, builds_time, builds_size, builds_pending, new_last_hour, files_count, downloads_last_hour, downloads_last_month, downloads_all, disk_size, disk_free, mem_size, mem_free, swap_size, swap_free, load_average, cpu_temp)[source]¶
13.22. piwheels.ranges¶
A set of utility routines for efficiently tracking byte ranges within a stream. These are used to track which chunks of a file have been received during file transfers from build slaves.
See FileJuggler
for the usage of these functions.
- piwheels.ranges.consolidate(ranges)[source]¶
Given a list of ranges in ascending order, this generator function returns the list with any overlapping ranges consolidated into individual ranges. For example:
>>> list(consolidate([range(0, 5), range(4, 10)])) [range(0, 10)] >>> list(consolidate([range(0, 5), range(5, 10)])) [range(0, 10)] >>> list(consolidate([range(0, 5), range(6, 10)])) [range(0, 5), range(6, 10)]
- piwheels.ranges.exclude(ranges, ex)[source]¶
Given a list of non-overlapping ranges in ascending order, and a range ex to exclude, this generator function returns ranges with all values covered by ex removed from any contained ranges. For example:
>>> list(exclude([range(10)], range(2))) [range(2, 10)] >>> list(exclude([range(10)], range(2, 4))) [range(0, 2), range(4, 10)]
- piwheels.ranges.intersect(range1, range2)[source]¶
Given two ranges range1 and range2 (which must both have a step of 1), returns the range formed by the intersection of the two ranges, or
None
if the ranges do not overlap. For example:>>> intersect(range(10), range(5)) range(0, 5) >>> intersect(range(10), range(10, 2)) >>> intersect(range(10), range(2, 5)) range(2, 5)
- piwheels.ranges.split(ranges, i)[source]¶
Given a list of non-overlapping ranges in ascending order, this generator function returns the list with the range containing i split into two ranges, one ending at i and the other starting at i. If i is not contained in any of the ranges, then ranges is returned unchanged. For example:
>>> list(split([range(10)], 5)) [range(0, 5), range(5, 10)] >>> list(split([range(10)], 0)) [range(0, 10)] >>> list(split([range(10)], 20)) [range(0, 10)]