Upload Management

The communication is handled via notification to allow continuing uploading even when the connection gets lost in the meanwhile.

The upload is performed chunk-wise. After initiating an Upload, this upload manager requests chunks of data via MusicDB Notifications from the clients. All clients are informed about the upload process, not only the client that initiated the upload. So each client can show the progress and state.

Task states:

  • "waitforchunk": A new chunk of data was requested, and is expected from the client

  • "uploadcomplete": The whole file is now available in the temporary upload directory

  • "uploadfailed": The upload failed

  • "notexisting" virtual state in case an Upload ID does not match an Upload. This task does not exist.

  • "preprocessed": The uploaded file was successfully pre-processed and is ready for importing

  • "invalidcontent": Pre-processing failed. The content was unexpected or invalid.

  • "integrated": The uploaded file was successfully integrated into the music directory

  • "integrationfailed": Integrating the uploaded file into the music directory failed

  • "startimport": Importing the integrated file into the music database started

  • "importfailed": Import process failed (importing the music or generating the artwork)

  • "importartwork": Importing succeeded and generating the artwork started

  • "importcomplete": Import process complete and successful

  • "remove": Upload is going to be removed - after this state appears, the task ID should no longer be considered as valid.

After upload is complete, the Management Thread takes care about post processing or removing no longer needed content

The uploaded file follows the following naming scheme: contenttype + - + checksum + . + source-file-extension

The upload manager also takes care about the validity of the uploaded file (via SHA-1 checksum).

The task state is persistently stored inside the uploads directory within a JSON file in a tasks sub-directory. The file name is the task ID (equivalent to the Upload ID) + .json.

UploadManager Class

class mdbapi.uploadmanager.UploadManager(config, database)[source]

This class manages uploading content to the server MusicDB runs on. All data is stored in the uploads-directory configured in the MusicDB configuration.

Parameters
Raises

TypeError – When the arguments are not of the correct type.

AnnotateUpload(uploadid, annotations)[source]

This method can be used to add additional information to an upload. This can be done during or after the upload process.

Parameters

uploadid (str) – ID to identify the upload

Returns

True on success, otherwise False

Raises

  • TypeError – When uploadid is not of type str

  • ValueError – When uploadid is not included in the Task Queue

GetTaskByID(uploadid)[source]

This method returns an existing task from the tasklist. The task gets identified by its ID aka Upload ID

When the task does not exits, the clients get an "InternalError" notification. The tasks state is then "notexisting".

Parameters

uploadid (str) – ID of the upload-task

Returns

A task dictionary

Raises

  • TypeError – When uploadid is not a string

  • ValueError – When uploadid is not a valid key in the Tasks-dictionary

GetTasks()[source]
Returns

The dictionary with all upload tasks

ImportArtwork(task)[source]

Task state must be "startimport" and content type must be "artwork"

Returns

True on success.

ImportVideo(task)[source]

Task state must be "startimport" and content type must be "video"

Returns

True on success.

ImportVideoArtwork(task)[source]
Returns

True on success

InitiateUpload(uploadid, mimetype, contenttype, filesize, checksum, sourcefilename)[source]

Initiates an upload of a file into a MusicDB managed file space. After calling this method, a notification gets triggered to request the first chunk of data from the clients. In case uploads are deactivated in the MusicDB Configuration, an "InternalError" Notification gets sent to the clients.

Parameters

  • uploadid (str) – Unique ID to identify the upload task

  • mimetype (str) – MIME-Type of the file (example: "image/png")

  • contenttype (str) – Type of the content: ("video", "album", "artwork")

  • filesize (int) – Size of the complete file in bytes

  • checksum (str) – SHA-1 check sum of the source file

  • sourcefilename (str) – File name (example: "test.png")

Raises

  • TypeError – When one of the arguments has not the expected type

  • ValueError – When contenttype does not have the expected values

IntegrateUploadedFile(uploadid, triggerimport)[source]

This method integrated the uploaded files into the music directory. The whole file tree will be created following the MusicDB naming scheme.

The upload task must be in preprocesses state. If not, nothing happens.

When triggerimport is true, the upload manager start importing the music. This happens asynchronously inside the Upload Manager Thread.

Parameters

uploadid (str) – ID to identify the upload

Returns

True on success, otherwise False

Raises

  • TypeError – When uploadid is not of type str

  • ValueError – When uploadid is not included in the Task Queue

IntegrateVideo(task)[source]

When an annotation needed for creating the video file path in the music directory is missing, False gets returned and an error message written into the log

LoadTasks()[source]

Loads all task from the JSON files inside the tasks-directory. The list of active tasks will be replaced by the loaded tasks.

Returns

Nothing

NewChunk(uploadid, rawdata)[source]

This method processes a new chunk received from the uploading client.

Parameters

  • uploadid (str) – Unique ID to identify the upload task

  • rawdata (bytes) – Raw data to append to the uploaded data

Returns

False in case an error occurs. Otherwise True.

Raises

TypeError – When rawdata is not of type bytes

NotifyClient(notification, task, message=None)[source]

This method triggers a client-notification.

There are three kind of notifications:

  • "ChunkRequest": A new chunk of data is requested

  • "StateUpdate": The state or annotations of an upload-task has been changed. See "state" value.

  • "InternalError": There is an internal error occurred during. See "message" value.

The notification comes with the current status of the upload process. This includes the following keys - independent of the state of the upload:

  • uploadid: ID of the upload the notification is associated with

  • offset: Offset of the requested data in the source file

  • chunksize: The maximum chunk size

  • state: The current state of the upload task

  • message: null/None or a message from the server

  • uploadtask: The task dictionary itself

  • uploadslist: Except for ChunkRequest events, the WebSocket server append the result of lib.ws.mdbwsi.MusicDBWebSocketInterface.GetUploads() to the notification

task can be None in case the notification is meant to be an information that a given upload ID is invalid.

Parameters

  • notification (str) – Name of the notification

  • task (dict) – Task structure

  • message (str) – (optional) text message (like an error message) to the client

Returns

Nothing

Raises

ValueError – When notification has an unknown notification name

PreProcessArtwork(task)[source]
Parameters

task (dict) – the task object of an upload-task

PreProcessUploadedFile(task)[source]

This method initiates pre-processing of an uploaded file. Depending on the contenttype different post processing methods are called:

The task must be in "uploadcomplete" state, otherwise nothing happens but printing an error message. If post processing was successful, the task state gets updated to "preprocessed". When an error occurred, the state will become "invalidcontent".

Parameters

task (dict) – the task object of an upload-task

Returns

True on success, otherwise False

PreProcessVideo(task)[source]
Parameters

task (dict) – the task object of an upload-task

RegisterCallback(function)[source]

Register a callback function that reacts on Upload related events. For more details see the module description at the top of this document.

Parameters

function – A function that shall be called on an event.

Returns

Nothing

RemoveCallback(function)[source]

Removes a function from the list of callback functions.

Parameters

function – A function that shall be called removed.

Returns

Nothing

RemoveTask(task)[source]

tasks/${Task ID}.json

RequestRemoveUpload(uploadid)[source]

This method triggers removing a specific upload. This includes the uploaded file as well as the upload task information and annotations.

The upload task can be in any state. When the remove-operation is triggered, its state gets changed to "remove".

Only the "remove" state gets set. Removing will be done by the Management Thread.

Parameters

uploadid (str) – ID of the upload-task

Returns

True on success

SaveTask(task)[source]

This method saves a task in the uploads directory under tasks/${Task ID}.json

Parameters

task (dict) – The task to save

Returns

Nothing

UpdateTaskState(task, state, errormessage=None)[source]

This method updates and saves the state of an task. An "StateUpdate" notification gets send as well.

If errormessage is not None, the notification gets send as "InternalError" with the message

Parameters

  • task (dict) – Task object to update

  • state (str) – New state

  • message (str) – Optional message

Returns

Nothing

UploadCompleted(task)[source]

This method continues the file management after an upload was completed. The following tasks were performed:

  • Checking the checksum of the destination file (SHA1) and compares it with the "sourcechecksum" from the task-dict.

When the upload was successful, it notifies the clients with a "UploadComplete" notification. Otherwise with a "UploadFailed" one.

Parameters

task (dict) – The task that upload was completed

Returns

True When the upload was successfully complete, otherwise False