Task Management Thread

The task management is responsible for managing

The whole process is split into several tasks managed by the TaskManagementThread().

The communication is handled via notification to allow continuing reporting of status updates without blocking and even when the connection gets lost in the meanwhile.

Furthermore uploading a file, integrating it into the music directory and importing it into the Music Database is possible in separate steps.

The whole process is split into several tasks. Each task has its own state. The task state is persistently stored inside the uploads directory within a JSON file in a tasks sub-directory. The file name is the task ID (equivalent to the Upload ID) + .json.

Possible Task states:

  • Unrelated states

    • "new": A new created, not yet initialized task.

    • "notexisting" In case a Task ID does not match any file. This task does not exist.

    • "invalidcontent": Processing failed. The content type was unexpected or invalid (Not: artwork, video, album).

    • "remove": Upload is going to be removed. The task itself as well. After this state appears, the task ID should no longer be considered as valid.

  • Upload related states:

    • "waitforchunk": A new chunk of data was requested, and is expected from the client

    • "uploadfailed": The upload failed

    • "uploadcomplete": The whole file is now available in the temporary upload directory and is ready for being preprocessed.

    • "preprocessing": The file is currently in preprocessing state. For example if an archive gets unpacked.

  • Integration related states:

    • "readyforintegration": The integration of the file can be started

    • "startintegration": Start the integration process

    • "integrating": The integration process has started

    • "integrationfailed": Integrating the uploaded file into the music directory failed

  • Import related states:

    • "readyforimport": The uploaded file was successfully integrated into the music directory. The content can now be imported into the MusicDB Database. Everything is now based on file management inside the managed Music Directory. From this state, "remove" gets triggered.

    • "startmusicimport": Importing the integrated file into the music database started

    • "importingmusic": The music import process has started

    • "importfailed": Import process failed (importing the music or generating the artwork)

    • "startartworkimport": Importing succeeded and generating the artwork started

    • "importingartwork": The artwork import process has started

    • "importcomplete": Import process complete and successful

  • File system scanning states:

    • "startfsscan":

    • "scanningfs":

    • "fsscanfailed":

    • "fsscancomplete":

To each task, there are several additional information, also stored in the JSON file. The following keys are in dictionary that represent a task:

  • Unrelated information

    • "id" (str): The task ID

    • "state" (str): One of the task states listed above

    • "contenttype" (str): Type of the content: ("video", "albumfile", "artwork", "any"). An album file can a song but also a booklet PDF, a video clip or any other additional content of an album.

    • "mimetype" (str): MIME-Type of the file (For example "image/png")

    • "annotations" (dict): Additional annotations that can be provided by the user and be optionally used by some task processing.

    • "initializationtime" (int): The unix time stamp when the task has been created

    • "updatetime" (int): Unix time stamp when the task has been updated the last time via UpdateTaskState(). After initialization it is None.

  • Upload related information (May not be valid if there was no upload task)

    • "filesize" (int): The size of the file in Bytes

    • "offset" (int): The amount of existing bytes of a file that is currently being uploaded.

    • "uploadpath" (str): Path of the file in that the uploaded data gets written to Relative to the Upload Directory.

    • "preprocessedpath" (str): Path of preprocessed data. If available, it should be preferred to "uploadpath". The path is relative to the Upload directory.

    • "sourcefilename" (str): Name of the original file that got uploaded.

    • "sourcechecksum" (str): Checksum of the original file that got uploaded.

  • Integration related information (May not be valid if the file is not yet integrated into the Music Directory)

    • "videopath" (str): Path to a video file, relative to the Music Directory

    • "albumpath" (str): Path to an album directory, relative to the Music Directory

    • "albumfilepath" (str): Path to file of an album, relative to the Music Directory

  • Artwork related

    • "awsourcetype" (str): Type of the artwork source: "imagefile", "songfile" or "videofile".

    • "awsourcepath" (str): Path to the artwork source

After upload is complete, the Management Thread takes care about post processing or removing no longer needed content The uploaded file follows the following naming scheme: contenttype + - + checksum + . + source-file-extension The upload manager also takes care about the validity of the uploaded file (via SHA-1 checksum).

After an import is completed successfully, all clients connected to the server get a "sys:refresh" notification to update their caches. This notification is generated via musicdb.mdbapi.server.UpdateCaches().

Task Management

TaskManager Class

class musicdb.taskmanagement.taskmanager.TaskManager(config, database)[source]

This is a base class that provides a common interface used by the TaskManagementThread() to perform the upload, integration and import tasks.

Tasks are stored as JSON file in the MusicDB data directory inside a sub directory tasks. When a user removed one of the JSON files inside that directory, the task will be set into the remove state internally. This state update will be stored in the previous removed file, so that this file appears short after removing. Anyway, as soon as the remove state got processed, the task file and related uploaded files will be removed again by MusicDB.

Parameters
Raises

TypeError – When the arguments are not of the correct type.

CreateNewTask()[source]

This method returns an empty but initialized task. The task state will be set to "new". The task ID will be created by CreateTaskID(). All other entries are set to None.

The new task will not be saved and not scheduled!

Returns

A new task dictionary

CreateTaskID()[source]

This method creates a new Task ID. In detail, it is a Version 4 Universally Unique Identifier (UUID) . It will be returned as a string.

Returns

A UUID to be used as entry ID

ExistsTaskFile(task)[source]

This method checks if the task file related to task exists in the tasks directory. If it is so, True gets returned. This method can be used to check if the task has been removed by the user. Reasons for the user to remove a task file can be fixing a stuck process.

This method does not more than checking if the task exists in the file system. It will not add or remove the task from the global task list processed by the TaskManagementThread()

Parameters

task (dict) – The task to check

Returns

True if the tasks exists in the file system, otherwise False

GetTaskByID(taskid)[source]

This method returns an existing task from the tasklist. The task gets identified by its ID aka Task ID

When the task does not exits, the clients get an "InternalError" notification. The tasks state is then "notexisting".

Parameters

taskid (str) – ID of the task

Returns

A task dictionary

Raises

  • TypeError – When taskid is not a string

  • ValueError – When taskid is not a valid key in the Tasks-dictionary

GetTasks()[source]
Returns

The dictionary with all upload tasks

InitiateProcess(taskid, mimetype, contenttype, filesize, checksum, sourcefilename, initialstate)[source]

Initiates an upload of a file into a MusicDB managed file space. After calling this method, a notification gets triggered to request the first chunk of data from the clients. In case uploads are deactivated in the MusicDB Configuration, an "InternalError" Notification gets sent to the clients.

Parameters

  • taskid (str) – Unique ID to identify the upload task

  • mimetype (str) – MIME-Type of the file (example: "image/png")

  • contenttype (str) – Type of the content: ("video", "album", "artwork")

  • filesize (int) – Size of the complete file in bytes

  • checksum (str) – SHA-1 check sum of the source file

  • sourcefilename (str) – File name (example: "test.png")

  • initialstate (str) – The Initial state of this process. See Import Tasks Management.

Raises

  • TypeError – When one of the arguments has not the expected type

  • ValueError – When contenttype does not have the expected values

LoadTasks()[source]

Loads all task from the JSON files inside the tasks-directory. The list of active tasks will be replaced by the loaded tasks.

Returns

Nothing

NotifyClient(notification, task, message=None)[source]

This method triggers a client-notification.

There are three kind of notifications:

  • "ChunkRequest": A new chunk of data is requested

  • "StateUpdate": The state or annotations of an upload-task has been changed. See "state" value.

  • "InternalError": There is an internal error occurred during. See "message" value.

The notification comes with the current status of the upload process. This includes the following keys - independent of the state of the upload:

  • taskid: ID of the upload the notification is associated with

  • offset: Offset of the requested data in the source file

  • chunksize: The maximum chunk size

  • state: The current state of the upload task

  • message: null/None or a message from the server

  • task: The task dictionary itself

  • tasklist: Except for ChunkRequest events, the WebSocket server append the result of musicdb.lib.ws.mdbwsi.MusicDBWebSocketInterface.GetCurrentTasks() to the notification

task can be None in case the notification is meant to be an information that a given upload ID is invalid.

Parameters

  • notification (str) – Name of the notification

  • task (dict) – Task structure

  • message (str) – (optional) text message (like an error message) to the client

Returns

Nothing

Raises

ValueError – When notification has an unknown notification name

RegisterCallback(function)[source]

Register a callback function that reacts on Upload, Integration or Import related events. For more details see the module description at the top of this document.

The function must expect two parameters: The notification type (a string) and a dictionary with the status. Details can be found in the NotifyClient() description.

Parameters

function – A function that shall be called on an event.

Returns

Nothing

RemoveCallback(function)[source]

Removes a function from the list of callback functions.

Parameters

function – A function that shall be called removed.

Returns

Nothing

RemoveTask(taskid)[source]

This method removed a task and all temporary data that belongs to it. It also removes the task from the tasks list.

Temporary files are "uploadpath", "preprocessedpath"

Parameters

taskid (str) – ID Of the task that shall be removed

Returns

Nothing

SaveTask(task)[source]

This method saves a task in the uploads directory under tasks/${Task ID}.json

Parameters

task (dict) – The task to save

Returns

Nothing

ScheduleTask(task)[source]

This method adds a new task into the list of tasks that will be processed by the ManagementThread().

Parameters

task (dict) – A new task

Returns

Nothing

UpdateTaskState(task, state, errormessage=None)[source]

This method updates and saves the state of an task. An "StateUpdate" notification gets send as well. If the task already is in the state, nothing happens. The "updatetime" value will be updated to the current unix time stamp.

If errormessage is not None, the notification gets send as "InternalError" with the message

Parameters

  • task (dict) – Task object to update

  • state (str) – New state

  • message (str) – Optional message

Returns

Nothing