This repository contains the code for writing to and reading from the MongoDB project database.
The purpose of this repo is:
- To provide a consistent vocabulary and structure for the mongodb data model.
- Defining data classes that conform to the datamodel with adequate validation.
- Writing documents to the database that conform to the data model.
- Common helper functions for people wanting to interact with MongoDB programatically.
It is imagined that, whoever is performing the data preprocessing will use this repo to upload their dataset to MongoDB. The only module that should need to be changed is symbols.py, which defines the vocabulary used for a dataset.
It's important to understand the underlying data model that is adopted by the project, which is described below.
To connect to the database, environmental variables must be set before running the code in the repo. Ask the data curator/manager of the project to give you a username and password. The variables that need to be set are as follows:
USER_NAME=your_user
PASSWORD=your_password
IP_ADDRESS=10.0.7.13:37011
DATABASE=txt2snd
ENCRYPTION=SCRAM-SHA-1
In PyCharm, environmental variables can be set as a run configuration by navigating to 'Run>Run...>Edit configurations' and then clicking the + button to add a new python configuration. There you can set the environmental variables as key-value pairs.
You can also simply run MongoDB locally with docker and then only the IP_ADDRESS and DATABASE needs to be specified. This consists of installing docker and then typing in:
docker pull mongo:latest
docker run -d -p 27017:27017 –name=mongo-example mongo:latest
This is all you need to do to download mongodb and run it in a container locally. The IP_ADDRESS would then be 'localhost:27017'. You would need to then create a database and set the DATABASE environmental variable accordingly.
If you want to interact with MongoDB via the MongoDB shell you can run the command:
docker exec -it mongo-example mongo
Otherwise, there are various MongoDB clients such as Studio3T which provides an advanced and useful GUI interface for interacting with MongoDB.
The data model is implemented in model.py. For a detailed description of the data model please read the data model section of this README.md.
A minimal example of creating a document according to this model is shown in the data_model_example.py script.
This essentially consists of:
- Creating fields for sections.
- Creating sections and passing the fields to the sections.
- Creating documents and passing sections to the document.
Under the hood this is simply storing data in a structure that lets us output json in a standard format with various validation along the way to ensure that consistent json is generated.
Here is the minimal example from data_model_example.py.
from symbols import *
from model import MongoDBDocument, MongoDBSection, MongoDBField
descriptive_section_data = {
"tags": ['tag1', 'tag2', 'tag3'],
"text": "Some description"
}
field_data = {"filtered-tags": ['tag1', 'tag2']}
descriptive_field = MongoDBField(name='my_field_name', type=FieldType.text, process_type=ProcessType.explicit,
process='Unprocessed.', sources=[], data=field_data)
print("FIELD")
print(descriptive_field.create_record())
descriptive_section = MongoDBSection(name=SectionType.descriptive,
description='Text and tags that describe audio content.',
original_attributes=descriptive_section_data,
fields=[descriptive_field])
print("DESCRIPTIVE SECTION")
print(descriptive_section.create_record())
audio_section_data = {
"duration": 5.4,
"bitdepth": 32
}
audio_section = MongoDBSection(name=SectionType.audio,
description='Audio data.',
original_attributes=audio_section_data,
fields=[])
print("AUDIO SECTION")
print(audio_section.create_record())
document = MongoDBDocument(audio_path='/some/relative/path',
sections=[descriptive_section, audio_section])
print("DOCUMENT")
print(document.create_record())
You can see that we are incrementally building up the document structure when printing the json output corresponding to the document with the create_record() command. The output of this is as follows:
FIELD
{'field_name': 'my_field_name', 'field_type': 'tags', 'process_type': 'explicit', 'process_description': 'Unprocessed.', 'field_description': 'Tags given by a user.', 'sources': [], 'data': {'filtered-tags': ['tag1', 'tag2']}}
DESCRIPTIVE SECTION
{'descriptive': {'tags': ['tag1', 'tag2', 'tag3'], 'text': 'Some description', 'fields': [{'field_name': 'my_field_name', 'field_type': 'tags', 'process_type': 'explicit', 'process_description': 'Unprocessed.', 'field_description': 'Tags given by a user.', 'sources': [], 'data': {'filtered-tags': ['tag1', 'tag2']}}], 'section_description': 'Text and tags that describe audio content.'}}
AUDIO SECTION
{'audio': {'duration': 5.4, 'bitdepth': 32, 'fields': [], 'section_description': 'Audio data.'}}
DOCUMENT
{'document_id': 'my_id_1', 'audio_path': 'some/relative/path', 'descriptive': {'tags': ['tag1', 'tag2', 'tag3'], 'text': 'Some description', 'fields': [{'field_name': 'my_field_name', 'field_type': 'tags', 'process_type': 'explicit', 'process_description': 'Unprocessed.', 'field_description': 'Tags given by a user.', 'sources': [], 'data': {'filtered-tags': ['tag1', 'tag2']}}], 'section_description': 'Text and tags that describe audio content.'}, 'audio': {'duration': 5.4, 'bitdepth': 32, 'fields': [], 'section_description': 'Audio data.'}}
Check out the examples folder for examples on using the data model and writing to mongodb. Remember to set the environmental variables so that you have database connectivity for the examples that read and write to the database.
Some indexes are added automatically when a new source collection is created to facilitate faster querying. This is largely to help support queries in the dashboard but should help the general sorts of queries users would want to perform.
To support filtering of the data, an additional top-level field is added to each document containing the concatenation of all the tags in the fields of that document. This is used as a text search index by MongoDB by default in this repo. An example of this is shown in write_freesound_example.py.
Symbols.py contains all the enum types that define the data model vocabulary. You should try and reuse these as much as possible for consistency. SectionType.descriptive and SectionType.audio are required for a valid MongoDBDocument but if there is a new section, say 'reviews', feel free to add it.
The enum types are as follows:
- SectionType : Section grouping of key-value pairs.
- ProcessType : The process used to create the data in a field.
- FieldType : The type of data in the field.
- SourceCollection: The name if the source dataset.
When adding a new dataset, you will clearly have to add a new entry into SourceCollection.
The data model is very simple and utilises the schema-free flexibility of MongoDB. MongoDB databases are made up of collections which themselves are made up of a list of documents. Documents are represented as BSON which is a superset of JSON that uses binary encoded structures to increase performance and some additional data types. In other words, it is an efficiently represented JSON with additional objects to facilitate proper database functionality.
Each collection in our project is a dataset of documents that give details about an audio clip. This follows the rough structure:
- document_1
- _id
- audio_path
- section_1
- section-description
- fields
- [field_1, field_2, ..., field_n]
- attribute_1
- attribute_2
- ...
- section_2
- section-description
- fields
- [field_1, field_2, ..., field_n]
- attribute_1
- attribute_2
- ...
- ...
- section_n
- document_2
- ...
Each document has a MongoDB auto-generated id called '_id' and an audio path. The audio_path is unique and has been decided upon and organised by the data curation team. The actual audio is not stored in MongoDB and needs to be downloaded separately. Acquiring model training data from mongodb along with its corresponding audio files will be discussed later in this readme.
Beyond the _id and audio path, there are two important structural features that we adhere to in our data model. These are Sections and Fields.
A section is essentially a grouping of coherent key-value pairs. There are two special keys which must be present in a section which are a section-description and a fields key. The section-description just describes the data in the section. The fields key will be discussed later.
There can be arbitrarily many sections in a document depending on availability of data of a particular dataset. Being able to have arbitrarily many sections is part of the power of schema-less databases of which MongoDB is one. This allows us to add new sections as we acquire new datasets and allows for inherent heterogeneity present in the available datasets to be stored without having to conform to a strict database schema that we would otherwise have to design beforehand.
Having said that... in our data model we still require two sections to be available for all documents to have minimal consistency across different collections. These sections are:
- audio:
- descriptive:
The audio section contains information about the audio file such as duration, format, bitrate etc. While the descriptive section contains text/tags that describe what is in the audio file.
All of the original data that is added to a document is within sections as simple key-value pair. For example, within the descriptive section we might have:
- descriptive:
- section-description: 'Text and tags describing the audio.'
- title : 'whoa nah really?'
- tags : ['whoa', 'really', 'voice', 'surprise']
- description: 'My friend didn't realised that beatles actually had the word "beat" in it and it wasn't spelled "The Beetles". This is him saying "whoa, nahhh, really?"'
- filename 'Man_whoa_surprise.wav'
Where keys: 'title', 'tags', 'description' and 'filename' are 'raw' data by virtual of not being the 'section-description' key and not being within fields. This allows usto preserve the original dataset within documents by a simple convention without having to explicitly mark keys as 'raw'.
The original data might be useful down the road for whatever reason, and it's best to have a copy of it in its original form in the database.
The sections needn't directly correspond to the original dataset structure, so for example, I've added filename under the descriptive section in the above example because it pertains to the contents of the audio.
However, the data used for training models often needs to go through various pre-processing steps and in fact, external resources can be used to augment existing data. This shouldn't be confused with the original raw data. Moreover, when we merge collections from different data sources, we require additional context for data to meaningfully merge across multiple collections. For example, the attribute 'description' in one dataset might correspond to the attribute 'text' in another dataset. There might also be textual attributes that don't really correspond between datasets where no explicit description is given. That brings us to the concept of fields, which allows for collections to have comparable annotated features that are seperated from raw data.
The fields key is a list of objects with a particular structure:
- field-name: the name of the field
- field-type: what type of field it is such as text, tags, mir
- process-name: the name of the process used to create this field
- process-type: the generic name of the type of process such as explicit, extracted, predicted etc.
- process-description: a brief description of what is involved in the processing of this field.
- description: A description of what this field contains.
- sources: a list of field-names if this field was derived from other fields. This may be helpful for understanding the processing involved in creating this field.
- data: json-serialisable data
In reality, the field type and data are the only fundamental attributes needed to work with the data. However, adding descriptions is helpful when working across multiple datasets that users are unfamiliar with, and adding field-types and process-types are useful for querying and filtering documents.
The database consists of original and derived collections. The original collection contains all the data from the original datasets and fields created by whoever was assigned to work on that dataset. Derived collections are constructed from original collections and perhaps also from other derived collections. A simple derived collection might simply be a collection consisting of all documents where the audio is under 5 seconds and where at least 3 tags are available.
Part of the power of storing datasets in collections with some kind of established data model is that it allows for new derived datasets to be created and shared very easily. For simple derived datasets, this can be expressed using the mongodb query language, while for more complex datasets, it may require that users programmatically interact with the database using a MongoDB client such as PyMongo.