Clean up build of input index file#234
Conversation
| def __init__ (self, config): | ||
| self.config = config | ||
| self.data = InputData.read() | ||
| self.info_files = InputInfoCollection.read() |
There was a problem hiding this comment.
This line creates a list of all info files which are present in the input folder (step 1 in comment).
| self.info_files = InputInfoCollection.read() | ||
|
|
||
| def run(self): | ||
| self._validate_info_files() |
There was a problem hiding this comment.
Here, all info files which are present in the input folder are validated (step 2 in comment).
|
|
||
| def run(self): | ||
| self._validate_info_files() | ||
| self._make_index_file_for_input() |
There was a problem hiding this comment.
This line creates the input index file and stores it (step 3 in comment).
| resource.location = str(info_filename.parent.relative_to(self.config.in_path) / dataset) | ||
| resources.append(resource) | ||
|
|
||
| self.index = GammaCatResourceIndex(resources).sort() |
There was a problem hiding this comment.
I am not sure whether this definition of the class instance variable self.index is pythonic or not. Currently, self.index is not used anywhere but I think in the future we will handle a lot of things via this index, hence, it is nice to have it as a class instance variable.
There was a problem hiding this comment.
If it's not used from elsewhere at the moment, better to keep it as a local variable only.
Use InputCollection class in make.py
Use InputInfoCollection to build index file
| step = self.config.step | ||
| if step == 'all': | ||
| self.process_all() | ||
| elif step == 'input-index': |
There was a problem hiding this comment.
This is not working anymore because it is not handled in this class.
| return InputData.read() | ||
|
|
||
| def process_all(self): | ||
| self.make_index_file_for_input() |
There was a problem hiding this comment.
Same as above. Not part of this class anymore.
| step=step, | ||
| ) | ||
| CollectionMaker(config).run() | ||
| if (step == 'input-index'): |
There was a problem hiding this comment.
I know, this looks ugly and need follow up change! But it works, hence, I would leave it as it is for now and change it later.
|
I don't understand the error of travis ci. It is a http error? |
| resource.location = str(info_filename.parent.relative_to(self.config.in_path) / dataset) | ||
| resources.append(resource) | ||
|
|
||
| self.index = GammaCatResourceIndex(resources).sort() |
There was a problem hiding this comment.
If it's not used from elsewhere at the moment, better to keep it as a local variable only.
|
Honestly, I don't know whether you can merge this now after merging #211 :-D |
2 participants
This PR is on top of #211. Hence, you should firstly review and merge #211 before looking at the diff of this PR!
The plan is to have to classes
InputCollectionandOutputCollectioneach handling - as the names suggests - every data in the input and output folders, respectively.This PR starts to write the
InputCollectionand it implements to first things:info.yamlfiles (stored in input folder) is created via the classInputInfoCollection(see inline comment).input_index_fileis created after (!) the info-files are validated (see inline comment).Step 2) and 3) are handled via a
runfunction inInputCollection(see inline comment).In the future these three steps should be the very first thing
make.pydoes! This PR does not (!) change the procedure ofmake.py, it only introduces a ugly way to run the upper steps whenmake.py collection --step input-indexis executed (see inline comment). There needs to be done a follow up PR which cleans upmake.py.The second small commit is only a small fix in the
_make_index_file_inputfunction so that it uses theto_list()function ofInputInfoCollection. But the functionality is not changed.