Add option to transfer post-processing scripts

[goerz]

I have a folder that contains a python script that I want to use for post-processing (doctr deploy --command ...), along with python modules that the script depends on. Since doctr runs the --command from within the deploy branch (gh-pages), that folder has to be available in doctr's checkout of gh-pages. However, I'd very much like to keep the folder on my master branch, not gh-pages, for a number of reasons:

• Chicken-and-egg problem: I can't set up doctr on a new repo that doesn't have a gh-pages branch yet: I'd have to create gh-pages manually and place the post-processing script in it
• The code for building the docs shouldn't be separated across different branches: the build-commands in my .travis.yml and the post-processing script are very much interdependent, so for any changes to the doc-building process I'd have to work simultaneously in two different branches of the project
• A contributor to the project (or myself after a couple of months, having forgotten the details of doctr) might be confused about where to find the post-processing script, and would probably not think to look on the gh-pages branch.
• Philosophically, I think of gh-pages as a "deployment branch" that I don't want to have to touch by hand.

I would propose to add e.g. a --transfer flag to the doctr command that would transfer files/folders from the build-branch (master/version-tag) to the deploy branch (gh-pages). The transferred files/folders should not be commited to gh-pages.

For the time being, I'm trying to work around this by copying the folder containing the post-processing script (.travis) to the deploy directory (cp -r .travis docs/build/html/_doctr) in my .travis.yml before the call to doctr deploy. However, I'm having a very hard time preventing the _doctr directory from being pushed to gh-pages. Isn't that what --exclude _doctr should do?

[asmeurer]

The --exclude thing is probably a bug.

--transfer as you describe it is already what the deploy directory is. I think we just need to add support for multiple deploy directories/files, and fix --exclude so that it works (and also document this whole thing in the recipes docs).

[goerz]

Do you mean multiple --built-docs (#221)? So if I had the documentation in docs/build/html and my post-processing scripts in .travis, I'd invoke doctr deploy as something like

doctr deploy --built-docs docs/build/html --built-docs .travis --exclude .travis $DEPLOY_DIR

Actually, that seems weird, since currently (if I undestand correctly) --built-docs and $DEPLOY_DIR form a pair, in that the folder specified by --build-docs will be copied to the $DEPLOY_DIR on the gh-pages branch. You'd need multiple pairings. It also wasn't quite clear to me from the documentation of --exclude whether currently its argument is relative to $DEPLOY_DIR or to the root of the gh-pages. It would seem that if there are multiple --built-docs/$DEPLOY_DIR, the --exclude would have to be relative to the root.

To be honest, I don't have a clear mental picture of how doctr deploy operates. It seems like something along the lines of

• create an empty temporary directory (let's call it root) and initialize it as a new temporary branch
• copy the --build-docs directory to root/$DEPLOY_DIR and git add it
• run --command with root as the current-working-directly, which can make changes within root and git add them
• create a commit on the temporary branch
• cherry-pick that commit onto the gh-pages branch and push

That wouldn't delete files that are missing in the --build-docs directory from gh-pages, though (which I think doctr does, so the above model likely isn't accurate).

Here's the simpler model that I initially thought was what doctr was doing (but definitely not what's happening):

• check out gh-pages to a temporary folder
• git rm -rf $DEPLOY_DIR in the gh-pages checkout if it exists
• copy the --built-docs directory to the $DEPLOY_DIR in gh-pages
• run the --command, which can modify any files (it wouldn't have to git add them if this model was correct)
• Commit on gh-pages (with git commit -a) and push

I have no idea what the --no-sync and --no-temp-dir options do. Anyway, sorry for ranting, and if I'm a bit thick on what you're proposing. It might be a good idea to give an overview of the correct mental model in the documentation, though.

For now, I've managed to get by post-processing to work by using

--command="git show $TRAVIS_COMMIT:.travis/docs_post_process.py > post_process.py && git show $TRAVIS_COMMIT:.travis/versions.py > versions.py && python post_process.py"

[goerz]

Reading the source code was illuminating regarding the deployment procedure. The missing piece was the .doctr-files log using during synchronization, and the fact that the --command runs after synchronization, but before git add/git rm. I created PR #356 to make this part of the documentation