Skip to content

Adds a filter that guarantees to return batches ending with a line br… - #119

Open
edim24 wants to merge 1 commit into
t1gor:masterfrom
edim24:master
Open

Adds a filter that guarantees to return batches ending with a line br…#119
edim24 wants to merge 1 commit into
t1gor:masterfrom
edim24:master

Conversation

@edim24

@edim24 edim24 commented Dec 23, 2024

Copy link
Copy Markdown
Contributor

In case of very large robots.txt files (https://edu-medic.ru/robots.txt) standard batch size (4096 bytes) can lead to incorrect tree building. Because line can be divided at random position.

At the very beginning of filters chain new filter was added. It ensures, that all next filters will receive lines ending with EOL symbol.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant