RSS scraper application which saves RSS feeds to a database and lets a user view and manage feeds they’ve added to the system through an API.
User can sign up in the system, follow/unfollow new or existing feeds, list followed feeds, get followed feeds' content, mark content as seen and ignore it. Feeds can be added to the system, feed content might be displayed, force update of a feed might be performed, failed to update feeds might be listed. Feeds can be found in https://rss.com/blog/popular-rss-feeds/
- GET /docs - Swagger with all provided endpoints
User Interface:
- POST /wancheez/register - Create a user with name wancheez
- POST /wancheez/follow {"feed_url": "http://rss.cnn.com/rss/cnn_topstories.rss"} - Follow (and register) new Feed by User
- GET /wancheez/feeds - Get user's feed list
- GET /wancheez/feeditems - Get items from the feeds to which user has subscribed
- POST /wancheez/feed_item/mark_read? {"item_id": "https://www.cnn.com/kamala-party/index.html"} - Mark feed item as read
- GET /wancheez/feeditems?unread=True - Get unread items from the feeds to which user has subscribed
- GET /wancheez/feed/items?unread=True&feed_id=640dce2a4ecda5b5b2fc2b6c - Get items of the particular feed (feed retrieved from /wancheez/feeds)
- DELETE /wancheez/unfollow?feed_id=640dce2a4ecda5b5b2fc2b6c - Unfollow feed
Feed Interface:
- POST /feed {"url": https://lifehacker.com/rss} - Add feed to the database
- GET /feeds - Get all registered feeds
- GET /feed/items?feed_url=https://lifehacker.com/rss - Get feed items of particular feed
- POST /feed/update {"feed_url": "https://lifehacker.com/rss"} - Force update of the feed
- GET /feeds/failed - Get all failed to update feeds
Feeds' content is being synchronized in an unattended manner with help of update-service. Period of synchronization based on ttl-parameter of a feed. In a several failed attempts, the synchronization process will be stopped. Manual synchronization will be required. Failed to update feeds might be found in /feeds/failed.
docker-compose build
docker-compose up
In the root dir:
pytest
The service is written with Python 3.x. MongoDB is used to store feeds info. MongoEngine as ORM. Feed synchronization is implemented in the separate service with pydantic. Redis is used as a broker for pydantic and storing attempts info. The service is designed to be loose coupled with Representation, Model and Service layers.
If required, synchronization might be adjusted in the conf/conf.py file.
Implement authorization with password. Switch to async MongoDB ORM.