persistence

Making-your-bot-persistent.md

@@ -7,20 +7,28 @@ In V12.0b1 we added a persistence mechanism to `telegram.ext`. This wiki page is
 - [Storing Bots](#storing-bots)
 
 ## What can become persistent?
-* The persistence structure is designed to make `bot_data`, `chat_data`, `user_data`, `ConversationHandler`'s states and `ExtBot.callback_data_cache` persistent.<br>
-Note that `Handler`'s with `run_async=True` are currently not supported since it uses `telegram.ext.utils.Promise` which can't be serialized
-* `Job`'s and the `job_queue` is not supported because the serialization of callbacks is too unstable to reliably make persistent for broad user-cases. However, the current `JobQueue` backend [APScheduler](https://apscheduler.readthedocs.io/en/stable/) has its own persistence logic that you can leverage.
+* The persistence structure is designed to make
+  * `bot_data`
+  * `chat_data`
+  * `user_data`,
+  * `ConversationHandler`'s states and
+  * `ExtBot.callback_data_cache` persistent.
+
+* `Job`'s and the `job_queue` is not supported.
+However, the current `JobQueue` backend [APScheduler](https://apscheduler.readthedocs.io/en/stable/) has its own persistence logic that you can leverage.
+See e.g. [`ptbcontrib/ptb_sqlalchemy_jobstore`](https://github.com/python-telegram-bot/ptbcontrib/tree/main/ptbcontrib/ptb_sqlalchemy_jobstore)
 * For a special note about `Bot` instances, see [below](#storing-bots)
 
 ## Included persistence classes
 Three classes concerning persistence in bots have been added.  
-* [BasePersistence](https://python-telegram-bot.readthedocs.io/en/latest/telegram.ext.basepersistence.html) - Is an interface class for persistence classes. If you create your own persistence classes to maintain a database-connection for example, you must inherit from `BasePersistence`  
+* [BasePersistence](https://python-telegram-bot.readthedocs.io/en/latest/telegram.ext.basepersistence.html) - Is an interface class for persistence classes.
+If you create your own persistence classes to maintain a database-connection for example, you must inherit from `BasePersistence` and ipmlement all abstract methods
 * [PicklePersistence](https://python-telegram-bot.readthedocs.io/en/latest/telegram.ext.picklepersistence.html) - Uses pickle files to make the bot persistent.  
-* [DictPersistence](https://python-telegram-bot.readthedocs.io/en/latest/telegram.ext.dictpersistence.html) - Uses in memory dicts and easy conversion to and from JSON to make the bot persistent. Note that this class is mainly intended as starting point for custom persistence
-        classes that need to JSON-serialize the stored data before writing them to file/database and does *not* actually write any data to file/database.
+* [DictPersistence](https://python-telegram-bot.readthedocs.io/en/latest/telegram.ext.dictpersistence.html) - Uses in memory dicts and easy conversion to and from JSON to make the bot persistent.
+Note that this class is mainly intended as starting point for custom persistence classes that need to JSON-serialize the stored data before writing them to file/database and does *not* actually write any data to file/database.
 
 ## 3rd party persistence classes
-Instead of manually handling a database to store data, consider implementing a subclass of `BasePersistence`. This allows you to simply pass an instance of that subclass to the `Updater/Dispatcher` and let PTB handle the loading, updating & storing of the data!
+Instead of manually handling a database to store data, consider implementing a subclass of `BasePersistence`. This allows you to simply pass an instance of that subclass to the `Application` and let PTB handle the loading, updating & storing of the data!
 
 If you want to create your own persistence class, please carefully read the docs on [BasePersistence](https://python-telegram-bot.readthedocs.io/en/latest/telegram.ext.basepersistence.html). It will tell you what methods you need to overwrite. 
 
@@ -30,27 +38,41 @@ These 3rd party packages contain persistence classes (the list is incomplete):
 * [python-telegram-bot-django-persistence](https://github.com/GamePad64/python-telegram-bot-django-persistence) - Uses Django ORM to store persistence data. It is most useful for projects, that use PTB and Django.
 
 ## What do I need to change?
+
 To make your bot persistent you need to do the following.
 
 - Create a persistence object (e.g. `my_persistence = PicklePersistence(filename='my_file')`)
-- Construct `Updater` with the persistence (`Updater('TOKEN', persistence=my_persistence, use_context=True)`). If you don't use the `Updater` class, you can pass the persistence directly to the `Dispatcher`.
+- Construct `Application` with the persistence (`Application.builder().token('TOKEN').persistence(persistence=my_persistence).build()`).
 
 This is enough to make `user_data`, `bot_data`, `chat_data` and `ExtBot.callback_data_cache` persistent.
 To make a conversation handler persistent (save states between bot restarts) you **must name it** and set `persistent` to `True`.
-Like `ConversationHandler(<no change>, persistent=True, name='my_name')`. `persistent` is `False` by default.
-Adding these arguments and adding the conversation handler to a persistence-aware updater/dispatcher will make it persistent.
+For example `ConversationHandler(..., persistent=True, name='my_name')`. `persistent` is `False` by default.
+Adding these arguments and adding the conversation handler to a persistence-aware `Application` will make it persistent.
+
+When starting the `Application` with `Application.start()` or `Application.run_{polling, webhook}`, it will automatically update the persistence in regular intervals.
+You can customize the interval via the corresponding argument of `Base/Pickle/Dict/…Persistence`.
 
 ## Refreshing at runtime
 
-If your persistence reads the data from an external database, the entries in this database could change at runtime. This is the case in particular, if the entries in the database are created by a 3rd party service independently of your bot. If you want to make sure that the data in `context.user/chat/bot_data` are always up-to-date, your persistence class should implement the methods `refresh_bot/chat/user_data`. Those will be called when in update comes in, before any of your callbacks are called.
+If your persistence reads the data from an external database, the entries in this database could change at runtime.
+This is the case in particular, if the entries in the database are created by a 3rd party service independently of your bot.
+If you want to make sure that the data in `context.user/chat/bot_data` are always up-to-date, your persistence class should implement the methods `refresh_bot/chat/user_data`.
+Those will be called when in update comes in, before any of your callbacks are called.
+
+These methods can also be useful to implement a lazy-loading strategy.
 
 ## Storing Bots
 
-As of v13, persistence will automatically try to replace `telegram.Bot` instances by [`REPLACED_BOT`](https://python-telegram-bot.readthedocs.io/en/stable/telegram.ext.basepersistence.html#telegram.ext.BasePersistence.REPLACED_BOT) and
-insert the bot set with [`set_bot`](https://python-telegram-bot.readthedocs.io/en/stable/telegram.ext.basepersistence.html#telegram.ext.BasePersistence.set_bot) upon loading of the data. This is to ensure that
-changes to the bot apply to the saved objects, too. For example, you might change the [[default values|Adding-defaults-to-your-bot]] used by the bot. If you change the bots token, this may
-lead to e.g. `Chat not found` errors. For the limitations on replacing bots see
-[`replace_bot`](https://python-telegram-bot.readthedocs.io/en/stable/telegram.ext.basepersistence.html#telegram.ext.BasePersistence.replace_bot) and [`insert_bot`](https://python-telegram-bot.readthedocs.io/en/stable/telegram.ext.basepersistence.html#telegram.ext.BasePersistence.insert_bot).
+Instances of `telegram.Bot` should not be serialized, because changes to the bot don't apply to the serialized object.
 
-This is relevant e.g., if you store Telegram objects like `Message` in `bot/user/chat_data`, as some of them have a `bot` attribute, which holds a reference to the `Dispatchers` bot.
- 
+For example, you might change the [[default values|Adding-defaults-to-your-bot]] used by the bot.
+Or if you change the bots token, this may lead to e.g. `Chat not found` errors.
+This is relevant e.g., if you store Telegram objects like `Message` in `bot/user/chat_data`, as some of them hold a reference to `Apllication.bot` (which is how the shortcuts like `Message.reply_text` work).
+
+The interface class `BaseRequest` does not question what kind of data you supply to its methods.
+Hence, each implementation should take care that it does not try to serialize `telegram.Bot` instances.
+For example, it can check if the data equals the attribute `BasePersistence.bot` (which will be the bot object used by the `Application`) and instead store a placeholder.
+When loading the data, the `BasePersistence.bot` can be reinserted instead of the placeholder.
+Indeed, this is basically what the built-in `PicklePersistence` does.
+
+For more technical details, please refer to the documentation of `{Base, Pickle}Persistence`.

Transition-guide-to-Version-14.0.md

@@ -103,9 +103,14 @@ The argument `users` is now optional as specified by the Bot API.
 
 ### `BasePersistence`
 
+#### Data must be copyable
+
+Any data passed to persistence will be copied with [`copy.deepcopy`](https://docs.python.org/3/library/copy.html#copy.deepcopy).
+This requirement is in place to avoid race conditions.
+
 #### Persisting `telegram.Bot` instances.
 
-In [[Version 13|Transition-guide-to-Version-13.0]], we introduced a mechanism that replaces any `telegram.Bot` instance with a placeholder automatically *before* `update_*_data` was called and inserts the instance back into the return value of `get_*_data`. Unfortunately, this mechanism has proven to be instable and also slow.
+In [[Version 13|Transition-guide-to-Version-13.0]], we introduced a mechanism that replaces any `telegram.Bot` instance with a placeholder automatically *before* `update_*_data` was called and inserts the instance back into the return value of `get_*_data`. Unfortunately, this mechanism has proven to be unstable and also slow.
 
 We have therefore decided to remove this functionality. `Bot` instances should still not be serialized, but handling this is now the responsibility of the specific implementation of `BasePersistence`. For example, `ext.PicklePersistence` uses the built-in functionality of the `pickle` module to achieve the same effect in a more reliable way.