OpenSource/telegram-crawler
OpenSource/telegram-crawler
1
0
Fork 0
mirror of https://github.com/MarshalX/telegram-crawler.git synced 2024-11-25 00:38:33 +01:00
Code Issues Projects Releases Packages Wiki Activity
telegram-crawler/data/web/blogfork.telegram.org/mtproto/mtproto-transports.html
GitHub Action c3a7e68b0b Update content of files
2024-02-14 12:10:59 +00:00

341 lines
24 KiB
HTML
Raw Blame History

<!DOCTYPE html>
<html class="">
<head>
<meta charset="utf-8">
<title>MTProto transports</title>
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta property="description" content="Here&#39;s a list of MTProto transport protocols (see the ISO/OSI recap for a full explanation):
Abridged
Intermediate
Padded…">
<meta property="og:title" content="MTProto transports">
<meta property="og:image" content="4a92b4139362e9f113">
<meta property="og:description" content="Here&#39;s a list of MTProto transport protocols (see the ISO/OSI recap for a full explanation):
Abridged
Intermediate
Padded…">
<link rel="icon" type="image/svg+xml" href="/img/website_icon.svg?4">
<link rel="apple-touch-icon" sizes="180x180" href="/img/apple-touch-icon.png">
<link rel="icon" type="image/png" sizes="32x32" href="/img/favicon-32x32.png">
<link rel="icon" type="image/png" sizes="16x16" href="/img/favicon-16x16.png">
<link rel="alternate icon" href="/img/favicon.ico" type="image/x-icon" />
<link href="/css/bootstrap.min.css?3" rel="stylesheet">
<link href="/css/telegram.css?236" rel="stylesheet" media="screen">
<style>
</style>
</head>
<body class="preload">
<div class="dev_page_wrap">
<div class="dev_page_head navbar navbar-static-top navbar-tg">
<div class="navbar-inner">
<div class="container clearfix">
<ul class="nav navbar-nav navbar-right hidden-xs"><li class="navbar-twitter"><a href="https://twitter.com/telegram" target="_blank" data-track="Follow/Twitter" onclick="trackDlClick(this, event)"><i class="icon icon-twitter"></i><span> Twitter</span></a></li></ul>
<ul class="nav navbar-nav">
<li><a href="//telegram.org/">Home</a></li>
<li class="hidden-xs"><a href="//telegram.org/faq">FAQ</a></li>
<li class="hidden-xs"><a href="//telegram.org/apps">Apps</a></li>
<li class=""><a href="/api">API</a></li>
<li class="active"><a href="/mtproto">Protocol</a></li>
<li class=""><a href="/schema">Schema</a></li>
</ul>
</div>
</div>
</div>
<div class="container clearfix">
<div class="dev_page">
<div id="dev_page_content_wrap" class=" ">
<div class="dev_page_bread_crumbs"><ul class="breadcrumb clearfix"><li><a href="/mtproto" >Mobile Protocol</a></li><i class="icon icon-breadcrumb-divider"></i><li><a href="/mtproto/mtproto-transports" >MTProto transports</a></li></ul></div>
<h1 id="dev_page_title">MTProto transports</h1>
<div id="dev_page_content"><!-- scroll_nav -->
<p>Here's a list of MTProto transport protocols (<a href="/mtproto#recap">see the ISO/OSI recap for a full explanation</a>):</p>
<ul>
<li><a href="#abridged">Abridged</a></li>
<li><a href="#intermediate">Intermediate</a></li>
<li><a href="#padded-intermediate">Padded intermediate</a></li>
<li><a href="#full">Full</a></li>
</ul>
<p>The server recognizes these different protocols (and distinguishes them from HTTP, too) by the header.
Additionally, the following transport features can be used: </p>
<ul>
<li><a href="#quick-ack">Quick ack</a></li>
<li><a href="#transport-errors">Transport errors</a></li>
<li><a href="#transport-obfuscation">Transport obfuscation</a></li>
</ul>
<p>Example implementations for these protocols can be seen in <a href="https://github.com/tdlib/td/blob/master/td/mtproto/TcpTransport.cpp">tdlib</a> and <a href="https://github.com/danog/MadelineProto/tree/v8/src/Stream/MTProtoTransport">MadelineProto</a>.</p>
<h3><a class="anchor" href="#abridged" id="abridged" name="abridged"><i class="anchor-icon"></i></a>Abridged</h3>
<p>The lightest protocol available.</p>
<ul>
<li>Overhead: Very small </li>
<li>Minimum envelope length: 1 byte</li>
<li>Maximum envelope length: 4 bytes</li>
</ul>
<p>Payload structure:</p>
<pre><code>+-+----...----+
|l| payload |
+-+----...----+
OR
+-+---+----...----+
|h|len| payload +
+-+---+----...----+</code></pre>
<p>Before sending <em>anything</em> into the underlying socket (see <a href="transports">transports</a>), the client must first send <code>0xef</code> as the first byte (the server <strong>will not</strong> send <code>0xef</code> as the first byte in the first reply).<br>
Then, payloads are wrapped in the following envelope:</p>
<p>If the packet length divided by four is smaller than 127:</p>
<ul>
<li>Length: payload length, divided by four, and encoded as a single byte</li>
<li>Payload: the MTProto payload</li>
</ul>
<p>If the packet length divided by four is bigger than or equal to 127, the following envelope must be used, instead:</p>
<ul>
<li>Header: A single byte of value <code>0x7f</code> (127)</li>
<li>Length: payload length, divided by four, and encoded as 3 length bytes (little endian)</li>
<li>Payload: the MTProto payload</li>
</ul>
<p><a href="#quick-ack">Quick ACK »</a> may be enabled for this transport. </p>
<p>To request a quick ACK from the server for an encrypted MTProto payload, use the following envelope for <em>outgoing</em> messages, instead of the one specified above.</p>
<p>If the packet length divided by four is smaller than 127:</p>
<ul>
<li>Length: payload length, divided by four, plus 128, and encoded as a single byte (<code>(length/2)+128</code> or <code>(length &gt;&gt; 2) | (1 &lt;&lt; 7)</code>); i.e. the most significant bit must be set.</li>
<li>Payload: the MTProto payload</li>
</ul>
<p>If the packet length divided by four is bigger than or equal to 127, the following envelope must be used, instead:</p>
<ul>
<li>Header: A single byte of value <code>0xff</code> (255)</li>
<li>Length: payload length, divided by four, and encoded as 3 length bytes (little endian); same as for a normal abridged envelope.</li>
<li>Payload: the MTProto payload</li>
</ul>
<p>The server will send quick ACK tokens by bswapping them (i.e. inverting the order of the 4 bytes of the ACK token) and sending them as a standalone 4-byte packet without a length header. </p>
<pre><code>+----+
|dcba|
+----+</code></pre>
<p>These quick ACK packets can be easily distinguished from normal abridged packets because the first byte will always have the most-significant bit set (because quick ACK tokens have the most-significant bit of the last byte set, and since they are bswapped the last byte will come first), and the length/header of normal payload packets coming from the server is always less than or equal to 127 (thus the most-significant bit is not set for normal payloads). </p>
<h3><a class="anchor" href="#intermediate" id="intermediate" name="intermediate"><i class="anchor-icon"></i></a>Intermediate</h3>
<p>In case 4-byte data alignment is needed, an <em>intermediate</em> version of the original protocol may be used.</p>
<ul>
<li>Overhead: small</li>
<li>Minimum envelope length: 4 bytes</li>
<li>Maximum envelope length: 4 bytes</li>
</ul>
<p>Payload structure: </p>
<pre><code>+----+----...----+
+len.+ payload +
+----+----...----+</code></pre>
<p>Before sending <em>anything</em> into the underlying socket (see <a href="transports">transports</a>), the client must first send <code>0xeeeeeeee</code> as the first int (four bytes, the server <strong>will not</strong> send <code>0xeeeeeeee</code> as the first int in the first reply).<br>
Then, payloads are wrapped in the following envelope:</p>
<ul>
<li>Length: payload length encoded as 4 length bytes (little endian)</li>
<li>Payload: the MTProto payload</li>
</ul>
<p><a href="#quick-ack">Quick ACK »</a> may be enabled for this transport. </p>
<p>To request a quick ACK from the server for an encrypted MTProto payload, add <code>0x80000000</code> to the <code>len</code> field before encoding it (equivalent to doing <code>len = len | (1 &lt;&lt; 31)</code>, i.e. set the most-significant bit of the length). </p>
<p>The server will send quick ACK tokens as a standalone 4-byte packet without a length header.</p>
<pre><code>+----+
|abcd|
+----+</code></pre>
<p>These quick ACK packets can be easily distinguished from normal intermediate packets because quick ACK tokens always have the most-significant bit of the last byte set, and trying to decode an ACK token as a little-endian 32-bit integer will always yield a value bigger than or equal to <code>0x80000000</code>, which can never be a valid packet length. </p>
<h3><a class="anchor" href="#padded-intermediate" id="padded-intermediate" name="padded-intermediate"><i class="anchor-icon"></i></a>Padded intermediate</h3>
<p>Padded version of the <a href="#intermediate">intermediate protocol</a>, to use with <a href="#transport-obfuscation">obfuscation enabled</a> to <strong>bypass ISP blocks</strong>.</p>
<ul>
<li>Overhead: small-medium</li>
<li>Minimum envelope length: random</li>
<li>Maximum envelope length: random</li>
</ul>
<p>Before sending <em>anything</em> into the underlying socket (see <a href="transports">transports</a>), the client must first send <code>0xdddddddd</code> as the first int (four bytes, the server <strong>will not</strong> send <code>0xdddddddd</code> as the first int in the first reply).<br>
Then, payloads are wrapped in the following envelope:</p>
<pre><code>+----+----...----+----...----+
|tlen| payload | padding |
+----+----...----+----...----+</code></pre>
<p>Envelope description:</p>
<ul>
<li>Total length: payload+padding length encoded as 4 length bytes (little endian)</li>
<li>Payload: the MTProto payload</li>
<li>Padding: A random padding string of length <code>0-15</code></li>
</ul>
<p><a href="#quick-ack">Quick ACK »</a> may be enabled for this transport. </p>
<p>To request a quick ACK from the server for an encrypted MTProto payload, add <code>0x80000000</code> to the <code>len</code> field before encoding it (equivalent to doing <code>len = len | (1 &lt;&lt; 31)</code>, i.e. set the most-significant bit of the length). </p>
<p>The server will send quick ACK tokens as a standalone 8 to 16-byte packet (excluding the length of the packet itself, encoded as usual), containing a 4-byte header with all bits set, followed by the ACK token (<code>abcd</code>), followed by 0 to 8 random padding bytes.</p>
<pre><code>+----+----+----+----...----+
|tlen|FFFF|abcd| padding |
+----+----+----+----...----+</code></pre>
<p>These quick ACK packets can be easily distinguished from normal intermediate padded packets because their length will be always smaller than or equal to 16, smaller than any MTProto packet.<br>
They can be distinguished from transport errors because the first 4 bytes of the payload are equal to <code>0xFFFFFFFF</code>, not a valid transport error. </p>
<h3><a class="anchor" href="#full" id="full" name="full"><i class="anchor-icon"></i></a>Full</h3>
<p>The basic MTProto transport protocol</p>
<ul>
<li>Overhead: medium</li>
<li>Minimum envelope length: 12 bytes (length+seqno+crc)</li>
<li>Maximum envelope length: 12 bytes (length+seqno+crc)</li>
</ul>
<p>Payload structure: </p>
<pre><code>+----+----+----...----+----+
|len.|seq.| payload |crc.|
+----+----+----...----+----+</code></pre>
<p>Envelope description:</p>
<ul>
<li>Length: length+seqno+payload+crc length encoded as 4 length bytes (little endian, the length of the length field must be included, too)</li>
<li>Seqno: the TCP sequence number for this TCP connection (different from the <a href="https://core.telegram.org/mtproto/description#message-sequence-number-msg-seqno">MTProto sequence number</a>): the first packet sent is numbered 0, the next one 1, etc.</li>
<li>payload: MTProto payload</li>
<li>crc: 4 CRC32 bytes computed using length, sequence number, and payload together.</li>
</ul>
<h3><a class="anchor" href="#transport-features" id="transport-features" name="transport-features"><i class="anchor-icon"></i></a>Transport features</h3>
<p>Additionally, the following transport features can be used: </p>
<h4><a class="anchor" href="#quick-ack" id="quick-ack" name="quick-ack"><i class="anchor-icon"></i></a>Quick ack</h4>
<p>Some of the TCP transports listed above support quick ACKs: quick ACKs are a way for clients to get quick receipt acknowledgements for packets. </p>
<p>To request a quick ack for a specific outgoing payload, clients must set the MSB of an appropriate field in the transport envelope (as described in the documentation for each transport protocol above). </p>
<p>Also, clients must generate and store a quick ACK token, associating it with the outgoing MTProto payload, by:</p>
<ul>
<li>Taking the first 32 bits of the SHA256 of the encrypted portion of the payload prepended by 32 bytes from the authorization key (the same hash generated when computing the <a href="/mtproto/description#message-key-msg-key">message key</a>, except that instead of taking the middle 128 bits, the first 32 bits are taken instead). </li>
<li>Setting the MSB of the last byte to 1: in other words, treat the 32 bits generated above as a little-endian integer, then add <code>0x80000000</code> to it (i.e. <code>ack_token = msg_key_long[0:4] | (1 &lt;&lt; 31)</code> on a little-endian system). </li>
</ul>
<p>Once the payload is successfully received, decrypted and accepted for processing by the server, the server will send back the same quick ACK token we generated above, using the encoding described in the documentation for each transport protocol. </p>
<p>Note that reception of a quick ACK <strong>does not</strong> indicate that any of the RPC queries contained in the message have succeeded, failed or finished execution at all, it simply indicates that they have been received, decrypted and accepted for processing by the server. </p>
<p>The server will still send <code>msgs_ack</code> constructors for content-related constructors and methods contained in payloads which were quick ACKed, as well as replies/errors for methods and constructors, as usual. </p>
<h4><a class="anchor" href="#transport-errors" id="transport-errors" name="transport-errors"><i class="anchor-icon"></i></a>Transport errors</h4>
<p>In the event of a transport error (missing auth key, transport flood, etc.), the server may send a packet (framed as an MTProto payload by the chosen MTProto transport) with a signed little-endian number of 4 bytes, whose <strong>absolute value</strong> contains the error code (the error itself is actually negative). </p>
<p>For example, error Code 403 corresponds to situations where the corresponding HTTP error would have been returned by the HTTP protocol. </p>
<p>Error 404 (auth key not found) is returned when the specified auth key ID cannot be found by the DC, during the initial MTProto handshake if any of the specified queries is incorrect, or during normal operation for example if some MTProto fields are incorrect (i.e. the MTProto packet length is bigger than the transport-specified packet length, and so on). </p>
<p>Error 429 (transport flood) is returned when too many transport connections are established to the same IP in a too short lapse of time, or if any of the <a href="/mtproto/service_messages#simple-container">container</a>/<a href="/mtproto/service_messages_about_messages#acknowledgment-of-receipt">service message limits</a> are reached. </p>
<p>Error 444 (invalid DC) is returned while <a href="/mtproto/auth_key#presenting-proof-of-work-server-authentication">creating an auth key</a>, <a href="#transport-obfuscation">connecting to an MTProxy</a> or in other contexts if an invalid DC ID is specified. </p>
<p>When using the <a href="/mtproto/transports#http">HTTP</a>/<a href="/mtproto/transports#https">HTTPS</a> transports, transport errors are not transmitted as specified above, instead they are simply returned as normal HTTP status codes (and the HTTP payload must be ignored). </p>
<h4><a class="anchor" href="#transport-obfuscation" id="transport-obfuscation" name="transport-obfuscation"><i class="anchor-icon"></i></a>Transport obfuscation</h4>
<p>Transport obfuscation is required to use the <a href="transports#websocket">WebSocket transports</a>.</p>
<p>Transport obfuscation to <strong>prevent ISP blocks</strong> is implemented using the following protocol, situated under the MTProto transport in the ISO/OSI stack, see the <a href="/mtproto#recap">recap</a>; this means that the payload is first wrapped in the <a href="/mtproto/mtproto-transports">MTProto transport envelope</a> (all transports are supported), and then obfuscated: </p>
<p>Prior to establishing the connection (and eventually sending the protocol header of a specific <a href="/mtproto/mtproto-transports">MTProto transport</a>), a 64-byte (512-bit) <strong>random</strong> initialization payload is generated.
During the generation process, special care must be taken in order to avoid a situation where that the first int (first four bytes) of the random string are equal to one of the known protocol identifiers (see above).<br>
In particular, the first four bytes must not be equal to <code>0xdddddddd</code> (padded intermediate), <code>0xeeeeeeee</code> (intermediate), <code>POST</code>, <code>GET</code>, <code>HEAD</code>, or any of the HTTP methods that are accepted by the MTProto servers.<br>
The first byte must also not be equal to <code>0xef</code> (abridged).
Bytes <code>4-8</code> must also not be equal to <code>0x00000000</code>, since that would indicate use of the full transport with the initial TCP sequence number (0).</p>
<p>The protocol identifier, if present, must be inserted in the initialization payload at byte offset <code>56</code>: if its length is less than 4, it must be padded using the protocol identifier itself, to make its length 4 (<code>0xef</code> =&gt; <code>0xefefefef</code>): the standalone protocol identifier must be not be sent aftwerwards.</p>
<p>This protocol is also (but not exclusively) used when connecting to MTProxies: <strong>only in this case</strong>, the DC ID in a specially encoded form must also be inserted in the initialization payload at offset <code>60</code>.
The encoding simply consists of the DC ID in two-byte signed little-endian form; <code>10000</code> has to be added to the DC ID to connect to the test servers; it has to be made negative if the DC we're connecting to is a media (not CDN) DC. </p>
<p>Next, a secondary initialization payload is generated by reversing the primary initialization payload. </p>
<p>Two keys are extracted from both initialization payloads, using bytes at offsets <code>8-40</code>: the key extracted from the primary payload is used as encryption key, the key extracted from the secondary payload is used as decryption key.</p>
<p>Two IVs are extracted from both initialization payloads, using bytes at offsets <code>40-56</code>: the IV extracted from the primary payload is used as encryption IV, the IV extracted from the secondary payload is used as decryption IV.</p>
<p><strong>Only if using MTProxy</strong>, the secret is used to provide connection with the MTProxy server.
The secret is a 16-byte string, usually distributed in its hexadecimal form along with the MTProxy host and port in <a href="/api/links#mtproxy-links">proxy deep links »</a>.</p>
<p>Often, a 17-byte version of the secret can be found: this simply indicates that the client should use a specific MTProto transport (based on the first byte, usually it's <code>0xdd</code>, to indicate that the padded intermediate protocol should be used <code>0xdddddddd</code>; however, clients should default to the padded intermediate transport whenever an additional byte in the secret is encountered).</p>
<p>The extracted encryption and decryption keys must be concatenated with the secret (the first byte of which should be ignored if it's the 17-byte version), and the SHA256 hash of such string should be used as encryption/decryption key.</p>
<p>The obtained encryption and decryption key/IV pairs must then be used with <strong>AES-256-CTR</strong> to encrypt and decrypt all outgoing and incoming payloads.<br>
The final value of the encryption/decryption counter after handling an MTProto payload must be used as IV for the next payload, until the TCP/WS connection is closed.<br>
In other words, reuse the two encryption and decryption <strong>AES-256-CTR</strong> OpenSSL instances until the connection is closed. </p>
<p>The first thing that must be encrypted using the encryption key is the initialization payload itself.
Then bytes <code>56-64</code> of the encrypted initialization payload are substituted in the original initialization payload: this is the part that contains the constant MTProto transport protocol identifier and the DC ID (<strong>only for MTProxies</strong>).</p>
<p>The final initialization payload must then be sent in the socket as <strong>first 64 bytes</strong> after the TCP handshake.</p>
<p>Example pseudocode for the generation of an MTProxy connection payload (media DC 4) using the obfuscated <a href="#padded-intermediate">padded intermediate</a> transport.<br>
<strong>Warning</strong>: do not use the specified proxy secret in any MTProxy exposed on the internet.</p>
<pre><code>protocol := 0xdddddddd
dc := 0xfcff
while True:
init := (56 random bytes) + protocol + dc + (2 random bytes)
if init[0] == 0xef:
continue
first_int := substr(init, 0, 4)
if first_int == 0x44414548 || first_int == 0x54534f50 || first_int == 0x20544547 || first_int == 0x4954504f || first_int == 0x02010316 || first_int == 0xdddddddd || first_int == 0xeeeeeeee:
continue
second_int := substr(init, 4, 4)
if second_int == 0x00000000:
continue
break
initRev := strrev(init)
encryptKey := substr(init, 8, 32)
encryptIV := substr(init, 40, 16)
decryptKey := substr(initRev, 8, 32)
decryptIV := substr(initRev, 40, 16)
secret := substr(0xdd99999999999999999999999999999999, 1, 16)
encryptKey = SHA256(encryptKey + secret)
decryptKey = SHA256(decryptKey + secret)
encryptedInit := CTR(encryptKey, encryptIV, init)
finalInit := substr(init, 0, 56) + substr(encryptedInit, 56, 8)
write(finalInit)</code></pre></div>
</div>
</div>
</div>
<div class="footer_wrap">
<div class="footer_columns_wrap footer_desktop">
<div class="footer_column footer_column_telegram">
<h5>Telegram</h5>
<div class="footer_telegram_description"></div>
Telegram is a cloud-based mobile and desktop messaging app with a focus on security and speed.
</div>
<div class="footer_column">
<h5><a href="//telegram.org/faq">About</a></h5>
<ul>
<li><a href="//telegram.org/faq">FAQ</a></li>
<li><a href="//telegram.org/privacy">Privacy</a></li>
<li><a href="//telegram.org/press">Press</a></li>
</ul>
</div>
<div class="footer_column">
<h5><a href="//telegram.org/apps#mobile-apps">Mobile Apps</a></h5>
<ul>
<li><a href="//telegram.org/dl/ios">iPhone/iPad</a></li>
<li><a href="//telegram.org/android">Android</a></li>
<li><a href="//telegram.org/dl/web">Mobile Web</a></li>
</ul>
</div>
<div class="footer_column">
<h5><a href="//telegram.org/apps#desktop-apps">Desktop Apps</a></h5>
<ul>
<li><a href="//desktop.telegram.org/">PC/Mac/Linux</a></li>
<li><a href="//macos.telegram.org/">macOS</a></li>
<li><a href="//telegram.org/dl/web">Web-browser</a></li>
</ul>
</div>
<div class="footer_column footer_column_platform">
<h5><a href="/">Platform</a></h5>
<ul>
<li><a href="/api">API</a></li>
<li><a href="//translations.telegram.org/">Translations</a></li>
<li><a href="//instantview.telegram.org/">Instant View</a></li>
</ul>
</div>
</div>
<div class="footer_columns_wrap footer_mobile">
<div class="footer_column">
<h5><a href="//telegram.org/faq">About</a></h5>
</div>
<div class="footer_column">
<h5><a href="//telegram.org/blog">Blog</a></h5>
</div>
<div class="footer_column">
<h5><a href="//telegram.org/apps">Apps</a></h5>
</div>
<div class="footer_column">
<h5><a href="/">Platform</a></h5>
</div>
<div class="footer_column">
<h5><a href="//telegram.org/press">Press</a></h5>
</div>
</div>
</div>
</div>
<script src="/js/main.js?47"></script>
<script src="/js/jquery.min.js?1"></script>
<script src="/js/bootstrap.min.js?1"></script>
<script>window.initDevPageNav&&initDevPageNav();
backToTopInit("Go up");
removePreloadInit();
</script>
</body>
</html>
Reference in a new issue View git blame Copy permalink