OpenSource/telegram-crawler
OpenSource/telegram-crawler
1
0
Fork 0
mirror of https://github.com/MarshalX/telegram-crawler.git synced 2024-11-24 00:07:21 +01:00
Code Issues Projects Releases Packages Wiki Activity
telegram-crawler/data/web/blogfork.telegram.org/mtproto/mtproto-transports.html
GitHub Action 7e2a25065a Update content of files
2023-10-22 15:31:15 +00:00

299 lines
18 KiB
HTML
Raw Blame History

<!DOCTYPE html>
<html class="">
<head>
<meta charset="utf-8">
<title>MTProto transports</title>
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta property="description" content="Here&#39;s a list of MTProto transport protocols (see the ISO/OSI recap for a full explanation):
Abridged
Intermediate
Padded…">
<meta property="og:title" content="MTProto transports">
<meta property="og:image" content="4a92b4139362e9f113">
<meta property="og:description" content="Here&#39;s a list of MTProto transport protocols (see the ISO/OSI recap for a full explanation):
Abridged
Intermediate
Padded…">
<link rel="icon" type="image/svg+xml" href="/img/website_icon.svg?4">
<link rel="apple-touch-icon" sizes="180x180" href="/img/apple-touch-icon.png">
<link rel="icon" type="image/png" sizes="32x32" href="/img/favicon-32x32.png">
<link rel="icon" type="image/png" sizes="16x16" href="/img/favicon-16x16.png">
<link rel="alternate icon" href="/img/favicon.ico" type="image/x-icon" />
<link href="/css/bootstrap.min.css?3" rel="stylesheet">
<link href="/css/telegram.css?236" rel="stylesheet" media="screen">
<style>
</style>
</head>
<body class="preload">
<div class="dev_page_wrap">
<div class="dev_page_head navbar navbar-static-top navbar-tg">
<div class="navbar-inner">
<div class="container clearfix">
<ul class="nav navbar-nav navbar-right hidden-xs"><li class="navbar-twitter"><a href="https://twitter.com/telegram" target="_blank" data-track="Follow/Twitter" onclick="trackDlClick(this, event)"><i class="icon icon-twitter"></i><span> Twitter</span></a></li></ul>
<ul class="nav navbar-nav">
<li><a href="//telegram.org/">Home</a></li>
<li class="hidden-xs"><a href="//telegram.org/faq">FAQ</a></li>
<li class="hidden-xs"><a href="//telegram.org/apps">Apps</a></li>
<li class=""><a href="/api">API</a></li>
<li class="active"><a href="/mtproto">Protocol</a></li>
<li class=""><a href="/schema">Schema</a></li>
</ul>
</div>
</div>
</div>
<div class="container clearfix">
<div class="dev_page">
<div id="dev_page_content_wrap" class=" ">
<div class="dev_page_bread_crumbs"><ul class="breadcrumb clearfix"><li><a href="/mtproto" >Mobile Protocol</a></li><i class="icon icon-breadcrumb-divider"></i><li><a href="/mtproto/mtproto-transports" >MTProto transports</a></li></ul></div>
<h1 id="dev_page_title">MTProto transports</h1>
<div id="dev_page_content"><!-- scroll_nav -->
<p>Here's a list of MTProto transport protocols (<a href="/mtproto#recap">see the ISO/OSI recap for a full explanation</a>):</p>
<ul>
<li><a href="#abridged">Abridged</a></li>
<li><a href="#intermediate">Intermediate</a></li>
<li><a href="#padded-intermediate">Padded intermediate</a></li>
<li><a href="#full">Full</a></li>
</ul>
<p>The server recognizes these different protocols (and distinguishes them from HTTP, too) by the header.
Additionally, the following transport features can be used: </p>
<ul>
<li><a href="#quick-ack">Quick ack</a></li>
<li><a href="#transport-errors">Transport errors</a></li>
<li><a href="#transport-obfuscation">Transport obfuscation</a></li>
</ul>
<p>Example implementations for these protocols can be seen in <a href="https://github.com/tdlib/td/blob/master/td/mtproto/TcpTransport.cpp">tdlib</a> and <a href="https://github.com/danog/MadelineProto/tree/v8/src/Stream/MTProtoTransport">MadelineProto</a>.</p>
<h3><a class="anchor" href="#abridged" id="abridged" name="abridged"><i class="anchor-icon"></i></a>Abridged</h3>
<p>The lightest protocol available.</p>
<ul>
<li>Overhead: Very small </li>
<li>Minimum envelope length: 1 byte</li>
<li>Maximum envelope length: 4 bytes</li>
</ul>
<p>Payload structure:</p>
<pre><code>+-+----...----+
|l| payload |
+-+----...----+
OR
+-+---+----...----+
|h|len| payload +
+-+---+----...----+</code></pre>
<p>Before sending <em>anything</em> into the underlying socket (see <a href="transports">transports</a>), the client must first send <code>0xef</code> as the first byte (the server <strong>will not</strong> send <code>0xef</code> as the first byte in the first reply).<br>
Then, payloads are wrapped in the following envelope:</p>
<ul>
<li>Length: payload length, divided by four, and encoded as a single byte, only if the resulting packet length is a value between <code>0x01..0x7e</code>.</li>
<li>Payload: the MTProto payload</li>
</ul>
<p>If the packet length divided by four is bigger than or equal to 127 (&gt;= <code>0x7f</code>), the following envelope must be used, instead:</p>
<ul>
<li>Header: A single byte of value <code>0x7f</code></li>
<li>Length: payload length, divided by four, and encoded as 3 length bytes (little endian)</li>
<li>Payload: the MTProto payload</li>
</ul>
<h3><a class="anchor" href="#intermediate" id="intermediate" name="intermediate"><i class="anchor-icon"></i></a>Intermediate</h3>
<p>In case 4-byte data alignment is needed, an <em>intermediate</em> version of the original protocol may be used.</p>
<ul>
<li>Overhead: small</li>
<li>Minimum envelope length: 4 bytes</li>
<li>Maximum envelope length: 4 bytes</li>
</ul>
<p>Payload structure: </p>
<pre><code>+----+----...----+
+len.+ payload +
+----+----...----+</code></pre>
<p>Before sending <em>anything</em> into the underlying socket (see <a href="transports">transports</a>), the client must first send <code>0xeeeeeeee</code> as the first int (four bytes, the server <strong>will not</strong> send <code>0xeeeeeeee</code> as the first int in the first reply).<br>
Then, payloads are wrapped in the following envelope:</p>
<ul>
<li>Length: payload length encoded as 4 length bytes (little endian)</li>
<li>Payload: the MTProto payload</li>
</ul>
<h3><a class="anchor" href="#padded-intermediate" id="padded-intermediate" name="padded-intermediate"><i class="anchor-icon"></i></a>Padded intermediate</h3>
<p>Padded version of the <a href="#intermediate">intermediate protocol</a>, to use with <a href="#transport-obfuscation">obfuscation enabled</a> to <strong>bypass ISP blocks</strong>.</p>
<ul>
<li>Overhead: small-medium</li>
<li>Minimum envelope length: random</li>
<li>Maximum envelope length: random</li>
</ul>
<p>Before sending <em>anything</em> into the underlying socket (see <a href="transports">transports</a>), the client must first send <code>0xdddddddd</code> as the first int (four bytes, the server <strong>will not</strong> send <code>0xdddddddd</code> as the first int in the first reply).<br>
Then, payloads are wrapped in the following envelope:</p>
<pre><code>+----+----...----+----...----+
|tlen| payload | padding |
+----+----...----+----...----+</code></pre>
<p>Envelope description:</p>
<ul>
<li>Total length: payload+padding length encoded as 4 length bytes (little endian)</li>
<li>Payload: the MTProto payload</li>
<li>Padding: A random padding string of length <code>0-15</code></li>
</ul>
<h3><a class="anchor" href="#full" id="full" name="full"><i class="anchor-icon"></i></a>Full</h3>
<p>The basic MTProto transport protocol</p>
<ul>
<li>Overhead: medium</li>
<li>Minimum envelope length: 12 bytes (length+seqno+crc)</li>
<li>Maximum envelope length: 12 bytes (length+seqno+crc)</li>
</ul>
<p>Payload structure: </p>
<pre><code>+----+----+----...----+----+
|len.|seq.| payload |crc.|
+----+----+----...----+----+</code></pre>
<p>Envelope description:</p>
<ul>
<li>Length: length+seqno+payload+crc length encoded as 4 length bytes (little endian, the length of the length field must be included, too)</li>
<li>Seqno: the TCP sequence number for this TCP connection (different from the <a href="https://core.telegram.org/mtproto/description#message-sequence-number-msg-seqno">MTProto sequence number</a>): the first packet sent is numbered 0, the next one 1, etc.</li>
<li>payload: MTProto payload</li>
<li>crc: 4 CRC32 bytes computed using length, sequence number, and payload together.</li>
</ul>
<h3><a class="anchor" href="#transport-features" id="transport-features" name="transport-features"><i class="anchor-icon"></i></a>Transport features</h3>
<p>Additionally, the following transport features can be used: </p>
<h4><a class="anchor" href="#quick-ack" id="quick-ack" name="quick-ack"><i class="anchor-icon"></i></a>Quick ack</h4>
<p>These MTProto transport protocols have support for quick acknowledgment.
In this case, the client sets the highest-order length bit in the query packet, and the server responds with a special 4 bytes as a separate packet.
They are the 32 higher-order bits of SHA256 of the encrypted portion of the packet prepended by 32 bytes from the authorization key (the same hash as computed for verifying the message key), with the most significant bit set to make clear that this is not the length of a regular server response packet; if the abridged version is used, bswap is applied to these four bytes.</p>
<h4><a class="anchor" href="#transport-errors" id="transport-errors" name="transport-errors"><i class="anchor-icon"></i></a>Transport errors</h4>
<p>In the event of a transport error (missing auth key, transport flood, etc.), the server may send a packet with a signed little-endian number of 4 bytes, whose <strong>absolute value</strong> contains the error code (the error itself is actually negative). </p>
<p>For example, error Code 403 corresponds to situations where the corresponding HTTP error would have been returned by the HTTP protocol.</p>
<p>Error 404 (auth key not found) is returned when the specified auth key ID cannot be found by the DC.</p>
<p>Error 429 (transport flood) is returned when too many transport connections are established to the same IP in a too short lapse of time, or if any of the <a href="/mtproto/service_messages#simple-container">container</a>/<a href="/mtproto/service_messages_about_messages#acknowledgment-of-receipt">service message limits</a> are reached.</p>
<p>Error 444 (invalid DC) is returned while <a href="/mtproto/auth_key#presenting-proof-of-work-server-authentication">creating an auth key</a>, <a href="#transport-obfuscation">connecting to an MTProxy</a> or in other contexts if an invalid DC ID is specified.</p>
<h4><a class="anchor" href="#transport-obfuscation" id="transport-obfuscation" name="transport-obfuscation"><i class="anchor-icon"></i></a>Transport obfuscation</h4>
<p>Transport obfuscation is required to use the <a href="transports#websocket">websocket transports</a>.</p>
<p>Transport obfuscation to <strong>prevent ISP blocks</strong> is implemented using the following protocol, situated under the MTProto transport in the ISO/OSI stack, see the <a href="/mtproto#recap">recap</a>; this means that the payload is first wrapped in the <a href="/mtproto/mtproto-transports">MTProto transport envelope</a> (all transports are supported), and then obfuscated: </p>
<p>Prior to establishing the connection (and eventually sending the protocol header of a specific <a href="/mtproto/mtproto-transports">MTProto transport</a>), a 64-byte (512-bit) <strong>random</strong> initialization payload is generated.
During the generation process, special care must be taken in order to avoid a situation where that the first int (first four bytes) of the random string are equal to one of the known protocol identifiers (see above).<br>
In particular, the first four bytes must not be equal to <code>0xdddddddd</code> (padded intermediate), <code>0xeeeeeeee</code> (intermediate), <code>POST</code>, <code>GET</code>, <code>HEAD</code>, or any of the HTTP methods that are accepted by the MTProto servers.<br>
The first byte must also not be equal to <code>0xef</code> (abridged).
Bytes <code>4-8</code> must also not be equal to <code>0x00000000</code>, since that would indicate use of the full transport with the initial TCP sequence number (0).</p>
<p>The protocol identifier, if present, must be inserted in the initialization payload at byte offset <code>56</code>: if its length is less than 4, it must be padded using the protocol identifier itself, to make its length 4 (<code>0xef</code> =&gt; <code>0xefefefef</code>): the standalone protocol identifier must be not be sent aftwerwards.</p>
<p>This protocol is also (but not exclusively) used when connecting to MTProxies: <strong>only in this case</strong>, the DC ID in a specially encoded form must also be inserted in the initialization payload at offset <code>60</code>.
The encoding simply consists of the DC ID in two-byte signed little-endian form; <code>10000</code> has to be added to the DC ID to connect to the test servers; it has to be made negative if the DC we're connecting to is a media (not CDN) DC. </p>
<p>Next, a secondary initialization payload is generated by reversing the primary initialization payload. </p>
<p>Two keys are extracted from both initialization payloads, using bytes at offsets <code>8-40</code>: the key extracted from the primary payload is used as encryption key, the key extracted from the secondary payload is used as decryption key.</p>
<p>Two IVs are extracted from both initialization payloads, using bytes at offsets <code>40-56</code>: the IV extracted from the primary payload is used as encryption IV, the IV extracted from the secondary payload is used as decryption IV.</p>
<p><strong>Only if using MTProxy</strong>, the secret is used to provide connection with the MTProxy server.
The secret is a 16-byte string, usually distributed in its hexadecimal form along with the MTProxy host and port in <a href="/api/links#mtproxy-links">proxy deep links »</a>.</p>
<p>Often, a 17-byte version of the secret can be found: this simply indicates that the client should use a specific MTProto transport (based on the first byte, usually it's <code>0xdd</code>, to indicate that the padded intermediate protocol should be used <code>0xdddddddd</code>; however, clients should default to the padded intermediate transport whenever an additional byte in the secret is encountered).</p>
<p>The extracted encryption and decryption keys must be concatenated with the secret (the first byte of which should be ignored if it's the 17-byte version), and the SHA256 hash of such string should be used as encryption/decryption key.</p>
<p>The obtained encryption and decryption key/IV pairs must then be used with <strong>AES-256-CTR</strong> to encrypt and decrypt all outgoing and incoming payloads.<br>
The final value of the encryption/decryption counter after handling an MTProto payload must be used as IV for the next payload, until the TCP/WS connection is closed.<br>
In other words, reuse the two encryption and decryption <strong>AES-256-CTR</strong> OpenSSL instances until the connection is closed. </p>
<p>The first thing that must be encrypted using the encryption key is the initialization payload itself.
Then bytes <code>56-64</code> of the encrypted initialization payload are substituted in the original initialization payload: this is the part that contains the constant MTProto transport protocol identifier and the DC ID (<strong>only for MTProxies</strong>).</p>
<p>The final initialization payload must then be sent in the socket as <strong>first 64 bytes</strong> after the TCP handshake.</p>
<p>Example pseudocode for the generation of an MTProxy connection payload (media DC 4) using the obfuscated <a href="#padded-intermediate">padded intermediate</a> transport.
<strong>Warning</strong>: do not use the specified proxy secret in any MTProxy exposed on the internet.</p>
<pre><code>protocol := 0xdddddddd
dc := 0xfcff
while 1:
init := (56 random bytes) + protocol + dc + (2 random bytes)
if init[0] == 0xef:
continue
first_int := substr(init, 0, 4)
if first_int == 0x44414548 || first_int == 0x54534f50 || first_int == 0x20544547 || first_int == 0x4954504f || first_int == 0x02010316 || first_int == 0xdddddddd || first_int == 0xeeeeeeee:
continue
second_int := substr(init, 4, 4)
if second_int == 0x00000000:
continue
break
initRev := strrev(init)
encryptKey := substr(init, 8, 32)
encryptIV := substr(init, 40, 16)
decryptKey := substr(initRev, 8, 32)
decryptIV := substr(initRev, 40, 16)
secret := substr(0xdd99999999999999999999999999999999, 1, 16)
encryptKey = SHA256(encryptKey + secret)
decryptKey = SHA256(decryptKey + secret)
encryptedInit := CTR(encryptKey, encryptIV, init)
finalInit := substr(init, 0, 56) + substr(encryptedInit, 56, 8)
write(finalInit)</code></pre></div>
</div>
</div>
</div>
<div class="footer_wrap">
<div class="footer_columns_wrap footer_desktop">
<div class="footer_column footer_column_telegram">
<h5>Telegram</h5>
<div class="footer_telegram_description"></div>
Telegram is a cloud-based mobile and desktop messaging app with a focus on security and speed.
</div>
<div class="footer_column">
<h5><a href="//telegram.org/faq">About</a></h5>
<ul>
<li><a href="//telegram.org/faq">FAQ</a></li>
<li><a href="//telegram.org/privacy">Privacy</a></li>
<li><a href="//telegram.org/press">Press</a></li>
</ul>
</div>
<div class="footer_column">
<h5><a href="//telegram.org/apps#mobile-apps">Mobile Apps</a></h5>
<ul>
<li><a href="//telegram.org/dl/ios">iPhone/iPad</a></li>
<li><a href="//telegram.org/android">Android</a></li>
<li><a href="//telegram.org/dl/web">Mobile Web</a></li>
</ul>
</div>
<div class="footer_column">
<h5><a href="//telegram.org/apps#desktop-apps">Desktop Apps</a></h5>
<ul>
<li><a href="//desktop.telegram.org/">PC/Mac/Linux</a></li>
<li><a href="//macos.telegram.org/">macOS</a></li>
<li><a href="//telegram.org/dl/web">Web-browser</a></li>
</ul>
</div>
<div class="footer_column footer_column_platform">
<h5><a href="/">Platform</a></h5>
<ul>
<li><a href="/api">API</a></li>
<li><a href="//translations.telegram.org/">Translations</a></li>
<li><a href="//instantview.telegram.org/">Instant View</a></li>
</ul>
</div>
</div>
<div class="footer_columns_wrap footer_mobile">
<div class="footer_column">
<h5><a href="//telegram.org/faq">About</a></h5>
</div>
<div class="footer_column">
<h5><a href="//telegram.org/blog">Blog</a></h5>
</div>
<div class="footer_column">
<h5><a href="//telegram.org/apps">Apps</a></h5>
</div>
<div class="footer_column">
<h5><a href="/">Platform</a></h5>
</div>
<div class="footer_column">
<h5><a href="//telegram.org/press">Press</a></h5>
</div>
</div>
</div>
</div>
<script src="/js/main.js?47"></script>
<script src="/js/jquery.min.js?1"></script>
<script src="/js/bootstrap.min.js?1"></script>
<script>window.initDevPageNav&&initDevPageNav();
backToTopInit("Go up");
removePreloadInit();
</script>
</body>
</html>
Reference in a new issue View git blame Copy permalink