Old Engine/File Store - Revision history

Pazaz at 01:31, 24 January 2022

2022-01-24T01:31:45Z

← Older revision		Revision as of 21:31, 23 January 2022
Line 1:		Line 1:
	This cache format is a sector-based virtual file system. It was used from [https://runescape.fandom.com/wiki/Update:New_game_update_system revision 234] (2004) to [https://runescape.fandom.com/wiki/Update:Game_engine_upgraded! revision 402] (2006). Before revision 234, [[Old Engine/File Archive\|File Archives]] were used instead.		This cache format is a sector-based virtual file system. It was used from [https://runescape.fandom.com/wiki/Update:New_game_update_system revision 234] (2004) to [https://runescape.fandom.com/wiki/Update:Game_engine_upgraded! revision 402] (2006). Before revision 234, [[Old Engine/File Archive\|File Archives]] were used instead. See [[Jagex Store]] for the next iteration of the cache format.

	It uses two file extensions: dat and idx. Idx should have the store ID following it, i.e. idx0. They represent “Data” and “Index,” the index is a lookup table for each entry with the minimum data necessary to begin reading an entry.		It uses two file extensions: dat and idx. Idx should have the store ID following it, i.e. idx0. They represent “Data” and “Index,” the index is a lookup table for each entry with the minimum data necessary to begin reading an entry.

Pazaz at 21:03, 6 December 2021

2021-12-06T21:03:41Z

@@ Line 38: / Line 38: @@
 ==Example==
 <syntaxhighlight lang="javascript">
-async read(file) {
+read(index = 0, entry = 0, extract = false) {
-	let temp = new Uint8Array(520);
+    this.idx[index].front().seek(entry * 6);
-	this.idx.seek(file * 6);
+    let data = new ByteBuffer(new Uint8Array(entrySize));
-	let size = ((temp[0] & 0xFF) << 16) + ((temp[1] & 0xFF) << 8) + (temp[2] & 0xFF);
+    while (data.available) {
-	let sector = ((temp[3] & 0xFF) << 16) + ((temp[4] & 0xFF) << 8) + (temp[5] & 0xFF);
+        this.dat.front().seek(sectorAt * 520);
-	if (size > this.maxFileSize) {
+    if (extract && index > 0) {
-		return new Uint8Array();
+        return new ByteBuffer(zlib.gunzipSync(data.raw)).raw;
+    }
-	if (sector <= 0 || sector > await this.dat.length() / 520) {
+    return data.raw;
+}
 </syntaxhighlight>

Pazaz: /* Format */

2021-09-08T05:55:26Z

Format

← Older revision		Revision as of 01:55, 8 September 2021
Line 26:		Line 26:
	Data sectors are aligned to 520-byte boundaries. The beginning 8 bytes are used as the sector header and the following 512 bytes will be the data for that sector. The last sector is not guaranteed to have a full 512 bytes and the format relies on the remaining entry size here instead.		Data sectors are aligned to 520-byte boundaries. The beginning 8 bytes are used as the sector header and the following 512 bytes will be the data for that sector. The last sector is not guaranteed to have a full 512 bytes and the format relies on the remaining entry size here instead.

	'''~~The~~ sector header uses 8 bytes per sector:'''		'''A sector header uses 8 bytes per sector:'''

	*2 bytes for the entry ID		*2 bytes for the entry ID

Pazaz: /* Format */

2021-09-08T05:52:20Z

Format

← Older revision		Revision as of 01:52, 8 September 2021
Line 17:		Line 17:
	This format does not have the ability to store file names so everything is typically referenced using a numerical identifier.		This format does not have the ability to store file names so everything is typically referenced using a numerical identifier.

	== Format ==		==Format==
	The index file uses 6 bytes per entry:		'''The index file uses 6 bytes per entry:'''

	* 3 bytes for the ~~file~~ size		*3 bytes for the entry size

	* 3 bytes for the starting data sector to begin reading at		*3 bytes for the starting data sector to begin reading at

	Data sectors are aligned to 520-byte boundaries. The beginning 8 bytes are used as the sector header and the following 512 bytes will be the data for that sector. The last sector is not guaranteed to have a full 512 bytes and the format relies on the remaining ~~file~~ size here instead.		Data sectors are aligned to 520-byte boundaries. The beginning 8 bytes are used as the sector header and the following 512 bytes will be the data for that sector. The last sector is not guaranteed to have a full 512 bytes and the format relies on the remaining entry size here instead.

	The sector header uses 8 bytes per sector:		'''The sector header uses 8 bytes per sector:'''

	* 2 bytes for the ~~file~~ ID		*2 bytes for the entry ID

	* 2 bytes for the current ~~file~~ part		*2 bytes for the current entry part

	* 3 bytes for the next sector to continue reading at		*3 bytes for the next sector to continue reading at

	* 1 byte for the store ID		*1 byte for the store ID

	== Example ==		==Example==
	<syntaxhighlight lang="javascript">		<syntaxhighlight lang="javascript">
	async read(file) {		async read(file) {

Pazaz at 05:47, 8 September 2021

2021-09-08T05:47:52Z

← Older revision		Revision as of 01:47, 8 September 2021
Line 1:		Line 1:
	This file ~~format,~~ .~~dat/.idx0-4,~~ was ~~introduced in~~ [https://runescape.fandom.com/wiki/Update:New_game_update_system revision 234]~~. It was replaced by .dat2/idx0-255 after~~ [https://runescape.fandom.com/wiki/Update:Game_engine_upgraded! revision ~~400~~]. ~~It allowed Jagex to push updates without forcing clients to redownload everything compared to using the original~~ [[Old Engine/File Archive\|File ~~Archive~~]] ~~format alone. File archives~~ were ~~inside of .idx0 when this format was~~ used.		This cache format is a sector-based virtual file system. It was used from [https://runescape.fandom.com/wiki/Update:New_game_update_system revision 234] (2004) to [https://runescape.fandom.com/wiki/Update:Game_engine_upgraded! revision 402] (2006). Before revision 234, [[Old Engine/File Archive\|File Archives]] were used instead.

	.~~idx files are essentially~~ lookup ~~tables~~ for the ~~.dat store~~.		It uses two file extensions: dat and idx. Idx should have the store ID following it, i.e. idx0. They represent “Data” and “Index,” the index is a lookup table for each entry with the minimum data necessary to begin reading an entry.

	~~A cache entry uses 6 bytes in the respective idx file~~:		'''The three main advantages of this format are:'''

	*3 bytes for the ~~file size~~		1) The client can incrementally update its local cache without having to re-download the entire cache each time.

	*3 bytes for the starting dat sector		2) The client can load files on-demand, as well as preload most files ahead of time.

	~~A dat sector~~ can be up to ~~520 bytes long, 8 bytes for~~ the ~~header and~~ the ~~remaining bytes for the cache entry data~~.		3) Maps and midis can be saved to the cache instead of being streamed from the server.

	A sector ~~header looks like:~~		It has an artificial size limit of 52.8 MB for main_file_cache.dat, the technical limit is around ~8.7GB. This value is calculated by taking the maximum unsigned value of a 24-bit integer and multiplying it by 520 (the size of a data sector).

	*2 bytes defining the file ID		Each entry also has an artificial size limit that was increased as time went on. In 317 this limitation was 500 KB, by 377 the limitation was raised to 600 KB.
	*2 bytes defining the file part (how many have been read)
	*3 bytes defining the next sector (seek here)
	*1 byte defining the ~~sector store (~~.~~idx file used)~~

	Example:<syntaxhighlight lang="javascript">		This format does not have the ability to store file names so everything is typically referenced using a numerical identifier.

			== Format ==
			The index file uses 6 bytes per entry:

			* 3 bytes for the file size

			* 3 bytes for the starting data sector to begin reading at

			Data sectors are aligned to 520-byte boundaries. The beginning 8 bytes are used as the sector header and the following 512 bytes will be the data for that sector. The last sector is not guaranteed to have a full 512 bytes and the format relies on the remaining file size here instead.

			The sector header uses 8 bytes per sector:

			* 2 bytes for the file ID

			* 2 bytes for the current file part

			* 3 bytes for the next sector to continue reading at

			* 1 byte for the store ID

			== Example ==
			<syntaxhighlight lang="javascript">
	async read(file) {		async read(file) {
	let temp = new Uint8Array(520);		let temp = new Uint8Array(520);

Pazaz at 05:02, 8 September 2021

2021-09-08T05:02:53Z

← Older revision		Revision as of 01:02, 8 September 2021
Line 1:		Line 1:
	This file format, .dat/.idx0-4, was introduced in [https://runescape.fandom.com/wiki/Update:New_game_update_system revision 234]. It was replaced by .dat2/idx0-255 after [https://runescape.fandom.com/wiki/Update:Game_engine_upgraded! revision 400]. It allowed Jagex to push updates without forcing clients to redownload everything compared to using the original [[Old Engine/File Archive\|File Archive]] format alone. File archives were ~~moved~~ inside of .idx0.		This file format, .dat/.idx0-4, was introduced in [https://runescape.fandom.com/wiki/Update:New_game_update_system revision 234]. It was replaced by .dat2/idx0-255 after [https://runescape.fandom.com/wiki/Update:Game_engine_upgraded! revision 400]. It allowed Jagex to push updates without forcing clients to redownload everything compared to using the original [[Old Engine/File Archive\|File Archive]] format alone. File archives were inside of .idx0 when this format was used.

	.idx files are essentially lookup tables for the .dat store.		.idx files are essentially lookup tables for the .dat store.
Line 5:		Line 5:
	A cache entry uses 6 bytes in the respective idx file:		A cache entry uses 6 bytes in the respective idx file:

	* 3 bytes for the file size		*3 bytes for the file size

	* 3 bytes for the starting dat sector		*3 bytes for the starting dat sector

	A dat sector can be up to 520 bytes long, 8 bytes for the header and the remaining bytes for the cache entry data.		A dat sector can be up to 520 bytes long, 8 bytes for the header and the remaining bytes for the cache entry data.
Line 13:		Line 13:
	A sector header looks like:		A sector header looks like:

	* 2 bytes defining the file ID		*2 bytes defining the file ID
	* 2 bytes defining the file part (how many have been read)		*2 bytes defining the file part (how many have been read)
	* 3 bytes defining the next sector (seek here)		*3 bytes defining the next sector (seek here)
	* 1 byte defining the sector store (.idx file used)		*1 byte defining the sector store (.idx file used)

	Example:<syntaxhighlight lang="javascript">		Example:<syntaxhighlight lang="javascript">

Pazaz at 05:01, 8 September 2021

2021-09-08T05:01:52Z

← Older revision		Revision as of 01:01, 8 September 2021
Line 1:		Line 1:
	This file format, .dat/.~~idx~~, was introduced in [https://runescape.fandom.com/wiki/Update:New_game_update_system revision 234].		This file format, .dat/.idx0-4, was introduced in [https://runescape.fandom.com/wiki/Update:New_game_update_system revision 234]. It was replaced by .dat2/idx0-255 after [https://runescape.fandom.com/wiki/Update:Game_engine_upgraded! revision 400]. It allowed Jagex to push updates without forcing clients to redownload everything compared to using the original [[Old Engine/File Archive\|File Archive]] format alone. File archives were moved inside of .idx0.

	~~It allowed Jagex~~ to ~~push updates without forcing clients to redownload everything~~.		.idx files are essentially lookup tables for the .dat store.

			A cache entry uses 6 bytes in the respective idx file:

			* 3 bytes for the file size

			* 3 bytes for the starting dat sector

			A dat sector can be up to 520 bytes long, 8 bytes for the header and the remaining bytes for the cache entry data.

			A sector header looks like:

			* 2 bytes defining the file ID
			* 2 bytes defining the file part (how many have been read)
			* 3 bytes defining the next sector (seek here)
			* 1 byte defining the sector store (.idx file used)

			Example:<syntaxhighlight lang="javascript">
			async read(file) {
			let temp = new Uint8Array(520);

			this.idx.seek(file * 6);
			await this.idx.read(temp, 0, 6);

			let size = ((temp[0] & 0xFF) << 16) + ((temp[1] & 0xFF) << 8) + (temp[2] & 0xFF);
			let sector = ((temp[3] & 0xFF) << 16) + ((temp[4] & 0xFF) << 8) + (temp[5] & 0xFF);

			if (size > this.maxFileSize) {
			return new Uint8Array();
			}

			if (sector <= 0 \|\| sector > await this.dat.length() / 520) {
			return new Uint8Array();
			}

			let data = new Uint8Array(size);
			let position = 0;

			for (let part = 0; position < size; part++) {
			if (sector === 0) {
			return new Uint8Array();
			}

			let available = size - position;
			if (available > 512) {
			available = 512;
			}

			this.dat.seek(sector * 520);
			await this.dat.read(temp, 0, available + 8);

			let sectorFile = ((temp[0] & 0xFF) << 8) + (temp[1] & 0xFF);
			let sectorPart = ((temp[2] & 0xFF) << 8) + (temp[3] & 0xFF);
			let nextSector = ((temp[4] & 0xFF) << 16) + ((temp[5] & 0xFF) << 8) + (temp[6] & 0xFF);
			let sectorStore = temp[7] & 0xFF;

			if (sectorFile !== file \|\| sectorPart !== part \|\| sectorStore !== this.store) {
			return new Uint8Array();
			}

			if (nextSector < 0 \|\| nextSector > await this.dat.length() / 520) {
			return new Uint8Array();
			}

			for (let i = 0; i < available; i++) {
			data[position++] = temp[i + 8];
			}

			sector = nextSector;
			}
			return data;
			}
			</syntaxhighlight>

Pazaz at 02:08, 8 September 2021

2021-09-08T02:08:25Z

← Older revision		Revision as of 22:08, 7 September 2021
Line 1:		Line 1:
	52.~~8 MB max~~ .~~dat size~~		This file format, .dat/.idx, was introduced in [https://runescape.fandom.com/wiki/Update:New_game_update_system revision 234].

	~~600 KB max entry size~~		It allowed Jagex to push updates without forcing clients to redownload everything.

Pazaz at 05:03, 7 September 2021

2021-09-07T05:03:07Z

← Older revision	Revision as of 01:03, 7 September 2021
Line 1:	Line 1:
		52.8 MB max .dat size

		600 KB max entry size

Pazaz: Created blank page

2021-09-07T02:52:16Z

Created blank page

New page