In other languages:

Desynchronization: Difference between revisions

From Official Factorio Wiki
Jump to navigation Jump to search
No edit summary
(→‎Comparing game state & save files: clarify the less human readable numbers)
 
(9 intermediate revisions by 2 users not shown)
Line 1: Line 1:
{{Languages}}
{{Languages}}
A desynchronization (desync) is when during multiplayer a player's game becomes desynchronised from the Factorio server. Their game has become different from the server's game.
A desynchronization (desync) is when during multiplayer a player's game becomes desynchronized from the Factorio server. Their game has become different from the server's game.


== What are desyncs ==
== What are desyncs ==


Desyncs can be caused either by a bug in the core Factorio game code or by a bug in a mod/scenario. They should not occur during regular activity.
Desyncs can be caused either by a bug in the core Factorio game code or by a bug in a mod/scenario. They should not occur during regular play.


=== Factorio multiplayer architecture ===
=== Factorio multiplayer architecture ===
Factorio multiplayer code uses Deterministic Lockstep to synchronize clients, this is a method of synchronizing a game from one computer to another by sending only the user inputs that control that game, rather than networking the state of the objects in the game itself. It means that all players need to simulate every single tick of the game identically. If any computer does something ever-so-slightly different, a desynchronization (desync) occurs. The game includes processes to ensure any issues with network traffic does not cause desyncs.
Factorio multiplayer code uses deterministic lockstep to synchronize clients. This is a method of synchronizing a game from one computer to another by sending only the user inputs that control that game, rather than networking the state of the objects in the game itself. It means that all player's games need to simulate every single tick of the game identically. If any computer does something ever-so-slightly different, a desynchronization (desync) occurs. The game includes processes to ensure any issues with network traffic do not cause desyncs.


=== Player experience of desyncs ===
=== Player experience of desyncs ===
Upon a desync between a player and server occurring that player immediately disconnects from the server and starts to reconnect. This reconnection isn't to allow the player to play the game, but to obtain the server's copy of the save game (map and game state) to include in a desync report. After the reconnection is complete the player is offered the options to save the report, quit or reconnect to the server to play.
Upon a player's game detecting a desync to the server, it immediately disconnects from the server and re-downloads the save game. This download isn't to allow the player to play the game, but to obtain the server's current copy of the save game (map and game state) to include in a desync report. After the download is complete and a desync report has been generated the player is offered the options to view the report, quit or reconnect to the server to play.
 
[[File:Desync report notification.jpg]]
[[File:Desync report notification.jpg]]


The desync report can be provided to either the mod/scenario makers or the Factorio game developers to help them resolve the bug.
The desync report can be provided to either the mod/scenario makers or the Factorio game developers to help them resolve the bug.
Depending upon the exact cause of the desync the player may be able to reconnect to the player and continue playing, or they may immediately desync on reconnecting.
Depending upon the exact cause of the desync, the player's game may be able to reconnect to the server to play, or they may find it immediately desyncs on reconnection.
 
=== Desync report ===
The desync report contains both the player's and server's copy of the save game (map and game state). The player's copy is from when the desync occurred and titled ''desynced-level''. With the server's copy being from when the automatic reconnection was made and titled ''reference-level.zip''. A copy of the players Factorio log file is included titled ''log.txt''. As the report contains two copies of the save game it may have a large file size. Furthermore, the save files that are made for the desync report contain more data than normal save files, further increasing their size.
 
The desync report does not state the cause of the desync but rather is a record of the game state at the time of the desync.
 
=== Next Steps ===
The next steps after a desync occurs is to start a desync investigation. In many situations restarting the server will work around the desync in the short term and allow all players to rejoin until the conditions for the bug to occur are re-created. However, a server restart isn't guaranteed success and may lead to no players being able to rejoin the server without desyncing.
 
== Desync Investigations ==
In most cases when desyncs occur and mods or scenarios are being used they are the cause and need to be handled first. If they are found to not be the cause then it should be reported as a [[#Desync_reporting_with_only_base_game|core Factorio desync error]]. In both cases identifying the activity that triggers the desync is an important step. For example, whether it is tied to a player's action or activity that happens in the game (i.e. a train arriving at a station) or is a time-related event. In some cases, the desync may not be reproducible and so only a code investigation can work towards a fix.
 
When desyncs occur and mods/scenarios are in the use there are 2 different approaches that can be taken to help get the issue fixed. The code investigation approach requires you to be familiar with Factorio modding and its coding rules. The less technical method is to identify the mod and activity that causes the desync through trial and error so it can be reported to the mod author.
 
=== Identifying desync causing mod through trial and error ===
This is a less technical method aiming to identify the mod and its activity that seems to cause the desync so it can be reported to the mod/scenario author. The activity that causes the desync should be established prior to these activities being done to confirm if they are having an impact. These steps may affect the gameplay and should be done on a test save.
* Removing any newly added mods and rolling back (returning to the previous version) any updated mods to see if the issue still occurs.
* Removing mods from the game to rule them out as the cause.
Once the desync can no longer be reproduced the suspected mod can be re-added/updated and the desync recreated to confirm the mod and activity are the cause of the desync. As these steps require the server to be restarted this may in itself fix any desyncs in the short term until the conditions for the bug to occur are re-created.


=== Desync Report ===
=== Code investigations with mods/scenarios involved ===
The desync report contains both the player's and server's copy of the save game (map and game state). The player's copy is from when the desync occurred and titled ''desynced-level''. With the server's copy being from when the automatic reconnection was made and titled ''reference-level.zip''. A copy of the players Factorio log file is included titled ''log.txt''.
The desync report is a useful reference for investigating desyncs but it does not state the cause of the desync. Comparing the desync files and use of heavy mode can assist in identifying which mods and activities to review initially. Ideally, it is supplemented by an account of what was occurring and being done by players at the point of the desync occurring to assist with narrowing down the activities of code to review. However in many complicated scenarios this may not be known prior to the investigation.


The desync report does not state the cause but is a record of the game state at the time of the desync.
==== Code that causes desyncs ====
There are a number of ways mod and scenario code can cause desyncs, such as due to incorrect variable scoping, requiring or API event handling. Examples of these can be found in [[Tutorial:Modding tutorial/Gangsir#Multiplayer_and_desyncs|Gangsir's modding tutorial multiplayer and desyncs section]], with the full technical definition of usage constraints for API events in the [https://lua-api.factorio.com/latest/Data-Lifecycle.html Factorio API data lifecycle]. Not all instances of non-compliance will lead to desyncs or errors, but most will. For this reason, it is recommended to always maintain compliance to avoid risk and time-consuming testing. Instances of non-compliance found should be raised to the code author.


== Desync investigations with mods/scenarios involved ==
==== Comparing game state & save files ====
The desync report is a useful reference for investigating desyncs but it does not state the cause of the desync. Ideally, it is supplemented by an account of what was occurring and being done by players at the point of the desync occurring.
The desync report files and heavy mode dumps contain a number of files that can be compared by any text comparison tool to look for differences. These files are generally sequelized dumps of Lua tables or c++ game objects and so are only partially human readable. The less human readable items can include floating point numbers serilaised (0x.HEXp+HEX) and ''magic'' unique identifier numbers for prototype instances in the c++ game state. These numbers and reference ids should generally be the same between files as the in-game entity they reference normally won't have changed between the 2 save game states being obtained.  
In most cases when desyncs occur and mods or scenarios are being used they are the cause. If they are found to not be the cause then it should be reported as a  [[#Desync_reporting_with_only_base_game|core Factorio desync error]].
When the data differs between files the source code of that mod/scenario must be examined for all interactions with that data item for any instances of non-compliance.


=== Code that causes desyncs ===
==== Examining the desync files ====
There are a number of ways mod code can cause desyncs due to incorrect variable scoping, requiring or API event handling. Examples of these can be found in the [[Tutorial:Modding tutorial/Gangsir#Multiplayer_and_desyncs|Gansir's modding tutorial multiplayer and desyncs section]], with the full technical definition of usage constraints for API events in the [https://lua-api.factorio.com/latest/Data-Lifecycle.html Factorio API data lifecycle]
Within each savegame's zip in the desync report are a number of files that when compared can show data changes. As the server copy of the save is obtained after the desync there may be genuine differences between the files. So differences aren't hard evidence of the cause of a desync, but only a pointer on where to investigate further.


The code of the mod can be reviewed if you are familiar with these requirements and it isn't too extensive or abstracted. The supporting information about what occurred at the time of the desync can assist with narrowing down the code to review. Any concerns should be raised to the code author.
* script.dat - contains the scenario and mods [https://lua-api.factorio.com/latest/Global.html Factorio global tables]. Generally, the save's persistent data should be the same between the 2 saves. On rare occasions, there can be a difference due to how the information is written out to the file. Should a mod causes a player's game to desync it can cause the player's game to stop processing other mod activity mid tick. This can lead those mod's global tables to have differences between the player and the server copies despite having no code defects themselves.
* level_with_tags_tick_XXXX.dat - contains the current game state together with additional saving tags to aid human readability. This can show if entity states or chat messages differ. ''level.dat'' is the same information, but without the descriptive tags for data values.


=== Comparing game state & save files ===
==== Using heavy mode command ====
The desync report and heavy mode dumps contain a number of files that can be compared to look for differences. These files are generally sequenalized dumps of Lua tables or game information and so are partially human readable. The non-human readable item includes table ids (0x.NUMBER) and ''magic'' numbers related to the c++ game state. These ids and references should generally be the same between files as the in-game entity they reference normally won't have changed.  
[[Console#Normal_commands|Heavy mode]] is an in-game option that can be enabled to make Factorio save and load the game every tick. This allows Factorio to highlight if there are any changes in the game state between the save and subsequent load. Should a state change occur, a message is shown in-game and the save and load game states recorded to a ''dumps'' folder in the players [[Application directory#User_data_directory|user data folder]].
Typically this process will trigger for incorrect changing of data as part of the on_load event or local/global variables being changed and not persisted. The recorded game state is a dump of the internal game state, including entities, but does not include the mod's global table. This allows the checking if an entity's state has become different between the load and save state.


=== Examining the desync files ===
This process can be used to partially test a mod for multiplayer compliance without the need of multiple players or a server.
Within each save game's zip in the desync report are a number of files that when compared can show data changes, and thus point to areas of the mod to review. However, as the server copy of the save is obtained after the desync there may be genuine differences between the files. So differences aren't hard evidence of the cause of a desync, but a pointer on where to investigate further.


* script.dat - contains the scenario and mods [https://lua-api.factorio.com/latest/Global.html Factorio global tables]. Generally, the saves persistent data should be the same between the 2 saves.
=== Desync reporting of core Factorio game ===
<!-- There are other files, but I don't have copies of old desync reports to double check what they show when --->
If no mod or scenario were involved in the desync, the desync report and activity information should be posted to the [https://forums.factorio.com/53851 Factorio forums bug report section].


=== Using heavy mode command ===
== See also ==
[[Console#Normal_commands|Heavy mode]] is an in-game option that can be enabled to make Factorio save and load the game very tick. This allows Factorio to highlight if there are any changes in the game state between the save and subsequent load. Should a state change occur a message is shown in-game and the game states recorded to a ''dumps'' folder in the players [[Application directory#User_data_directory|Factorio Data folder]].
* [https://www.factorio.com/blog/post/fff-63 FFF 63] - How heavy mode works
Typically this process will trigger for incorrect changing of data as part of the on_load event or local/global variables being changed and not persisted. The recorded game state is a dump of the internal game state and so typically less helpful in identifying the exact nature of the mod's state change, just that one has occured.


== Desync reporting of core Factorio game ==
[[Category:Technical]]
If no mod or scenario were involved in the desync, the desync report and activity information should be posted to the [https://forums.factorio.com/viewtopic.php?f=7&t=3638 Factorio forums bug report section].

Latest revision as of 23:12, 10 August 2019

A desynchronization (desync) is when during multiplayer a player's game becomes desynchronized from the Factorio server. Their game has become different from the server's game.

What are desyncs

Desyncs can be caused either by a bug in the core Factorio game code or by a bug in a mod/scenario. They should not occur during regular play.

Factorio multiplayer architecture

Factorio multiplayer code uses deterministic lockstep to synchronize clients. This is a method of synchronizing a game from one computer to another by sending only the user inputs that control that game, rather than networking the state of the objects in the game itself. It means that all player's games need to simulate every single tick of the game identically. If any computer does something ever-so-slightly different, a desynchronization (desync) occurs. The game includes processes to ensure any issues with network traffic do not cause desyncs.

Player experience of desyncs

Upon a player's game detecting a desync to the server, it immediately disconnects from the server and re-downloads the save game. This download isn't to allow the player to play the game, but to obtain the server's current copy of the save game (map and game state) to include in a desync report. After the download is complete and a desync report has been generated the player is offered the options to view the report, quit or reconnect to the server to play.

Desync report notification.jpg

The desync report can be provided to either the mod/scenario makers or the Factorio game developers to help them resolve the bug. Depending upon the exact cause of the desync, the player's game may be able to reconnect to the server to play, or they may find it immediately desyncs on reconnection.

Desync report

The desync report contains both the player's and server's copy of the save game (map and game state). The player's copy is from when the desync occurred and titled desynced-level. With the server's copy being from when the automatic reconnection was made and titled reference-level.zip. A copy of the players Factorio log file is included titled log.txt. As the report contains two copies of the save game it may have a large file size. Furthermore, the save files that are made for the desync report contain more data than normal save files, further increasing their size.

The desync report does not state the cause of the desync but rather is a record of the game state at the time of the desync.

Next Steps

The next steps after a desync occurs is to start a desync investigation. In many situations restarting the server will work around the desync in the short term and allow all players to rejoin until the conditions for the bug to occur are re-created. However, a server restart isn't guaranteed success and may lead to no players being able to rejoin the server without desyncing.

Desync Investigations

In most cases when desyncs occur and mods or scenarios are being used they are the cause and need to be handled first. If they are found to not be the cause then it should be reported as a core Factorio desync error. In both cases identifying the activity that triggers the desync is an important step. For example, whether it is tied to a player's action or activity that happens in the game (i.e. a train arriving at a station) or is a time-related event. In some cases, the desync may not be reproducible and so only a code investigation can work towards a fix.

When desyncs occur and mods/scenarios are in the use there are 2 different approaches that can be taken to help get the issue fixed. The code investigation approach requires you to be familiar with Factorio modding and its coding rules. The less technical method is to identify the mod and activity that causes the desync through trial and error so it can be reported to the mod author.

Identifying desync causing mod through trial and error

This is a less technical method aiming to identify the mod and its activity that seems to cause the desync so it can be reported to the mod/scenario author. The activity that causes the desync should be established prior to these activities being done to confirm if they are having an impact. These steps may affect the gameplay and should be done on a test save.

  • Removing any newly added mods and rolling back (returning to the previous version) any updated mods to see if the issue still occurs.
  • Removing mods from the game to rule them out as the cause.

Once the desync can no longer be reproduced the suspected mod can be re-added/updated and the desync recreated to confirm the mod and activity are the cause of the desync. As these steps require the server to be restarted this may in itself fix any desyncs in the short term until the conditions for the bug to occur are re-created.

Code investigations with mods/scenarios involved

The desync report is a useful reference for investigating desyncs but it does not state the cause of the desync. Comparing the desync files and use of heavy mode can assist in identifying which mods and activities to review initially. Ideally, it is supplemented by an account of what was occurring and being done by players at the point of the desync occurring to assist with narrowing down the activities of code to review. However in many complicated scenarios this may not be known prior to the investigation.

Code that causes desyncs

There are a number of ways mod and scenario code can cause desyncs, such as due to incorrect variable scoping, requiring or API event handling. Examples of these can be found in Gangsir's modding tutorial multiplayer and desyncs section, with the full technical definition of usage constraints for API events in the Factorio API data lifecycle. Not all instances of non-compliance will lead to desyncs or errors, but most will. For this reason, it is recommended to always maintain compliance to avoid risk and time-consuming testing. Instances of non-compliance found should be raised to the code author.

Comparing game state & save files

The desync report files and heavy mode dumps contain a number of files that can be compared by any text comparison tool to look for differences. These files are generally sequelized dumps of Lua tables or c++ game objects and so are only partially human readable. The less human readable items can include floating point numbers serilaised (0x.HEXp+HEX) and magic unique identifier numbers for prototype instances in the c++ game state. These numbers and reference ids should generally be the same between files as the in-game entity they reference normally won't have changed between the 2 save game states being obtained. When the data differs between files the source code of that mod/scenario must be examined for all interactions with that data item for any instances of non-compliance.

Examining the desync files

Within each savegame's zip in the desync report are a number of files that when compared can show data changes. As the server copy of the save is obtained after the desync there may be genuine differences between the files. So differences aren't hard evidence of the cause of a desync, but only a pointer on where to investigate further.

  • script.dat - contains the scenario and mods Factorio global tables. Generally, the save's persistent data should be the same between the 2 saves. On rare occasions, there can be a difference due to how the information is written out to the file. Should a mod causes a player's game to desync it can cause the player's game to stop processing other mod activity mid tick. This can lead those mod's global tables to have differences between the player and the server copies despite having no code defects themselves.
  • level_with_tags_tick_XXXX.dat - contains the current game state together with additional saving tags to aid human readability. This can show if entity states or chat messages differ. level.dat is the same information, but without the descriptive tags for data values.

Using heavy mode command

Heavy mode is an in-game option that can be enabled to make Factorio save and load the game every tick. This allows Factorio to highlight if there are any changes in the game state between the save and subsequent load. Should a state change occur, a message is shown in-game and the save and load game states recorded to a dumps folder in the players user data folder. Typically this process will trigger for incorrect changing of data as part of the on_load event or local/global variables being changed and not persisted. The recorded game state is a dump of the internal game state, including entities, but does not include the mod's global table. This allows the checking if an entity's state has become different between the load and save state.

This process can be used to partially test a mod for multiplayer compliance without the need of multiple players or a server.

Desync reporting of core Factorio game

If no mod or scenario were involved in the desync, the desync report and activity information should be posted to the Factorio forums bug report section.

See also