Event Data during Gameplay?

Discuss and distribute tools and methods for modding. Moderator - Grognak
bathroomscales
Posts: 7
Joined: Thu Mar 24, 2016 7:33 pm

Event Data during Gameplay?

Postby bathroomscales » Thu Mar 24, 2016 7:39 pm

Hi,

I'm working on a scifi story-based mod with a lot of cool story mechanics. One thing I'm considering would involve adding A LOT(!!!!!!) of events. Like orders of magnitude more events than are in the default game. (I'd definitely be using a script to help generate everything :D)

But something occurred to me as I started on this. I don't know how FTL uses the data.dat file, and that could cause some issues with a giant list of events.

For example, if the game just loads the whole data.dat file at runtime and stores it all in memory... my mod might be too big & either harm performance, or outright break things! I'd like to release this mod for others, so that would definitely be a problem.

But if FTL dynamically loads events from the data.dat file as needed, I imagine this would be fine?

Does anyone know anything about how the data.dat information is handled under the hood?

Thanks!
sul
Posts: 121
Joined: Sat Jan 30, 2016 4:22 pm

Re: Event Data during Gameplay?

Postby sul » Fri Mar 25, 2016 9:59 pm

Just some notes that may help you if you want to take on that task:

- There is not really any limit to the number of events you can add. Last time I added procedurally generated events from a 20Mb text file, it went smoothly maybe 1 more second load time at the initial screen.
- There is however some important problems depending on how you write your events. The one golden rule is to never reconnect branches of your xml tree of events, see this discussion between me and kartoFlane:
viewtopic.php?f=12&t=17442&p=99949&hilit=memory#p99949
bathroomscales
Posts: 7
Joined: Thu Mar 24, 2016 7:33 pm

Re: Event Data during Gameplay?

Postby bathroomscales » Sat Mar 26, 2016 2:39 am

very nice!!! thanks so much, this link & your post seem to pretty comprehensively say that it'll be ok :)

I'm very curious though... how did you manage to regroup selected bifurcations in the first place?? In the case of a non-list event started by a player's choice, the event is always explicitly defined in code (with text elements and everything). How could you have multiple choices lead to the same explicitly-defined, non-named event?
sul
Posts: 121
Joined: Sat Jan 30, 2016 4:22 pm

Re: Event Data during Gameplay?

Postby sul » Mon Mar 28, 2016 12:15 am

bathroomscales wrote:very nice!!! thanks so much, this link & your post seem to pretty comprehensively say that it'll be ok :)

I'm very curious though... how did you manage to regroup selected bifurcations in the first place?? In the case of a non-list event started by a player's choice, the event is always explicitly defined in code (with text elements and everything). How could you have multiple choices lead to the same explicitly-defined, non-named event?

You could regroup bifurcations through event loading (this only works for named events). For example:

<event name="EVENT0">
<text> event 0</text>
<choice><text> to event 1</text><event load="EVENT1"/></choice>
</event>

<event name="EVENT1">
<text> event 1 </text>
<choice><text> to event 3</text><event load="EVENT3"/></choice>
</event>

<event name="EVENT2">
<text> event 2 </text>
<choice><text> to event 2</text><event load="EVENT3"/></choice>
</event>

<event name="EVENT3">
<text> event 3 </text>
<choice><text> finish</text><event/></choice>
</event>

made a branch event0-->event1 and event0-->event2, then regroups selected bifurcations in event1-->event3 and event2-->event3. This is actually what you SHOULDNT DO. Even if FTL will reluctantly accept this, it violates the xml structure so will consume a looot of memory, maybe even crash loading if xml tree is very big. A trick to avoid this is to create identical event3a and event3b, where now event1-->event3a and event2-->event3b. There are other tricks in the books, in other post you talked about the wiki page maybe its a good place to put some informations.
User avatar
kartoFlane
Posts: 1488
Joined: Mon Jan 14, 2013 10:20 pm

Re: Event Data during Gameplay?

Postby kartoFlane » Mon Mar 28, 2016 11:51 am

sul wrote:made a branch event0-->event1 and event0-->event2, then regroups selected bifurcations in event1-->event3 and event2-->event3. This is actually what you SHOULDNT DO. Even if FTL will reluctantly accept this, it violates the xml structure so will consume a looot of memory, maybe even crash loading if xml tree is very big. A trick to avoid this is to create identical event3a and event3b, where now event1-->event3a and event2-->event3b.

Is it something you've tested and confirmed, or is it just personal theory? I don't recall there ever being any issues with multiple events linking to the same "closing" event, only loops are forbidden. As far as I'm aware, both vanilla game and CE use this extensively.
Also from a programmatic point of view, there's no reason (at least, no immediately obvious one) why the former approach would consume more memory than the latter (in fact your 'trick' would end up consuming marginally more, as the game would have to create two separate intermediate event structures holding exact same data)
Superluminal2 - a ship editor for FTL
sul
Posts: 121
Joined: Sat Jan 30, 2016 4:22 pm

Re: Event Data during Gameplay?

Postby sul » Mon Mar 28, 2016 2:58 pm

kartoFlane wrote:
sul wrote:made a branch event0-->event1 and event0-->event2, then regroups selected bifurcations in event1-->event3 and event2-->event3. This is actually what you SHOULDNT DO. Even if FTL will reluctantly accept this, it violates the xml structure so will consume a looot of memory, maybe even crash loading if xml tree is very big. A trick to avoid this is to create identical event3a and event3b, where now event1-->event3a and event2-->event3b.


Is it something you've tested and confirmed, or is it just personal theory? I don't recall there ever being any issues with multiple events linking to the same "closing" event, only loops are forbidden. As far as I'm aware, both vanilla game and CE use this extensively.
Also from a programmatic point of view, there's no reason (at least, no immediately obvious one) why the former approach would consume more memory than the latter (in fact your 'trick' would end up consuming marginally more, as the game would have to create two separate intermediate event structures holding exact same data)


KartoFlane, this is all personal experimentation but quite confirmed by tests in many instances. It also makes sense with respect to the theoretical discussion we had. Here are some discussion points.

Rules:
Regrouping selected bifurcations (events from multiple choices) within a same xml tree needs to be avoided. It will be accepted by FTL without crash, however it can lead to important loading time issues for very large xml trees. Note that regrouping random bifurcations (events from eventList) is acceptable. In addition events that are from different xml trees (with different root events) can regroup to a same event. Each xml tree has an unique "root event", that can be a beacon on the star map or the result from a ship encounter like destroyed, deadCrew, etc.

With respect to your comments:
- From reading the data.dat files, I dont recall FTL Vanilla/AE using selected bifurcations regroupment, only random bifurcations regroupment. But even if it is the case there is no issues as the xml structures of those events remain small generally.
- A theoretical argument is that regrouping selected bifurcations violates the xml structure. If the events were nameless and regrouped in the classical form <event>..</event>, without using event load, it would appear obvious that regroupments are impossible. What is puzzling is that random bifurcations can still be regrouped seamlessly, but that is another issue.

Some Practical examples:

- Mod Testing Environment: This is a medium size menu, with a few selected bifurcation regroupments, that has loading time issues (as commented by TaxiDriver on this page). I am only speculating here, as it is not my mod and i didnt experiment on it but the coincidence makes sense. This is just one example I remember but you can find other mods where it also happens.
viewtopic.php?t=22613

- FTL Scramble Menu: This is a very large menu, with no selected bifurcations regroupments, that loads seamlessly. Note that there is a trick to endlessly reopen the menu (ship hostile upon crew death), but i am diverging from the subject.
viewtopic.php?t=29104

- FTL Scramble Battle Mod: This is menu with an extremely large chain of events (around 20Mb of text), with no selected bifurcations regroupment, that load seamlessly. In particular, the chain of event (called the "infinite random mode" in game) has a huge amount of random bifurcations regroupments: I checked that if only a SINGLE selected bifurcation regroupment is included in that chain the mod cannot load.
viewtopic.php?t=29104

- Many other experimentations around this but it is hard to present examples. Before finding the trick to endlessly reopen the scramble menu, i had tested many types of xml tree structures to avoid selected bifurcations regroupments that would never load.
User avatar
kartoFlane
Posts: 1488
Joined: Mon Jan 14, 2013 10:20 pm

Re: Event Data during Gameplay?

Postby kartoFlane » Mon Mar 28, 2016 5:29 pm

Huh, that's pretty interesting finding, thanks for elaborating. Yet another quirk to FTL's already sizable collection :P
Superluminal2 - a ship editor for FTL
bathroomscales
Posts: 7
Joined: Thu Mar 24, 2016 7:33 pm

Re: Event Data during Gameplay?

Postby bathroomscales » Tue Mar 29, 2016 11:50 pm

As usual, thank you for dropping some knowledge sul :) I hadn't realized you could load regular events (I thought it was just eventLists)! That opens up some interesting possibilities.. though a lot are limited by the no-regrouping-rule

However, my programmer's intuition has to side with karto! I hate to question your clearly-extensive testing (at least until I've done a good bit of my own), but it really seems odd that loadtimes on the XML would balloon in these cases. Surely they just note the @load name to look up, then move on?

I'm eager to try these myself. From your & karto's wording it sounds like you've both done relatively small events with selected-regrouping without affecting load performance? Perhaps there is some confounding factor in the larger tests?

If FTL is in fact doing some quirky loading that investigates the @load instead of merely store-and-countinue, that opens up a lot of possibilities as to the specific cause(s) of slowdown. And could give some clues for how to mitigate it :)
bathroomscales
Posts: 7
Joined: Thu Mar 24, 2016 7:33 pm

Re: Event Data during Gameplay?

Postby bathroomscales » Tue Mar 29, 2016 11:53 pm

(Also.. OT.. is there a way to get my account upgraded so every post doesn't have to wait for approval? While it's fun to discuss all this theory in a slow old-timey, written-letter kind of way... it is quite annoying :P)
User avatar
kartoFlane
Posts: 1488
Joined: Mon Jan 14, 2013 10:20 pm

Re: Event Data during Gameplay?

Postby kartoFlane » Wed Mar 30, 2016 8:05 pm

bathroomscales wrote:(Also.. OT.. is there a way to get my account upgraded so every post doesn't have to wait for approval? While it's fun to discuss all this theory in a slow old-timey, written-letter kind of way... it is quite annoying :P)

I suppose that's a recent countermeasure taken against the bot spam we've been getting here recently. I guess 10 posts is the limit, same as for private messages.
Superluminal2 - a ship editor for FTL