Post by giftfish on Jan 5, 2017 21:06:26 GMT
Ported from the old forum. Highlights only.
Plot Database
Postby giftfish » 29 Mar 2016, 11:55
I'm putting together a default PlotDB.db for the toolset, but am waffling on the best way to handle a certain situation.
The tool right now has 3 tabs: ME1, ME2, and ME3. Fine. Makes sense. However, BioWare, in their infinite wisdom, apparently decided after importing a game from ME1 into ME2, ME2 would zero-out all ME1 plots and reuse them. I think. At least, as far as I can tell. Kinko mentioned something about this once and I didn't quite understand, but after looking at the source plot file for ME2 Modified Gibbed, I sort of... see what's going on.
What seems to be the case is the following, using the choice of Anderson as councilor in ME1:
--ME1 Bool = 5435
--ME2's import bool for ME1 decision = 1556
--ME3 import bool for ME1 decision = 15435
To make it even more confusing...
--In ME3 GIbbed, all 3 bools =T
--In ME2 Modified Gibbed, 1556=T ONLY in the ME2 Raw Plot Table, and 5435 is False in both the ME1 and ME2 Raw Plot Tables.
This pattern makes it incredibly difficult to know if you've identified the correct bool. It seems like some ME1 bools have been overwritten, others haven't, and good luck telling the difference. I'd like to strangle whoever thought this was a good idea :evil:
So, the question then becomes: what's the best way to organize the fact that ME1 plots have 3 different bools/ints. Do I put ME1 bools in every single tab? The "real" ME1 bools in the ME1 tab, and then the ME2/ME3 "versions" of the same bools in their respective tabs? That's a lot of duplicate entries >.<
kinkojiro -- if I'm not understanding this all correctly, please let me know. It's confusing as hell.
sircxyrtyx -- Is this lunacy something we could maybe account for when we redo the GUI for this tool? Maybe we could put all info on the same tab "trilogy plots" and have a game column for ME1/2/3? Then somehow make it possible to assign additional bools that are game-specific, or have a "Notes" column... or something?
Postby SirCxyrtyx » 29 Mar 2016, 13:46
That sounds like the best idea. Each tab having the bools that are relevant to making mods for that game.
Those all sound like solid suggestions to me.
Postby giftfish » 29 Mar 2016, 14:14
I kind of like my second idea more. The first one is of course straightforward, but the second goes along with the new GUI concept, the new toolset name, and the new focus on the trilogy as a collective whole. Plus, it avoids having to repeat plots for ME1 3 times, which, now that we actually are getting some decent modding tools, that could end up being a lot of plots. I likely have over 500-1000 plots sitting around for ME3. Maybe more. I can't imagine having to repeat 1000+ plots for ME1 in all 3 tabs.
Another added bonus is that merging them into the same area would allow the user to search all plots for all games at once, rather than confining them to different tabs.
----------
Okay. Well, unless we actually want to change the GUI before the stable (likely not), then that means I need to modify the list of ME1 plots I have right now to give them the "real" ME1 bool and not ME3's version of the bool.
Postby The Fob » 29 Mar 2016, 15:01
Just a quick question on the example: Wasn't there a problem with the decision on who is councilor at the end of ME1? IIRC, the ME1 import save for ME2 was created before the player makes that decision, so the bool is not yet set in this save. This is why Miranda asks you who you chose during the shuttle ride after the ME2 prologue. Who is councilor in ME2 is actually set in this conversation because they didn't have the ME1 bool in the save. If I remember this right, the councilor decision should be a special case in any event.
I can tell you that I used a couple of ME1 bools for the MEHEM emails and they worked fine in all my tests (e.g. the variable about whether Sha'ira gave you the trinket or not, I got this variable from some ME1 variable database on some obscure corner of the internet). But I may have just gotten lucky.
Postby giftfish » 29 Mar 2016, 15:17
@fob -- I'm actually getting this stuff (in part) from the ME2 Modified Gibbed Editor source code. Specifically, the plot document, which is just a xml file.
There are actually two totally different sections for ME1 plots (which you can see in the tool GUI, also). There's "ME1 Plot Table, Legacy", which has the actual ME1 bools, and then "ME1 Imported Plot Table", which has the bools set in ME2 for ME1.
The ME1 Plot Project and the commonly referenced pastebin out there, contains the ME3 bools for ME1, as far as I can tell. Basically, any 5-digit number (16930) is an ME3 plot, as plots never reached 5 digits until ME3. This is likely why your bools for the emails worked
Postby Kinkojiro » 30 Mar 2016, 06:01
How it works is like this:
ME1 post DLC has plot bools 0-7650
ME2 imported some bools selectively from ME1 into its own table and kept a copy of the ME1 plot table separately. Its own bools were in 0-7650.
ME3 imports the ME2 bools 0-7650 then adds the ME1 table at 10000-17650. Its own bools are above 17650.
Gibbed save editors are not 100% accurate when they say this bool signals that - from memory I have found a couple of times they are wrong. I wouldn't take it as gospel. I think he took sequenced checkstates and the notations. We all known from looking at sequencing things sometimes are stated or programmed incorrectly.
Postby giftfish » 31 Mar 2016, 12:17
Alright, so FemShep has done some work on this long-neglected tool. It's looking much better, but some decisions need to be made about layout.
First, some quick info on context and scope.
1. The existing format of plot information in the database is very reminiscent of the Gibbed Save Editor. I suspect WV might have used the code from it, so it's this sort of thing in a single line:
There's nothing wrong with this, but it actually conveys a lot of information. It's worth considering if there's a better way to organize it, as the use is no longer for editing saves, but as a research and modding aid for content modders. We want the information to be easy to enter/locate and highly-navigable.
Here's how it could read instead (with column contents defined by hashes):
This would ensure the entry would appear under Priority: Mars in the tool, which essentially helps the modder know what game files to look in.
2. The intent is to include a default database with the tool; currently, there is none so all players have to start from scratch. In addition, the tool can now export/import CSV format. That means I can upload a copy of the default plot file to Google Docs and give folks permission to edit. I could open it to anyone, but I'd like to minimize error.
Therefore, the intent would be to give experienced modders interested in contributing, permission to edit the document. It would be awesome for us to be able to share information on all the plots we've tracked down while modding. Then, K/Sir/Heff can download the most recent version and pop it into the toolset before releasing each new rev
Okay, all that said, here's an example of what things look like in the tool right now. I've added a variety of bools and ints:
What I want to focus on is the Category/States columns, but before I do that, I want to point out a couple of things:
On to the Category/State columns. These are the bread and butter of the tool. The challenge is that a huge variety of information can be added to these columns. I've tried to show some of that variety in the content I've added to the test file above.
The basic question is: how many more columns do we add?
Too many means some might not ever get used. Too few and you're having to cram too much information into the same column, which then impacts sortability and navigation. In addition, unless there's some type of order as to what can be put in the columns, it will be easy for sorting to become problematic. For example, having mission names in two different columns.
I've already asked FemShep to implement one more column, "Category 2", and to change "State" to "State/Value". That way, an integer value range can be placed in the column, as well.
I've been continuing to brainstorm about this, and I think a solution is forming in my mind. I think every entry needs an actual category, and then we drill down from there. Here's a slew of them I came up with in the last 30s:
So, each entry gets assigned one of those. Then in the next column we start to get more specific:
At this point, it starts to get more complicated, additional categories won't always be necessary (but probably will). If not, it might be time to move onto the State/Value column. I'll keep going with the same example ("X" = n/a):
And, we keep going:
I think folks are probaly getting the idea. This would be a total of 4 columns: 3 categories and 1 State/Value. An entry would require using Category 1 and the State/Value, but that's it use of anything else would be optional. I mean that in the sense not that the tool will force you, but that would be the minimum amount of information you'd want.
Anyone seeing a better way to do this? 3 columns of categories? 4? 2?
As a final aside, of course everyone is allowed to format their own database to their liking (assuming they use the tool at all). But, if the folks interested in using the tool can agree on some basic ways to do things, it's less work for each individual.
Postby The Fob » 31 Mar 2016, 17:49
Looks pretty good to me on the screenshot. I wouldn't add more columns I think. Too much clutter and we may not be able to keep a super consistent categorizing sceme anyway because some variables are e.g. used in multiple missions.
Makes me wonder by the way, would it be possible to run a scanner through all ME3 files, look for SeqAct_CheckState and SeqAct_SetState objectsand make a list automatically? Almost (if not) all of the Check- and SetState objects I see in the sequence editor have comments, these could be imported into the State fields, the file names where these are found (first) could be imported into the Category field.
I realize, this will still miss heaps of variables since a lot of them are embedded in conditionals but still, might give us an impressive list to start with.
In any case, I am happy to add to a google document of course.
Postby Kinkojiro » 03 Apr 2016, 07:20
Just to give my 2c on this worthy endeavour. Simpler would be better.
I would drop the seperate columns for ME1->ME2 and ME2->ME3 conversions. Just have one column for ME3 units (as all ME2 are the same) and a column for the ME1 bools (which should be only ME3 (10k-17679 minus 10k) filled automatically (i.e. add a ME1 bool and it fills the ME3 column or visa versa).
The main reason is there are so many duplications. For example quite often missions have bools for "mission done", journal mission done, etc. Just because ME2 has someone who is councillor in one bool and ME3 has a duplicate in another bool is not really important. Both sets of bools should be listed under there relevant ME3 number, as this works 100% of the time with ME2 and ME1 with the exception of the filled ME1 bools. Otherwise it is just confusing. At most a note is needed saying "also duplicated at 1556".
For the categories I would stick to just one (90% of cases the mission, otherwise "Normandy", "Love interest" etc.). Then you have description, then notes. At most a second is needed if this is checked in a conversation I guess.
If we make this too complicated I suspect people won't use it. Most of the type idiosyncracies should be in the notes. If you go to use a bool then the note saying broken will be just as effective as having a column that only has a few broken bools ticked. Columns are really only useful if you want to sort on them, and I can't think of a situation that I would want to sort on a broken bool column.
Any hope this is useful. An improvement to the plot manager is very welcome.
Plot Database
Postby giftfish » 29 Mar 2016, 11:55
I'm putting together a default PlotDB.db for the toolset, but am waffling on the best way to handle a certain situation.
The tool right now has 3 tabs: ME1, ME2, and ME3. Fine. Makes sense. However, BioWare, in their infinite wisdom, apparently decided after importing a game from ME1 into ME2, ME2 would zero-out all ME1 plots and reuse them. I think. At least, as far as I can tell. Kinko mentioned something about this once and I didn't quite understand, but after looking at the source plot file for ME2 Modified Gibbed, I sort of... see what's going on.
What seems to be the case is the following, using the choice of Anderson as councilor in ME1:
--ME1 Bool = 5435
--ME2's import bool for ME1 decision = 1556
--ME3 import bool for ME1 decision = 15435
To make it even more confusing...
--In ME3 GIbbed, all 3 bools =T
--In ME2 Modified Gibbed, 1556=T ONLY in the ME2 Raw Plot Table, and 5435 is False in both the ME1 and ME2 Raw Plot Tables.
This pattern makes it incredibly difficult to know if you've identified the correct bool. It seems like some ME1 bools have been overwritten, others haven't, and good luck telling the difference. I'd like to strangle whoever thought this was a good idea :evil:
So, the question then becomes: what's the best way to organize the fact that ME1 plots have 3 different bools/ints. Do I put ME1 bools in every single tab? The "real" ME1 bools in the ME1 tab, and then the ME2/ME3 "versions" of the same bools in their respective tabs? That's a lot of duplicate entries >.<
kinkojiro -- if I'm not understanding this all correctly, please let me know. It's confusing as hell.
sircxyrtyx -- Is this lunacy something we could maybe account for when we redo the GUI for this tool? Maybe we could put all info on the same tab "trilogy plots" and have a game column for ME1/2/3? Then somehow make it possible to assign additional bools that are game-specific, or have a "Notes" column... or something?
Postby SirCxyrtyx » 29 Mar 2016, 13:46
That sounds like the best idea. Each tab having the bools that are relevant to making mods for that game.
giftfish wrote:
sircxyrtyx -- Is this lunacy something we could maybe account for when we redo the GUI for this tool? Maybe we could put all info on the same tab "trilogy plots" and have a game column for ME1/2/3? Then somehow make it possible to assign additional bools that are game-specific, or have a "Notes" column... or something?
sircxyrtyx -- Is this lunacy something we could maybe account for when we redo the GUI for this tool? Maybe we could put all info on the same tab "trilogy plots" and have a game column for ME1/2/3? Then somehow make it possible to assign additional bools that are game-specific, or have a "Notes" column... or something?
Those all sound like solid suggestions to me.
Postby giftfish » 29 Mar 2016, 14:14
I kind of like my second idea more. The first one is of course straightforward, but the second goes along with the new GUI concept, the new toolset name, and the new focus on the trilogy as a collective whole. Plus, it avoids having to repeat plots for ME1 3 times, which, now that we actually are getting some decent modding tools, that could end up being a lot of plots. I likely have over 500-1000 plots sitting around for ME3. Maybe more. I can't imagine having to repeat 1000+ plots for ME1 in all 3 tabs.
Another added bonus is that merging them into the same area would allow the user to search all plots for all games at once, rather than confining them to different tabs.
----------
Okay. Well, unless we actually want to change the GUI before the stable (likely not), then that means I need to modify the list of ME1 plots I have right now to give them the "real" ME1 bool and not ME3's version of the bool.
Postby The Fob » 29 Mar 2016, 15:01
Just a quick question on the example: Wasn't there a problem with the decision on who is councilor at the end of ME1? IIRC, the ME1 import save for ME2 was created before the player makes that decision, so the bool is not yet set in this save. This is why Miranda asks you who you chose during the shuttle ride after the ME2 prologue. Who is councilor in ME2 is actually set in this conversation because they didn't have the ME1 bool in the save. If I remember this right, the councilor decision should be a special case in any event.
I can tell you that I used a couple of ME1 bools for the MEHEM emails and they worked fine in all my tests (e.g. the variable about whether Sha'ira gave you the trinket or not, I got this variable from some ME1 variable database on some obscure corner of the internet). But I may have just gotten lucky.
Postby giftfish » 29 Mar 2016, 15:17
@fob -- I'm actually getting this stuff (in part) from the ME2 Modified Gibbed Editor source code. Specifically, the plot document, which is just a xml file.
There are actually two totally different sections for ME1 plots (which you can see in the tool GUI, also). There's "ME1 Plot Table, Legacy", which has the actual ME1 bools, and then "ME1 Imported Plot Table", which has the bools set in ME2 for ME1.
The ME1 Plot Project and the commonly referenced pastebin out there, contains the ME3 bools for ME1, as far as I can tell. Basically, any 5-digit number (16930) is an ME3 plot, as plots never reached 5 digits until ME3. This is likely why your bools for the emails worked
Postby Kinkojiro » 30 Mar 2016, 06:01
How it works is like this:
ME1 post DLC has plot bools 0-7650
ME2 imported some bools selectively from ME1 into its own table and kept a copy of the ME1 plot table separately. Its own bools were in 0-7650.
ME3 imports the ME2 bools 0-7650 then adds the ME1 table at 10000-17650. Its own bools are above 17650.
Gibbed save editors are not 100% accurate when they say this bool signals that - from memory I have found a couple of times they are wrong. I wouldn't take it as gospel. I think he took sequenced checkstates and the notations. We all known from looking at sequencing things sometimes are stated or programmed incorrectly.
Postby giftfish » 31 Mar 2016, 12:17
Alright, so FemShep has done some work on this long-neglected tool. It's looking much better, but some decisions need to be made about layout.
First, some quick info on context and scope.
1. The existing format of plot information in the database is very reminiscent of the Gibbed Save Editor. I suspect WV might have used the code from it, so it's this sort of thing in a single line:
[Kaidan] Hostile at the Cerberus Corpse on MarsThere's nothing wrong with this, but it actually conveys a lot of information. It's worth considering if there's a better way to organize it, as the use is no longer for editing saves, but as a research and modding aid for content modders. We want the information to be easy to enter/locate and highly-navigable.
Here's how it could read instead (with column contents defined by hashes):
Priority: Mars - Conversation - Kaidan - Hostile at Cerberus CorpseThis would ensure the entry would appear under Priority: Mars in the tool, which essentially helps the modder know what game files to look in.
2. The intent is to include a default database with the tool; currently, there is none so all players have to start from scratch. In addition, the tool can now export/import CSV format. That means I can upload a copy of the default plot file to Google Docs and give folks permission to edit. I could open it to anyone, but I'd like to minimize error.
Therefore, the intent would be to give experienced modders interested in contributing, permission to edit the document. It would be awesome for us to be able to share information on all the plots we've tracked down while modding. Then, K/Sir/Heff can download the most recent version and pop it into the toolset before releasing each new rev
Okay, all that said, here's an example of what things look like in the tool right now. I've added a variety of bools and ints:
What I want to focus on is the Category/States columns, but before I do that, I want to point out a couple of things:
- "Broken" -- As we know, certain plots in the games don't get set properly, and then can't be imported into the next game. Two great examples are Jacob's father's fate in ME2, and Liara's rekindle status in LoTSB. The checkmark for "Broken" is to indicate a broken plot. When checking the box, additional information about the problem can be put into "Notes".
- ME1 > ME2 ID, ME2 > ME3 ID -- These are specifically to deal with the adjustment ME2 makes to ME1 plots upon import. Obviously, this means only ME1 plots will have these columns filled in. It'll be very nice to have these all referenced on the same line.
On to the Category/State columns. These are the bread and butter of the tool. The challenge is that a huge variety of information can be added to these columns. I've tried to show some of that variety in the content I've added to the test file above.
The basic question is: how many more columns do we add?
Too many means some might not ever get used. Too few and you're having to cram too much information into the same column, which then impacts sortability and navigation. In addition, unless there's some type of order as to what can be put in the columns, it will be easy for sorting to become problematic. For example, having mission names in two different columns.
I've already asked FemShep to implement one more column, "Category 2", and to change "State" to "State/Value". That way, an integer value range can be placed in the column, as well.
I've been continuing to brainstorm about this, and I think a solution is forming in my mind. I think every entry needs an actual category, and then we drill down from there. Here's a slew of them I came up with in the last 30s:
Achievement, Mission, Pet, Relationship, Conversation, GAW, Player, Bonus Power, Armor, DLC, Upgrade, Galaxy Map, EmailSo, each entry gets assigned one of those. Then in the next column we start to get more specific:
Paramour, Priority:Tuchanka, Space Hamster, Romance, Ashley, Hanar & Drell, Alignment, Reave, Project Overlord, Advanced Mineral Scanner, Eagle Nebula, ThaneAt this point, it starts to get more complicated, additional categories won't always be necessary (but probably will). If not, it might be time to move onto the State/Value column. I'll keep going with the same example ("X" = n/a):
X, Mordin, Acquired, Thane, Poetry Book Given, Medigel upgrade, Paragon, X, Played, Activated, Amun, Huerta inviteAnd, we keep going:
X, Cured genophage, X, Active, X, X, Current Value, X, X, X, Unlocked, 0-2I think folks are probaly getting the idea. This would be a total of 4 columns: 3 categories and 1 State/Value. An entry would require using Category 1 and the State/Value, but that's it use of anything else would be optional. I mean that in the sense not that the tool will force you, but that would be the minimum amount of information you'd want.
Anyone seeing a better way to do this? 3 columns of categories? 4? 2?
As a final aside, of course everyone is allowed to format their own database to their liking (assuming they use the tool at all). But, if the folks interested in using the tool can agree on some basic ways to do things, it's less work for each individual.
Postby The Fob » 31 Mar 2016, 17:49
Looks pretty good to me on the screenshot. I wouldn't add more columns I think. Too much clutter and we may not be able to keep a super consistent categorizing sceme anyway because some variables are e.g. used in multiple missions.
Makes me wonder by the way, would it be possible to run a scanner through all ME3 files, look for SeqAct_CheckState and SeqAct_SetState objectsand make a list automatically? Almost (if not) all of the Check- and SetState objects I see in the sequence editor have comments, these could be imported into the State fields, the file names where these are found (first) could be imported into the Category field.
I realize, this will still miss heaps of variables since a lot of them are embedded in conditionals but still, might give us an impressive list to start with.
In any case, I am happy to add to a google document of course.
Postby Kinkojiro » 03 Apr 2016, 07:20
Just to give my 2c on this worthy endeavour. Simpler would be better.
I would drop the seperate columns for ME1->ME2 and ME2->ME3 conversions. Just have one column for ME3 units (as all ME2 are the same) and a column for the ME1 bools (which should be only ME3 (10k-17679 minus 10k) filled automatically (i.e. add a ME1 bool and it fills the ME3 column or visa versa).
The main reason is there are so many duplications. For example quite often missions have bools for "mission done", journal mission done, etc. Just because ME2 has someone who is councillor in one bool and ME3 has a duplicate in another bool is not really important. Both sets of bools should be listed under there relevant ME3 number, as this works 100% of the time with ME2 and ME1 with the exception of the filled ME1 bools. Otherwise it is just confusing. At most a note is needed saying "also duplicated at 1556".
For the categories I would stick to just one (90% of cases the mission, otherwise "Normandy", "Love interest" etc.). Then you have description, then notes. At most a second is needed if this is checked in a conversation I guess.
If we make this too complicated I suspect people won't use it. Most of the type idiosyncracies should be in the notes. If you go to use a bool then the note saying broken will be just as effective as having a column that only has a few broken bools ticked. Columns are really only useful if you want to sort on them, and I can't think of a situation that I would want to sort on a broken bool column.
Any hope this is useful. An improvement to the plot manager is very welcome.