Welcome to Keen Software House Forums! Log in or Sign up to interact with the KSH community.
  1. You are currently browsing our forum as a guest. Create your own forum account to access all forum functionality.

Programmable Block Inter-Grid Communication Guide

Discussion in 'Programming (In-game)' started by rexxar, Jan 19, 2017.

  1. Elfi Wolfe Apprentice Engineer

    and what happens when they blueprint your ship and script?
  2. Bleuhazenfurfle Apprentice Engineer

    HTTP in SE… I suppose it was inevitable. For all it's spacey gravimetric jumpy goodness, SE seems to be about 10 years behind real life. Just, please, keep it minimal — short but obvious header names, minimal parsing complexity, etc.

    I think I'd rather like to see an example of the spec you're using, though, sooner rather than later. Getting discussion on it before it becomes widely used, is a good thing — if we can get most of the early scripts using a good basic format, maybe we can get a bit of a standard happening before it's to late. For example, please tell me you specify timestamps in Unix format, rather than that horrid ISO rubbish that HTTP loves so much, and how are you planning on specifying and embedding the hash, and just what ARE you using?

  3. Me 10 Jin Apprentice Engineer

    @Bleuhazenfurfle Here's a rough outline for my transmission scheme. It is subject to change.
    /// '+' is string concatenation. Header values are terminated by '\n'
    message		: "{" + scheme_marker + headers + "}" + message_body
    headers		: auth_hdrs + message_info + recipients
    auth_hdrs	: sender + timestamp + signature
    sender		: "From=" + entity_alias
    signature	: "Signature=" + HMAC( hash_fn, shared_secret, string_to_sign )
    message_info	: message_id + message_length + message_hash
    recipients	: "To=" entity_alias + recipients | ""
    A credentials set is just 3 tokens: an alias, a secret, and another alias (a grid ID). The 'shared_secret' used in signing a message is a combination of all 3 credentials. The 'string_to_sign' is a concatenation of message headers (minus the signature header, of course).

    The 'scheme_marker' and message length header allow any other message of any other type to follow it in a transmission. Since multiple messages can be sent at once, I'll have no need to manage outgoing transmissions.

    A consumer consuming a message follows these step to authenticate it. Success means the message body is fed to a PB or just pushed on a queue. Failure at any step aborts the process and the message handler moves on.
    1. The message_id is checked against a dictionary (avoid duplicates).
    2. The timestamp is checked against the clock (somewhat redundant, might get cut).
    3. The sender's alias is checked against a list of known credentials sets.
    4. The consumer's aliases are checked against the message's list of recipients.
    5. Using its known credentials, the consumer calculates the HMAC and compares it to the signature.
    6. The message hash is calculated and compared to that header's value (avoid bad data).
    Disseminating credential sets is up to the user, either configure it manually, do it through the grid terminal system, or use some secure transmission scheme. I would not advise broadcasting entire credential sets.
    • Informative Informative x 1
  4. Badger Trainee Engineer

    Before calling TransmitMessage seems to not send if EnableBroadCast has not been enabled for a few seconds before calling.

    Some of enabled the radio, sends message then disables but it does not seem to like it unless I manually enable the radio a few seconds before calling
    	foreach (IMyRadioAntenna ant in ants)   
                ant.SetValueBool("EnableBroadCast", true);
                ant.TransmitMessage(Me.GetPosition().ToString("0.0"), MyTransmitTarget.Default);   
                ant.SetValueBool("EnableBroadCast", false);
                Echo("Sending pos "+ Me.GetPosition().ToString("0.0"));
  5. Bleuhazenfurfle Apprentice Engineer

    So an actual message might look something like:

    Signature=CSUM HMACHEAD
    To=TO1 TO2
    Seems reasonable. Parsing would be, basically: check start of the string is "{SETP\n", then proceeding one line at a time (I presume C# has an iterator that will let you do that, and then simply take the remainder? A filestream, perhaps?) until we encounter a like starting with }, split on the = and shove into a Dictionary (possibly converting the key name to a consistent case first). We can then pick values out of the Dictionary at will.

    Couple thoughts I had, just to toss into the bucket:
    • "A credentials set is just 3 tokens" ... "is a combination of all 3 credentials". I presume you mean simply that 'shared_secret' is just 3 those tokens, and that the from field is used to look up the second and third tokens of the credential?
    • I'd personally keep the header names short: "Sign" instead of "Signature" (or even shorter for the basic required set), these things may need to be stored by scripts.
    • The HMAC scheme is assumed in the scheme_marker? I'd prefer it was prepended to the hashs themselves. (You said you were using an insecure hash, I presume it's some kind of simple checksum?)
    • Personally, I'd probably pull the signature out of the headers, so you generate the entire header string (including braces), and then send it, it's hash, a newline, then the message body: we shouldn't need to mess with the headers of in-flight messages, don't need to worry about header order (definitely don't want to have to sort and re-compose!), and this has the benefit also of allowing you to read the closing brace as just another header line (unless it's quicker to search ahead for a "\n}" and split into lines).
    • Hash and timestamp format/encoding? Binary is out since it could contain \n's and }'s, do we have base64 readily available? (Otherwise use hex?)
    • Timestamp ("Now" or "When", in Unix format with optional fractional seconds) is useful for superscedence, where you intentionally transmit a second message with the same message_id. An example of that, is a script that periodically outputs it's current state: instead of the message_id being a nonce, it would be a fixed string (you'd then probably toss in a real nonce header just to mix up the hmac, and prefer that over the message id when present), and the newer of the two messages would win.
    • Expires header is useful in a store-and-forward system, as well as simply knowing when it's safe to remove the duplicate message information from the cache. Allow either timestamp to be used as a substitute for the other, and perhaps only relay messages that have at least one. Could also abbreviate this with an optional "lifetime" appended to the timestamp? [ie. timestamp = timestamp_header or expires_header or 0, expires = expires_header or (timestamp + (timestamp_lifetime or default_expire)) ]
    • I'd also be tempted to generate the entire message without the scheme marker, and then prepend both the scheme marker and the total message length (omitting the message length header, since it can be derived once we've parsed the header), allowing you to simply ignore a message format you don't recognise, so long as it follows at least: scheme, whitespace, size, whitespace, remainder.
    My version would end up as:

    {SETP 103
    To=TO1 TO2
    Nt=TIMESTAMP 600
    Slightly more cryptic, but I think it'd work well enough.
  6. Me 10 Jin Apprentice Engineer

    Parsing starts with recognizing the scheme marker, then it either looks for a closing "\n}" and splits the string in between, or it consumes line by line, stopping when the line begins with "}". I personally prefer it line-by-line, but you're free to roll your strings however you want, I'm not here to judge.

    The 'shared_secret' is basically hash_fn( public_alias + secret_token + my_grid_id ). The purpose of having the grid ID in there is to make it harder to keep valid sender credentials through copypasta.

    Yep. The scheme marker indicates the hash function. I just haven't settled on which function to use. I'm leaning towards CRC32 or something of that complexity. SHA2 is way too much and I'm not sure that lib is available in scripts.

    That's the same as saying you're going to put a header after the headers. When you add a car to a train, you just call it 'a train', not 'a train plus one train car'.

    (double) ( DateTime.Now() - DateTime( '1970/01/01' ) ).TotalDays

    Message management is outside the scope of authentication. Once a message is authenticated, other functions can handle management, do logic, formulate replies, etc....

    This is my refined message scheme. Extension is allowed via 'additional_header'.
    // Ordering
    message     : "{" + scheme_marker + headers + "}" + message_body
    headers     : auth_headers + signature + message_info + recipients + more_headers
    auth_headers   : sender + timestamp
    message_info   : message_id + message_length + message_hash
    recipients   : recipient + recipients | ""
    more_headers   : additional_header + more_headers | ""
    // Names
    sender     : "From=" + token + "\n"
    timestamp   : "At=" + decimal + "\n"
    recipient   : "To=" + token + "\n"
    signature   : "Signature=" + HMAC_value + "\n"
    message_id   : "MSG ID=" + token + "\n"
    message_length   : "MSG Len=" + numeric + "\n"
    message_hash   : "MSG Hash=" + hash_fn( message_body ) + "\n"
    // Values. 
    HMAC_value   : HMAC( hash_fn, shared_secret, string_to_sign ) as hexadecimal
    string_to_sign   : auth_headers + message_info + recipients
    token     : [\x20-\x7e]+
    decimal     : [0-9]+(\.[0-9]*)?
    numeric     : [0-9]+
    Here's an example complete message.
    {?? From=me
    MSG ID=random
    MSG Len=11
    MSG Hash=1234567890abcdef
    }haiya dooin
  7. NoirFusuky Trainee Engineer

    I have a German Tutorial :)

  8. Martin R Wolfe Trainee Engineer

    Does anyone know if there is a way to read which programmable block is assigned to an antenna? I would like to be able to have a PB test local antennas to see if incoming data came from an antenna or via a run argument. The test being if no enabled local antennas have the PB as a target the argument came by a run command.

    The reason for this test is for configuration as in my script CustomData is used by the other local ProgrammableBlocks to pass outgoing transmission data. Currently I am using CustomData for both configuration and transmission. So configuration currently requiers all program blocks to stop running on timer whilst it takes place. With this test in place timers can continue to run all the blocks as normal and outgoing data flow will be queued while configuration takes place.
  9. Wicorel Senior Engineer

    None currently.
  10. Martin R Wolfe Trainee Engineer

    Disappointing but understandable as inter-grid communication is still new. I suppose for easily controllable configuration I'll just have to send the configuration from another PB on the same or locally connected grid.
  11. Malware Master Engineer

    The intention is to have a dedicated comms PB. This PB can then communicate with others.

    What do you mean? While what takes place? PBs don't run asynchronously...
  12. Martin R Wolfe Trainee Engineer

    My intention was to directly enter configuration via the comms PB custom data. However with the other PBs still running their is a chance of either out going data loss or corruption manually typing into custom data as it is used as the outgoing buffer from the local PBs to the coms PB. This buffer is then processed and added to the PB's outgoing transmission queue when the coms PB is triggered by a timer. The PB attempts to send the contents of the transmission queue every timer call.

    Using one of the other PBs to send over the configuration now I have through about it seems to be the best solution as there would be no interruption in either outgoing or ingoing coms just the changes that the new configuration implements taking place.
  13. mric Trainee Engineer

    Why would custom data be corrupted ?
    Can PB run in parallel ? I thought PB execution was not threaded.
  14. Malware Master Engineer

    No parallel, no threading.
    • Agree Agree x 1
  15. Martin R Wolfe Trainee Engineer

    The corruptin is data loss. CustomData may have changed betwenn opening the CustomData dialogue and hitting the OK button.

    1. CustomData="Transmit: Grid2, ProgBlock3, TurnOnBattery: All;"
    2. Player opens CustomData dialogue box
    3. Timer triggers Comms PB
    4. CommBP processes CustomData
    5. CustomData is now an empty string
    6. User adds a line in the dialogue box which instructs the Coms BP to change configuration
    7. A PB adds a transmit instruction to CustomData
    8. User then hits ok which now leaves CustomData containing the original transmit instruction and the configuration without the new transmit instruction.
    Last edited: Mar 10, 2017
  16. Wicorel Senior Engineer

    Why put transmit requests into another pb custom data? Just have the transmitting pb keep its own queue. It's not that much code...
  17. Martin R Wolfe Trainee Engineer

    The Coms PB also detects if the target is on the local grid and only sends over antennas if not fond there. So if it is transmitting to a ship or drone it first checks if it is docked to the local grid. It also encrypts outgoing data so if data is being accepted from allies or enemies the source of incoming data can be determined as to which encryption key decodes it.
  18. gothosan Junior Engineer

    I was thinking of a better approach of using the unique block ID of each antenna and keeping a list of that in the Storage: this way you could know if a message is friendly or owned or enemy and target messages to specifice grids even over radio broadcast.
  19. Wicorel Senior Engineer

    I think block ids change on game load...
  20. gothosan Junior Engineer

    Just made a fast test, block ID stay the same on reload
  21. Malware Master Engineer

    @Wicorel @gothosan The entire purpose of the IDs are to uniquely identify that particular entity. Yes, are serialized to the save game and is retained. Grid Id's however might change during merge/unmerge.
  22. gothosan Junior Engineer

    It might also be a good way to get blocks from local grid without the need to check if they belong to local grid, for that their ID can be saved into the Storage.
  23. Martin R Wolfe Trainee Engineer

    So something like check ID against storage of allowed blocks. If that fails its an unknown block and it will either need to be rejected or checked against the local grid depending on how the list of known blocks was built in the first place. So once a block is known it no longer matters what grid it is on and the name of said grid.

    That is fine for checking allowed blocks but if the block changes grid weather it is currently connected to the local grid structure will still need to be checked although this would now be very simple as it is ID being checked not CustomNames of the block and grid.
  24. gothosan Junior Engineer

    Yes, I basically use that as part of the communication protocol to know from which antenna a message was received and since I'm still working on my OS script this will kinda add immersion while end user won't need to mess too much with special block names for the script to run.
  25. LightDemon Trainee Engineer

    Is this system working for anyone else? I have a programmable block selected in my primary antenna config but that block is not run when a message is transmitted from a separate antenna and programmable block. I tried the toggle broadcast trick but it didn't make a difference.
  26. gothosan Junior Engineer

    The antenna need to be set to trigger the PB, also you need to check who send the message.
    Is it your own antenna, an allied one or enemy one?
    I can suggest also to make a simple output on the PB, like Echo(argument); to write the received argument to the PB ditailbox.
    If all that fail maybe there is a bug.
  27. Wicorel Senior Engineer

    I'm not having any problems.

    You didn't say "separate grid'. The messages are only transmitted between grids and not the same grid. So in addition to what gothosan said, check that.
  28. LightDemon Trainee Engineer

    Yes I am broadcasting from a separate grid an own both antennas and program blocks. I'll keep poking at it and see if i figure anything out. At least it sounds like I'm trying to do it correctly.
  29. Fengist Trainee Engineer

    Works for me. I have 2 drones chatting back and forth sending target info.

    What I AM having issues with is;

    myAntenna.SetValueBool("IsBroadcasting", true); 
    That's giving me a null ref and I thought I copied it pretty much from the OP (I tried it with and without the bool). Anyone else know if this value isn't working any more????

    Nevermind, figured it out. When all else fails, spit the actions out to an LCD... right?

    Antenna.SetValueBool("EnableBroadCast", false);
    Last edited: Apr 23, 2017
  30. gothosan Junior Engineer

    Did you first made the reference to a block?
    myAntenna = GridTerminalSystem.GetBlockWithName(antennaName) as IMyRadioAntenna;