Why is there an issue with Steam Workshop mod downloads?

I've been trying to troubleshoot problems with some mods, and I've been told repeatedly that the problems are due to corruption of the mods during download; that to fix them, you should delete the mods and force them to redownload. Rexxar said once in chat that the SE client does validate files, but they're being corrupted on Steam's end.

I've only seen this issue mentioned with a few mods, and I haven't found any references to it being a problem with other games with mods on the Steam Workshop. I also haven't seen it mentioned very often on this forum.

Robust data transmission is pretty much the basis of the Internet; error detection is built into the network stack at several layers. If the data is being corrupted, I'd guess it's either happening through some mangling of data at either end of the pipe, when it's stored on Steam's servers or when it's stored by our clients. Either way, this would look to me to be a very serious bug.

It seems much more likely to me that there is some other issue that's being misattributed to mod corruption.

 

Are you certain that those mods weren't broken by an update? There are lots of mods that are created then abandoned after the author loses interest.

 

It'd make sense that the mods were broken by updates. The mod author is still maintaining the mods, however, so it's not an issue of abandonment. Partly, I'm trying to get some feedback on whether this is really a general problem, rather than just a problem with a particular set of mods.

I'd write off "data corruption" as a mistaken idea on the mod author's part, except I've seen Rexxar say that mod corruption is an issue, and there's a stickied post on the Steam Forum about troubleshooting mods that says that it happens and to try to fix it by deleting the sbm files to force re-downloads.

And, a couple of the mods did work after forcing a re-download, so something was wrong.

 

I have a similar issue, where my friend's clients seem lazy about downloading mods when joining a peer-to-peer game or a dedicated server. The only way to remedy this is to send them a large compressed file containing all the mod files used on the host. If I don't do this, my friends can't see certain blocks or observe changes that the mods make. Really annoying to play IT when hosting

 

Were you sending them compressed files because they weren't bothering to download the mods, or because the game client or server was corrupting the files?

 

They were subscribed to the collection I had created. However, certain mods were:
1. Not in their "Mods" folder of their game, even after subscribing to the mod and loading into a server.
2. Present in their mods folder but would not show blocks/effects in the game.

The only way to ensure that all mods were loaded is by giving them an uncorrupted copy of all the mods manually. Hence the zip containing all server-side mod files.

I think case 2 is a side-effect of corruption on mod download. I think there might be a checksum mismatch or the checking of mods is not rigorous enough to care about corrupted mods and loads anyway.
This might be worth a look: here and here
Mainly the CheckMismatchmods method and how MySteamWorkshop checks for mismatched mods. Code hasn't been updated in a while, so they might have changed how they verify.

EDIT: Steam goofed everything

 

ENABLE_WORKSHOP_MODS = false; in Sandbox.game.dll and my server start with my mods ;3

 

Okay, so it sounds like there really is a problem with mod corruption. Given how dependent this community is on mods shared through the Steam Workshop, I'd think that calling this a major bug would be an understatement. You have to wonder how many bug reports that testers say they can't replicate, are due to corrupted mods.

I'm not much of a coder, but I don't see anything in MySteamWorkshop about checksumming the files; it looks like it only logs an error if Steam is offline, or if the mod doesn't download at all. There could be functions for that elsewhere, though.

Part of what's puzzled me about this is, if SE downloads files that have been corrupted, but SE detects that they're corrupted, why SE doesn't just repeat the download attempt several times, and if it keeps failing, warn the user and ask if it should keep trying. Failing silently doesn't make sense. And no, just putting in the log doesn't really count as warning the user. And like I was saying, the Internet is built on robust file transfer; i.e., checking for bad packets and resending any that fail. There are error detection routines at multiple levels of the Internet stack. It's not that errors never happen -- they're quite common. It's that you carefully check whether an error may have occurred, through the use of checksums and similar mechanisms, and resend anything that was in error.

That also sounds bad -- though, if Rexxar was right previously, that data was being corrupted on Steam's end, then you'd think Steam changing how they handle mod downloads would probably be good news, once the transition was complete.

In my case, I've been playing exclusively singleplayer, and haven't touched a dedicated server for SE. Most of the mods I've downloaded, and nearly all I've downloaded in terms of file size, are DoctorOctoganapus's planet mods -- and the discussion threads are full of repeated complaints about the planets not showing up, and DoctorOctoganapus explaining that it's probably because the download was corrupt, and you should delete the sbm file and try again. (I feel bad that they're putting so much work into these beautiful planets and systems, and just get stuck answering the same questions over and over again.)

That would be bad enough if it were a rare problem. But judging by those threads, it isn't. I wanted to check it wasn't just those planet mods that were having problems, before mentioning them specifically. At least, when a planet fails to load, it's relatively obvious there's something missing. It's still irritating, as setting up one of those systems involves several rounds of creating a world, checking which planets didn't spawn, deleting those sbm files to force a download, and repeating -- it takes a few hours, given the long time to spawn a new world. But what renewed my concern about this, and makes me more worried, is that I noticed a few planets lacked heightmaps, which DoctorOctoganapus said may also be the result of file corruption -- which would mean, that SE is downloading corrupted files, and still trying to use them.

I've played other games with mods on the Steam Workshop, particularly Skyrim and Skyrim SE; the file sizes of those mods dwarfs the file sizes of the mods for SE I've downloaded, and I don't recall having any trouble with file corruption for Skyrim mods, nor seeing any reports of such trouble. There is a popular site for downloading mods for Bethesda games, outside the Steam Workshop, but my impression was some modders preferred to use it because it allowed for more flexibility in how they created their mods, not because it was working around an issue with file corruption on the Steam Workshop.

If it really is a problem on Steam's end, then we should consider setting up an alternate site for downloading mods.

If it's a problem on Space Engineer's end, then this is a bug that is much more serious than any other bug I've heard of for SE. Reliable, robust file transfer is an absolute necessity for a game dependent on shared files.

EDIT: That other thread about the Steam change is pretty active. Looks like a serious issue.

 

How do you do that? Where are you placing this variable?

Edit: Nvm, found your reply elsewhere.

 

Personally...I would always test a bug with no mod enabled before making a bug report. That way, you can eliminate the mod as a direct cause.

BUT....it is worrying that the mod download process from Steam may be messed up, as that has really serious implications for any mod maker. If your mod is highlighted as a crash causer, you would want to find a solkution, and potentially spend a lot of time looking for something that isn't there.

Also, not everyone is going to want to go through the hassle of deleting the contents of the mod folder and forcing re-download. would REALLY help if all mods had clear identification in them to tell you which mod is which, because the numeric file names are impossible to identify. Some modders don;t even bother with a thubnail logo to help you.

Is there a good reason why the mod name is not part of the mods own folder?

 

No. Steam just sucks. It sucked when it first released, it sucks now, and it has sucked the entire time between release and now.

 

To summarize:
1) Mods are frequently corrupted in download;
2) Space Engineers doesn't always notice;
3) When Space Engineers does notice, it doesn't try to re-download the file;
4) When Space Engineers does notice, it doesn't warn the user that there's a problem.
5) It's a pain-in-the-ass for users to force re-downloads, involving picking through a log file and trying to work out which arbitrarily numbered sbm file is the problem.

#1 is extremely bad, but Keen claims it's Steam's fault, not Keen's. But even assuming that's the case, I'm not seeing what Keen's done about the aspects of the problem they could address, 2-5.

What's most astonishing here is that this is a major problem that's apparently just being completely ignored.
--- Automerge ---
Shifting to a pro-active question: is there any reason, legal or technical, why we couldn't have a community-sponsored website for Space Engineer mods, independent of the Steam Workshop?

 

The file names aren't arbitrarily named. The numbers of the mod correspond with the numbers in the URL of the mods home page.

If you go into your mod folder, you can manually delete the mods and they will reload when you attempt to connect to the server again. If you examine the server before joining it, you'll get a list of mods. You can even open a webpage to each of them in Steam's browser. Write down the numbers and then find them in the mod folder. Select them and simply delete them.

Mod corruption isn't new, it just seems to be more frequent at the moment.

 

Well, arbitrary in the sense that the filenames aren't directly meaningful to an end user. You've got to go through a few steps to determine that it's the planet Jynu that isn't spawning, so you need to delete "%AppData%/Space Engineers/Mods/700672566.sbm".

I don't know if the mod corruption is happening more frequently, but I remember noticing it months ago, the first time I tried out the Novana System, and several of the planets didn't spawn. I'd worked through the problem, iteratively downloading mods until I got them all working, and didn't really think about it again, until I decided to try some of his older stuff. Then I was trying to figure out how many of the problems were due to changes in SE that the planet mods hadn't been updated to cope with, and how many were due to mod corruption. Last night, I was trying to re-download the Novana System collection again, and I had to re-download some of the mods several times before I had them all.

There's a lot of steps to go through to recognize there's a problem, find the correct procedure to fix the problem, and find the details -- pathnames and so forth -- you need, and then it takes a while to do it and to verify the problem's been fixed. I'm an IT person (as are a lot of us here), so this isn't wildly unfamiliar to me, but it would be faster and more reliable if it was done automatically. And even glancing at the discussion threads for some of the mods I'm talking about, it's clear that many users ask for help after noticing things are broken, but then throw up their hands and decide to put up with broken mods, because the fix is too much trouble. Imagine how many end users don't get as far as looking at those discussion threads.

This all feels like a throwback to the 80s, when I was trying to download files from BBS systems using XMODEM, and it only usually worked. Except, we've had better protocols for decades now. My day job is testing device drivers for network cards; I know that there's more sophisticated error correction in the hardware than we had at the application layer in the 80s. I'm not sure where in the process the problem is occurring, and it's surprising that it's happening at all. But there are definitely parts of the download process under Keen's control where they could implement verification and re-download files that fail verification. Quick-and-dirty method: clients hash files on upload, and verify hashes on download. Choose a good hash algorithm, and the chances of a hash collision are negligible.

 

I'm a child of Bell Labs and Hayes AT commands myself. Parity bits anyone?

So this is mainly happening on Workshop items that contain modded planets? Maybe it's a texture thing?