Many may have noticed that amsat.org was offline for about 20 hours yesterday. We are investigating the root cause. Something impacted the TCP stack on the server. That caused the telemetry server to spin out of control. In that state it did two things, neither of them good and neither of them in the design spec.
Firstly, the telemetry server spewed error messages to the log at such a rate that it impacted the website and other systems on amsat.org. A fix will be implemented so this does not happen again.
Secondly, the telemetry server accepted connections from FoxTelem, confirmed it had received the data and then failed to process it. This should not be possible given the error handling logic, so we are trying to work out how to prevent in the future.
Those of you who keep track of your uploads, will notice that some records were not updated yesterday, even though they were sent. I'm going to release a tool to allow re-upload of the missing day. Please don't purge your FOXDB files if you have data.
Let me know if you have questions or suggestions 73
Chris
Let's blame Clint ;)
Thanks Chris, Joe, whomever...for making it go again.
Mark N8MH
On Tue, Feb 27, 2018 at 10:24 AM, Chris Thompson g0kla@arrl.net wrote:
Many may have noticed that amsat.org was offline for about 20 hours yesterday. We are investigating the root cause. Something impacted the TCP stack on the server. That caused the telemetry server to spin out of control. In that state it did two things, neither of them good and neither of them in the design spec.
Firstly, the telemetry server spewed error messages to the log at such a rate that it impacted the website and other systems on amsat.org. A fix will be implemented so this does not happen again.
Secondly, the telemetry server accepted connections from FoxTelem, confirmed it had received the data and then failed to process it. This should not be possible given the error handling logic, so we are trying to work out how to prevent in the future.
Those of you who keep track of your uploads, will notice that some records were not updated yesterday, even though they were sent. I'm going to release a tool to allow re-upload of the missing day. Please don't purge your FOXDB files if you have data.
Let me know if you have questions or suggestions 73
Chris
-- Chris E. Thompson chrisethompson@gmail.com g0kla@arrl.net _______________________________________________ Sent via AMSAT-BB@amsat.org. AMSAT-NA makes this open forum available to all interested persons worldwide without requiring membership. Opinions expressed are solely those of the author, and do not reflect the official views of AMSAT-NA. Not an AMSAT-NA member? Join now to support the amateur satellite program! Subscription settings: http://www.amsat.org/mailman/listinfo/amsat-bb
participants (2)
-
Chris Thompson -
Mark L. Hammond