Timeout throws unhandled exception

[somme89]

Observed in releases 2024.1.0 and 2024.2.0

When running an SshCommand with a timeout there's a risk of getting an unhandled exception if the underlying connection is disconnected/disposed before the command completes. The command will hang until the timeout is triggered and when the the timeout is triggered an exception is raised which cannot be caught.

Timeouts are implemented using a cancellation token that executes a delegate command when triggered. The problem seems to be that the delegate does not catch any exceptions thrown when it attempts to cancel the SshCommand and exceptions are not propagated to the client application.

SSH.NET/src/Renci.SshNet/SshCommand.cs

Lines 292 to 304 in 29997ae

	_ = _channel.SendExecRequest(CommandText);
	if (CommandTimeout != Timeout.InfiniteTimeSpan)
	{
	_cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
	_cts.CancelAfter(CommandTimeout);
	cancellationToken = _cts.Token;
	}
	if (cancellationToken.CanBeCanceled)
	{
	_tokenRegistration = cancellationToken.Register(static cmd => ((SshCommand)cmd!).CancelAsync(), this);
	}

Code where the exception happens:

 try
    {
        cmdObj = ssh.CreateCommand(cmd);
        cmdObj.CommandTimeout = TimeSpan.FromSeconds(timeout);
        cmdObj.Execute();
        int exitStatus = cmdObj.ExitStatus ?? -1;
        res = (cmdObj.Result, exitStatus, cmdObj.Error);
        cmdObj?.Dispose();
        success = true;
    }
    catch (Exception e)
    {
        Log.Error(e.ToString());
    }

The code works well under normal circumstances, but after we wrote an integration test that causes the connection (ssh) to be disconnected immediately after a command is executed, the problem was discovered.

The exception causes the application to terminate with the following stacktrace captured using an exception logger attached to "AppDomain.CurrentDomain.UnhandledException"

---> Renci.SshNet.Common.SshConnectionException: Client not connected.
at Renci.SshNet.Session.SendMessage(Message message)
at Renci.SshNet.Channels.ChannelSession.SendSignalRequest(String signalName)
at Renci.SshNet.SshCommand.CancelAsync(Boolean forceKill, Int32 millisecondsTimeout)
at Renci.SshNet.SshCommand.<>c.b__43_0(Object cmd)
at System.Threading.CancellationTokenSource.Invoke(Delegate d, Object state, CancellationTokenSource source)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException)
--- End of inner exception stack trace ---
at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException)
at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool)
at System.Threading.TimerQueue.FireNextTimers()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
2024-12-24 02:22:44.990 +00:00 [FTL] Inner exception: Renci.SshNet.Common.SshConnectionException: Client not connected.
at Renci.SshNet.Session.SendMessage(Message message)
at Renci.SshNet.Channels.ChannelSession.SendSignalRequest(String signalName)
at Renci.SshNet.SshCommand.CancelAsync(Boolean forceKill, Int32 millisecondsTimeout)
at Renci.SshNet.SshCommand.<>c.b__43_0(Object cmd)
at System.Threading.CancellationTokenSource.Invoke(Delegate d, Object state, CancellationTokenSource source)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException)

A workaround has been implemented by wrapping the command execution in a separate task and using a Task.Delay(timeout) to implement the timeout. Using the built-in SshCommand.CommandTimeout causes the unhandled exception to be raised.

[Rob-Hague]

Thanks, I have put up a change to swallow the exception in the cancellation callback. The change fixes a resulting indefinite hang upon client disconnection, but it will still wait until the timeout before the task is complete. Currently a client disconnect event is not propagated to the right places for a full fix

[somme89]

Thank you Rob!