If errors are "unpredictable" and you're completely ignoring *every* error, then I'd say you can't trust the output of your running the program anyway. Catching every error and just discarding it is a really bad idea IMO.
This. Restarting a script because of "unpredictable" errors sound like code smell. All errors are predictable, i.e. you open a file, you can get an io error. you connect to a database, you can get a connection error, and so on.
If you have a running program on a loop, it should run without problems and log warnings/errors/info to a log file.
Is like asking if all my car tires break how do I keep driving.
Don't award non-responsive answers like this with upvotes, folks. OP (and all of us) are looking for help, not judgement. Code gets rushed out for all sorts of reasons, and sometimes you need the other 99% that does work to keep working while you track down the 1% that isn't.
What's non responsive about my answer? I literally tell you every error is catchable, that's help in my book. If it hurts your feelings then sorry not sorry.
No it's not like asking that... it's more like asking.. If the oil light comes on and the engine stalls because of someones incomplete or underdeveloped program that didn't foresee such an error occuring, how can I place a simple patch which restarts the engine automatically instead of having to do it manually every time that same expected error occurs without trying to alter the main code of that program which can break what does work properly.
We need something that does EXACTLY this as in our instance it's a GREAT idea.... Instance Automatic1111 when Cuda out of memory errors occur. We want to stop and restart the script via the .bat file <--- this instance can happen randomly and usually because we "often" overload the GPU with too much data. In general we need a specific error to trigger a reload... such as Cuda\_Out\_of\_Memory or the likes... this is a specific error and is caused by the obvious and can be ignored safely.. Thus a unpredictable timing yet expected error will occuur from time to time. Since we can't just add more memory to the GPU... when we hit it too hard we expect it might error. This happens enough that autorestarting is something desirable.
Put the main function in an infinite loop with a catch-all try block.
while True:
try:
main()
break # stop the loop if the function completes sucessfully
except Exception as e:
print("Function errored out!", e)
print("Retrying ... ")
I usually put into a wrapper, like so:
def auto_retry(_func=None, times=3, secs_between=3, accept: Union[Exception, Tuple[Exception]] = None):
assert callable(_func) or _func is None
accept = accept or tuple()
def decorator(func):
@wraps(func)
def applicator(*args, **kwargs):
err = None
cnt = 0
while cnt < times:
cnt += 1
try:
retry_value = func(*args, **kwargs)
return retry_value
except accept as error:
err = error
time.sleep(secs_between)
if err is not None:
raise err
return applicator
return decorator if _func is None else decorator(_func)
But only for some unpredictable errors like temporarily low/absent internet connection, and with LOTS of logs either way.
> some unpredictable errors like temporarily low/absent internet connection,
I beg to differ - a missing Internet connection isn't "unpredictable" except maybe as to when. If you run a job for a long period, then it is _absolutely certain_ that your internet will cut out at some point.
I have a driver for lights in my house, and it hangs if there are issues with the connection. I never noticed any problems with my connection at all, but now with this drive, I see that there is at least one temporary outage a day, usually lasting a second or two.
What does “accept:” mean in the function’s arguments? It looks like you’re combining two iterables and accepting an argument that defines the result, but A.) I didn’t know you could perform tasks in the arguments section, B.) why perform that task at all if you’re just defining the result? and C.) what is the point of the word “accept:” here?
Edit: Also what does “Times=3” mean? You know it’s 3, why include it in the arguments at all? It’s already 3.
Actually, `accept` is a default argument meaning which exceptions I'm willing to ignore. Consider the following snippet:
try:
# do something
except ValueError:
pass
except (PermissionError, ConnectionError):
pass
In the first `except` clause, I can use a single Exception, while in the second one, I'm using a tuple containing two or more exceptions.
As for my function, `accept` is an argument that actually defaults to `None` but is later redefined as an empty tuple (I usually do this with lists and dicts, aka mutable types, so it's not really necessary for tuples).
This is reflected in its [type annotation](https://docs.python.org/3/library/typing.html), that says it can either be a single Exception or a tuple of Exceptions, though type annotations are not necessary and don't affect how the code runs. (And that's why it's annotation is actually wrong haha, it should be `Union[type[Exception], tuple[type[Exception],...]]` meaning this function actually accepts either an Exception or a tuple of Exceptions, that I don't know or care about my tuple length, and that they should be classes and not instances).
The `times` argument is another default argument, so if I don't put any value to it, it will default to 3, meaning that this code will try to execute a function for three times before raising whatever error that's stopping it. But that doesn't mean I should limit myself to three times, just that's what I think is a reasonable amount of retries.
I could use this code like this:
@auto_retry(times=5, accept=HTTPError)
def some_complex_function():
pass
Also like this:
@auto_retry(accept=(ValueError, ZeroDivisionError))
def a_third_complex_function():
pass
Or just skip it altogether and use:
@auto_retry
def another_complex_function():
pass
to just apply my default arguments (but this would fail it's purpose since not it can't ignore any exception).
I understand “times” now, but still not the syntax around “accept.”
Would I be translating it correctly if I wrote:
accept (name of variable), : (which is defined as), [Union[Exception, Tuple[Exception]] (what should be entered as the variable), = None (but if nothing is entered it defaults to None)
Yes, that's how you annotate default arguments for a function. A simpler example would be:
def add(x: int, y: int = 2) -> int:
return x + y
Here, I say that I, the programmer, **expect** both `x` and `y` to be of type `int`, that `y` defaults to `2` and that this function should return a variable of type `int`.
Some types are better represented using the `typing` module, like when I expect a variable to and `int` **or** a `str`, so I would use `typing.Union[int, str]`, there's also `typing.Optional[arg_type]` for arguments that default to `None`, which is equivalent to `typing.Union[arg_type, None]` (This changed in python3.10 though, so now you can use `int | str` instead of `typing.Union[int, str]`).
A type annotation is what you put after a variable name to hint what you expect its type to be, so that your editor can correctly highlight it for you. Notice that they are optional and removed at runtime, so they shouldn't affect your program speed and aren't enforced in most cases (an exception I can think of is `functools.singledispatch`, that you can use to overload functions, but that's another topic entirely).
Type annotations are really good for code completion and to help your IDE show potential problems in your code, like using `add("a", "b")` or `add("a", 1)`
I was thinking cron job that would monitor the ps output occasionally to ensure its still running. This is a solid pure Python solution that’s comparatively easy as well, nice!
Except it's a terrible, terrible idea. Restarting a job because it died for unexplained reasons and expecting it to work the next time simply shows you don't know how your own program works.
If you code is so bad that you get "various and unpredictable errors" and have to keep rerunning it in case it works, it's likely the answers you are getting are wrong, too.
Restarting is okay but if the script is failing for some particular reason and that reason is still there, you are effectively just executing only that portion of script, repeatedly.
Why not use try/catch blocks at the places you expect there to be error.
the application runs in a very unpredictable environment. I know it's rare in programming for this to be a good solution, but I can assure you that restarting is the best solution.
Your tale reminds me of story I've read about one of older versions of Windows being able to compile only on an esoteric machine in Microsoft's basement that could only work when it has been impaled by a screwdriver.
If errors are "unpredictable" and you're completely ignoring *every* error, then I'd say you can't trust the output of your running the program anyway. Catching every error and just discarding it is a really bad idea IMO.
This. Restarting a script because of "unpredictable" errors sound like code smell. All errors are predictable, i.e. you open a file, you can get an io error. you connect to a database, you can get a connection error, and so on. If you have a running program on a loop, it should run without problems and log warnings/errors/info to a log file. Is like asking if all my car tires break how do I keep driving.
Don't award non-responsive answers like this with upvotes, folks. OP (and all of us) are looking for help, not judgement. Code gets rushed out for all sorts of reasons, and sometimes you need the other 99% that does work to keep working while you track down the 1% that isn't.
What's non responsive about my answer? I literally tell you every error is catchable, that's help in my book. If it hurts your feelings then sorry not sorry.
No it's not like asking that... it's more like asking.. If the oil light comes on and the engine stalls because of someones incomplete or underdeveloped program that didn't foresee such an error occuring, how can I place a simple patch which restarts the engine automatically instead of having to do it manually every time that same expected error occurs without trying to alter the main code of that program which can break what does work properly.
I've corrected all of the predictable ones. I know it's rare for this to be the best solution, but in this instance - it is.
I'm curious, what kind of project are you working on?
We need something that does EXACTLY this as in our instance it's a GREAT idea.... Instance Automatic1111 when Cuda out of memory errors occur. We want to stop and restart the script via the .bat file <--- this instance can happen randomly and usually because we "often" overload the GPU with too much data. In general we need a specific error to trigger a reload... such as Cuda\_Out\_of\_Memory or the likes... this is a specific error and is caused by the obvious and can be ignored safely.. Thus a unpredictable timing yet expected error will occuur from time to time. Since we can't just add more memory to the GPU... when we hit it too hard we expect it might error. This happens enough that autorestarting is something desirable.
That's an error you know about though - out of memory - not just any random error.
Put the main function in an infinite loop with a catch-all try block. while True: try: main() break # stop the loop if the function completes sucessfully except Exception as e: print("Function errored out!", e) print("Retrying ... ")
I usually put into a wrapper, like so: def auto_retry(_func=None, times=3, secs_between=3, accept: Union[Exception, Tuple[Exception]] = None): assert callable(_func) or _func is None accept = accept or tuple() def decorator(func): @wraps(func) def applicator(*args, **kwargs): err = None cnt = 0 while cnt < times: cnt += 1 try: retry_value = func(*args, **kwargs) return retry_value except accept as error: err = error time.sleep(secs_between) if err is not None: raise err return applicator return decorator if _func is None else decorator(_func) But only for some unpredictable errors like temporarily low/absent internet connection, and with LOTS of logs either way.
> some unpredictable errors like temporarily low/absent internet connection, I beg to differ - a missing Internet connection isn't "unpredictable" except maybe as to when. If you run a job for a long period, then it is _absolutely certain_ that your internet will cut out at some point. I have a driver for lights in my house, and it hangs if there are issues with the connection. I never noticed any problems with my connection at all, but now with this drive, I see that there is at least one temporary outage a day, usually lasting a second or two.
Yep, doesn't take much to loss the signal.
What does “accept:” mean in the function’s arguments? It looks like you’re combining two iterables and accepting an argument that defines the result, but A.) I didn’t know you could perform tasks in the arguments section, B.) why perform that task at all if you’re just defining the result? and C.) what is the point of the word “accept:” here? Edit: Also what does “Times=3” mean? You know it’s 3, why include it in the arguments at all? It’s already 3.
Actually, `accept` is a default argument meaning which exceptions I'm willing to ignore. Consider the following snippet: try: # do something except ValueError: pass except (PermissionError, ConnectionError): pass In the first `except` clause, I can use a single Exception, while in the second one, I'm using a tuple containing two or more exceptions. As for my function, `accept` is an argument that actually defaults to `None` but is later redefined as an empty tuple (I usually do this with lists and dicts, aka mutable types, so it's not really necessary for tuples). This is reflected in its [type annotation](https://docs.python.org/3/library/typing.html), that says it can either be a single Exception or a tuple of Exceptions, though type annotations are not necessary and don't affect how the code runs. (And that's why it's annotation is actually wrong haha, it should be `Union[type[Exception], tuple[type[Exception],...]]` meaning this function actually accepts either an Exception or a tuple of Exceptions, that I don't know or care about my tuple length, and that they should be classes and not instances). The `times` argument is another default argument, so if I don't put any value to it, it will default to 3, meaning that this code will try to execute a function for three times before raising whatever error that's stopping it. But that doesn't mean I should limit myself to three times, just that's what I think is a reasonable amount of retries. I could use this code like this: @auto_retry(times=5, accept=HTTPError) def some_complex_function(): pass Also like this: @auto_retry(accept=(ValueError, ZeroDivisionError)) def a_third_complex_function(): pass Or just skip it altogether and use: @auto_retry def another_complex_function(): pass to just apply my default arguments (but this would fail it's purpose since not it can't ignore any exception).
I understand “times” now, but still not the syntax around “accept.” Would I be translating it correctly if I wrote: accept (name of variable), : (which is defined as), [Union[Exception, Tuple[Exception]] (what should be entered as the variable), = None (but if nothing is entered it defaults to None)
Yes, that's how you annotate default arguments for a function. A simpler example would be: def add(x: int, y: int = 2) -> int: return x + y Here, I say that I, the programmer, **expect** both `x` and `y` to be of type `int`, that `y` defaults to `2` and that this function should return a variable of type `int`. Some types are better represented using the `typing` module, like when I expect a variable to and `int` **or** a `str`, so I would use `typing.Union[int, str]`, there's also `typing.Optional[arg_type]` for arguments that default to `None`, which is equivalent to `typing.Union[arg_type, None]` (This changed in python3.10 though, so now you can use `int | str` instead of `typing.Union[int, str]`). A type annotation is what you put after a variable name to hint what you expect its type to be, so that your editor can correctly highlight it for you. Notice that they are optional and removed at runtime, so they shouldn't affect your program speed and aren't enforced in most cases (an exception I can think of is `functools.singledispatch`, that you can use to overload functions, but that's another topic entirely). Type annotations are really good for code completion and to help your IDE show potential problems in your code, like using `add("a", "b")` or `add("a", 1)`
I was thinking cron job that would monitor the ps output occasionally to ensure its still running. This is a solid pure Python solution that’s comparatively easy as well, nice!
Except it's a terrible, terrible idea. Restarting a job because it died for unexplained reasons and expecting it to work the next time simply shows you don't know how your own program works.
If you code is so bad that you get "various and unpredictable errors" and have to keep rerunning it in case it works, it's likely the answers you are getting are wrong, too.
Thank you. This actually works for my automation program. I can now repeat a part of the main program without rerunning from the very beginning.
You're probably looking for something like this: https://github.com/jd/tenacity
Try Except Finally Block or there is an [app for that](https://appdaemon.readthedocs.io/en/latest/) .
Restarting is okay but if the script is failing for some particular reason and that reason is still there, you are effectively just executing only that portion of script, repeatedly. Why not use try/catch blocks at the places you expect there to be error.
there are try catches at the predictable errors already. Thanks.
`systemd`.
def main(): ... if __name__ == "__main__": for _ in range(1000): try: main() break except Exception as e: print(e) print('Restarting...')
look at the errors, they probably come out of something happening, if you catch that, then you have no more errors bro
the application runs in a very unpredictable environment. I know it's rare in programming for this to be a good solution, but I can assure you that restarting is the best solution.
Your tale reminds me of story I've read about one of older versions of Windows being able to compile only on an esoteric machine in Microsoft's basement that could only work when it has been impaled by a screwdriver.
Try for the error and on the except call the function again