add_requests with a token that has an expiry time

[DoctorEvil92]

I would use add_requests to add a lot of requests to be scheduled, these requests would run for a while. If I schedule each of these requests with a token that has a 30 minute expiry time, at some point all the requests will start failing and the token has to be renewed.

My idea was this: once the request fails, get a new token value and use send_request to retry the request. But then I also need to send this token value to every other request in the queue somehow, I don't know if that is possible?

[janbuchar]

Hello @DoctorEvil92 and thanks for your interest in Crawlee! Just to be sure, what kind of token are you talking about?

[DoctorEvil92]

Auth token that has to be passed in Authorization header, like {"Authorization":"Bearer ..."} It's needed on a site for it to give you data.
I was doing some research, would this be done with pre_navigation_hook? Because the thing is that if you call a lot of requests with the same token, they will eventually start failing when the token expires.