I understand the complexity in python of tying together Flask + Celery + DB + app server etc, but what would you use instead (your comment seems to imply there's a language/framework that has the aforementioned things bundles/more integrated)?
In terms of languages that facilitate doing work concurrently or in parallel with the HTTP request/response cycle:
- Java / Scala / Clojure
- Go
- Erlang (add Elixir if you are comfortable with its maturity)
Not meant to be an exhaustive list of languages. Celery is great but there are plenty of situations where chucking units of work into a job queue or running a cron job feels like an incomplete solution.
These for the most part don't help you persist/offload/schedule asynchronous jobs off the main process. If your process dies or gets restarted, the async job dies with it. Celery persists, queues, schedules and distributes jobs for you, it's different.
For something more similar to Go/Java/Scala/Clojure/whatever, you can use Python's gevent or asyncio.
Implementing a task queue is not that hard in Django's management commands. See https://docs.djangoproject.com/en/1.8/howto/custom-managemen... - it covers 90% of the what you'll need. When you start to scale out, you might need a different solution, but this will work and be easy to deploy for the mostly everything.
As an example - I use this to create contracts with LaTeX asynchronously. The contract is created and then sent to the user by email. A management command runs every 5 minutes to check if there are any contracts that need to be generated.
the .net ecosystem is the kind of thing i was thinking about, but i'm pretty sure java has the same kind of solution.
as for the other persons answering me: i'm not only talking about long request. think about something like scheduling emailings or reports or ios notification , periodically or based on some db state change.
not something you'd want to handle with just an async framework.
the downvote made me look for a more precise example , so look at quartz for a robust enterprise grade lib for jobs and task management ( and its .net alternative).
i suppose celery is, on the paper, comparable in terms of fearures, yet the one time i had to use it made me feel like playing with a brittle toy, and in the end, i had to use a "recycle on every jobs" setting to prevent memory leaks from crashing everything. not that i believe celery is leaking memory, but that tying all the pieces together is a real pita.
