When tps_shutdown is called as part of the cleanup process there is a
chance that one of the taskprocessors that references the
tps_singletons object is still running. The change is to allow for
tps_shutdown to check tps_singleton's container count and give the
running taskprocessors a chance to finish. If after
AST_TASKPROCESSOR_SHUTDOWN_MAX_WAIT (10) seconds there are still
container references we shutdown anyway as this is most likely a bug
due to a taskprocessor not being unreferenced.
ASTERISK-29365
Change-Id: Ia932fc003d316389b9c4fd15ad6594458c9727f1
Serializer pools have previously existed in Asterisk. However, for the most
part the code has been duplicated across modules. This patch abstracts the
code into an 'ast_serializer_pool' object. As well the code is now centralized
in serializer.c/h.
In addition serializer pools can now optionally be monitored by a shutdown
group. This will prevent the pool from being destroyed until all serializers
have completed.
Change-Id: Ib1e906144b90ffd4d5ed9826f0b719ca9c6d2971
Added "like" support for 'core show taskprocessors'. Now you
can specify a specific set of taskprocessors (or just one) by
adding the keyword "like" to the above command, followed by
your search criteria.
Change-Id: I021e740201e9ba487204b5451e46feb0e3222464
Added two new CLI commands to reset stats for taskprocessors. You can
reset stats for a single, specific taskprocessor ('core reset
taskprocessor <taskprocessor>'), or you can reset all taskprocessors
('core reset taskprocessors'). These commands will reset the counter for
the number of tasks processed as well as the max queue size.
Change-Id: Iaf17fc4ae29396ab0c6ac92408fc7bdc2f12362d
Since the new names went in, the maximum taskprocessor name is too
short. This patch increases the name field to a length to better
handle the new names.
Change-Id: I32f32d6926f25c8ef5a91303fd2988d2c2858877
A size_t is not always an unsigned long.
* Use the %zu format specifier in the ast_cli() printf format string since
AST_VECTOR_SIZE() returns a size_t value.
Change-Id: Ib102dd36bbe6c2a7a4ce6870ae9110d978dd7e98
To prevent one subsystem's taskprocessors from causing others
to stall, new capabilities have been added to taskprocessors.
* Any taskprocessor name that has a '/' will have the part
before the '/' saved as its "subsystem".
Examples:
"sorcery/acl-0000006a" and "sorcery/aor-00000019"
will be grouped to subsystem "sorcery".
"pjsip/distributor-00000025" and "pjsip/distributor-00000026"
will bn grouped to subsystem "pjsip".
Taskprocessors with no '/' have an empty subsystem.
* When a taskprocessor enters high-water alert status and it
has a non-empty subsystem, the subsystem alert count will
be incremented.
* When a taskprocessor leaves high-water alert status and it
has a non-empty subsystem, the subsystem alert count will be
decremented.
* A new api ast_taskprocessor_get_subsystem_alert() has been
added that returns the number of taskprocessors in alert for
the subsystem.
* A new CLI command "core show taskprocessor alerted subsystems"
has been added.
* A new unit test was addded.
REMINDER: The taskprocessor code itself doesn't take any action
based on high-water alerts or overloading. It's up to taskprocessor
users to check and take action themselves. Currently only the pjsip
distributor does this.
* A new pjsip/global option "taskprocessor_overload_trigger"
has been added that allows the user to select the trigger
mechanism the distributor uses to pause accepting new requests.
"none": Don't pause on any overload condition.
"global": Pause on ANY taskprocessor overload (the default and
current behavior)
"pjsip_only": Pause only on pjsip taskprocessor overloads.
* The core pjsip pool was renamed from "SIP" to "pjsip" so it can
be properly grouped into the "pjsip" subsystem.
* stasis taskprocessor names were changed to "stasis" as the
subsystem.
* Sorcery core taskprocessor names were changed to "sorcery" to
match the object taskprocessors.
Change-Id: I8c19068bb2fc26610a9f0b8624bdf577a04fcd56
Replace usage of ao2_container_alloc with ao2_container_alloc_hash or
ao2_container_alloc_list. Remove ao2_container_alloc macro.
Change-Id: I0907d78bc66efc775672df37c8faad00f2f6c088
Task processors are retrieved using a 'get or create' pattern. The
singleton container was unlocked between the get and create steps so
it's possible that two threads could create task processors with the
same name at the same time.
Change-Id: Id64fae94a6a1e940ddf38fde622dcd4391635382
Merge storage for the stats object and name string into the main
allocation for struct ast_taskprocessor.
Change-Id: I74fe9a7f357f0e6d63152f163cf5eef6428218e1
Add attribute_warn_unused_result to ast_taskprocessor_push,
ast_taskprocessor_push_local and ast_threadpool_push. This will help
ensure we perform the necessary cleanup upon failure.
Change-Id: I7e4079bd7b21cfe52fb431ea79e41314520c3f6d
Since v12 the number of taskprocessors in the system has increased a lot.
Small systems can easily have over a hundred and larger systems can have
thousands.
Most uses of the tps_singletons container deal with creating and
destroying the taskprocessors. However, the pjsip distributor looks up
taskprocessors/serializers by name frequently. It needs to find the
serializer for incoming SIP responses to distribute them to the
appropriate serializer.
Change-Id: Ice0603606614ba49f7c0c316c524735c064e7e43
The task processor queue reached X scheduled tasks message was originally
intended to get logged only once per task processor to prevent spamming
the log. This is no longer necessary since high and low water thresholds
can better control when the message is logged.
It is beneficial to generate the warning each time a task processor
reaches the high water level because PJSIP stops processing new requests
while any high water alert is active. Without this change you would have
to enable at least debug level 3 logging to know about a repeated alert
trigger.
* Made generate the warning message whenever a task is pushed into the
task processor that triggers the high water alert.
* Appended 'again' to the warning for a repeated high water alert trigger.
Change-Id: Iabf75a004f7edaf1e5e8c323099418e667cac999
ASTERISK_REGISTER_FILE no longer has any purpose so this commit removes
all traces of it.
Previously exported symbols removed:
* __ast_register_file
* __ast_unregister_file
* ast_complete_source_filename
This also removes the mtx_prof static variable that was declared when
MTX_PROFILE was enabled. This variable was only used in lock.c so it
is now initialized in that file only.
ASTERISK-26480 #close
Change-Id: I1074af07d71f9e159c48ef36631aa432c86f9966
* The high water check in ast_taskprocessor_alert_set_levels() would
trigger immediately if the new high water level is zero and the queue was
empty.
* The high water check in taskprocessor_push() was off by one.
Change-Id: I687729fb4efa6a0ba38ec9c1c133c4d407bc3d5d
If both channels which should be masqueraded
are in the same serializer:
1st channel will be locked waiting condition 'complete'
2nd channel will be locked waiting condition 'suspended'
On heavy load system a chance that both channels will be in
the same serializer 'pjsip/distibutor' is very high.
To reproduce compile res_pjsip/pjsip_distributor.c with
DISTRIBUTOR_POOL_SIZE=1
Steps to reproduce:
1. Party A calls Party B (bridged call 'AB')
2. Party B places Party A on hold
3. Party B calls Voicemail app (non-bridged call 'BV')
4. Party B attended transfers Party A to voicemail using REFER.
5. When asterisk masquerades calls 'AB' and 'BV',
a deadlock is happened.
This patch adds a suspension indicator to the taskprocessor.
When a session suspends/unsuspends the serializer
it sets the indicator to the appropriate state.
The session checks the suspension indicator before
suspend the serializer.
ASTERISK-26145 #close
Change-Id: Iaaebee60013a58c942ba47b1b4930a63e686663b
When taskprocessors get backed up, there is a good chance that we are
being overloaded and need to defer adding new work to the system.
* Implemented a high/low water alert mechanism for modules to check if the
system is being overloaded and take appropriate action. When a
taskprocessor is created it has default congestion levels set. A
taskprocessor can later have those congestion levels altered for specific
needs if stress testing shows that the taskprocessor is a symptom of
overloading or needs to handle bursty activity without triggering an
overload alert.
* Add CLI "core show taskprocessor" low/high water columns.
* Fixed __allocate_taskprocessor() to not use RAII_VAR(). RAII_VAR() was
never a good thing to use when creating a taskprocessor because of the
nature of how its references needed to be cleaned up on a partial
creation.
* Made res_pjsip's distributor check if the taskprocessor overload alert
is active before placing a message representing brand new work onto a
distributor serializer.
ASTERISK-26088
Reported by: Richard Mudgett
Change-Id: I182f1be603529cd665958661c4c05ff9901825fa
You have to call ast_taskprocessor_unref() outside of the taskprocessor
implementation code. Taskprocessor use since v12 has become more
transient than just the singleton uses in earlier versions.
Change-Id: If7675299924c0cc65f2a43a85254e6f06f2d61bb
* Add new API call to get a sequence number for use in human friendly
taskprocessor names.
* Add new API call to create a taskprocessor name in a given buffer and
append a sequence number.
Change-Id: Iac458f05b45232315ed64aa31b1df05b875537a9
Update the CLI "core show taskprocessors" output format to not be
distorted because UUID names are longer than previously used taskprocessor
names.
Change-Id: I1a5c82ce3e8f765a0627796aba87f8f7be077601
In practical tests, we have seen certain taskprocessors, specifically
Stasis subscription taskprocessors, cross the recently-added high-water
mark and emit a warning. This high-water mark warning is only intended
to be emitted when things have tanked on the system and things are
heading south quickly. In the practical tests, the Stasis taskprocessors
sometimes had a max depth of 180 tasks in them, and Asterisk wasn't in
any danger at all.
As such, this ups the high-water mark to 500 tasks instead. It also
redefines the SIP threadpool request denial number to be a multiple of
the taskprocessor high-water mark.
Change-Id: Ic8d3e9497452fecd768ac427bb6f58aa616eebce
We have observed situations where the SIP threadpool may become
deadlocked. However, because incoming traffic is still arriving, the SIP
threadpool's queue can continue to grow, eventually running the system
out of memory.
This change makes it so that incoming traffic gets rejected with a 503
response if the queue is backed up too much.
Change-Id: I4e736d48a2ba79fd1f8056c0dcd330e38e6a3816
If a taskprocessor's queue grows large, this can indicate that there
may be a problem with tasks not leaving the processor or else that
the number of available task processors for a given type of task is
too low. This patch makes it so that if a taskprocessor's task queue
grows above 100 queued tasks that it will emit a warning message.
Warning messages are emitted only once per task processor.
ASTERISK-25518 #close
Reported by: Jonathan Rose
Change-Id: Ib1607c35d18c1d6a0575b3f0e3ff5d932fd6600c
When unreferencing a taskprocessor its reference count is checked
to determine if it should be unlinked from the taskprocessors
container and its listener shut down. In between the time when the
reference count is checked and unlinking it is possible for
another thread to jump in, find it, and get a reference to it. If
the thread then uses the taskprocessor it may find that it is not
in the state it expects.
This change locks the taskprocessors container during almost the
entire unreference operation to ensure that any other thread which
may attempt to find the taskprocessor has to wait.
ASTERISK-25295
Change-Id: Icb842db82fe1cf238da55df92e95938a4419377c
Git does not support the ability to replace a token with a version
string during check-in. While it does have support for replacing a
token on clone, this is somewhat sub-optimal: the token is replaced
with the object hash, which is not particularly easy for human
consumption. What's more, in practice, the source file version was often
not terribly useful. Generally, when triaging bugs, the overall version
of Asterisk is far more useful than an individual SVN version of a file. As a
result, this patch removes Asterisk's support for showing source file
versions.
Specifically, it does the following:
* Rename ASTERISK_FILE_VERSION macro to ASTERISK_REGISTER_FILE, and
remove passing the version in with the macro. Other facilities
than 'core show file version' make use of the file names, such as
setting a debug level only on a specific file. As such, the act of
registering source files with the Asterisk core still has use. The
macro rename now reflects the new macro purpose.
* main/asterisk:
- Refactor the file_version structure to reflect that it no longer
tracks a version field.
- Remove the "core show file version" CLI command. Without the file
version, it is no longer useful.
- Remove the ast_file_version_find function. The file version is no
longer tracked.
- Rename ast_register_file_version/ast_unregister_file_version to
ast_register_file/ast_unregister_file, respectively.
* main/manager: Remove value from the Version key of the ModuleCheck
Action. The actual key itself has not been removed, as doing so would
absolutely constitute a backwards incompatible change. However, since
the file version is no longer tracked, there is no need to attempt to
include it in the Version key.
* UPGRADE: Add notes for:
- Modification to the ModuleCheck AMI Action
- Removal of the "core show file version" CLI command
Change-Id: I6cf0ff280e1668bf4957dc21f32a5ff43444a40e
Since 'core stop now' and 'core restart now' do not stop modules,
it is unsafe for most of the core to run cleanups. Originally all
cleanups used ast_register_atexit, and were only changed when it
was shown to be unsafe. ast_register_atexit is now used only when
absolutely required to prevent corruption and close child processes.
Exceptions that need to use ast_register_atexit:
* CDR: Flush records.
* res_musiconhold: Kill external applications.
* AstDB: Close the DB.
* canary_exit: Kill canary process.
ASTERISK-24142 #close
Reported by: David Brillert
ASTERISK-24683 #close
Reported by: Peter Katzmann
ASTERISK-24805 #close
Reported by: Badalian Vyacheslav
ASTERISK-24881 #close
Reported by: Corey Farrell
Review: https://reviewboard.asterisk.org/r/4500/
Review: https://reviewboard.asterisk.org/r/4501/
........
Merged revisions 433495 from http://svn.asterisk.org/svn/asterisk/branches/11
........
Merged revisions 433497 from http://svn.asterisk.org/svn/asterisk/branches/13
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@433498 65c4cc65-6c06-0410-ace0-fbb531ad65f3
Performing a directed call pickup resulted in a deadlock when PJSIP
channels were involved.
A masquerade needs to hold onto the channel locks while it swaps channel
information between the two channels involved in the masquerade. With
PJSIP channels, the fixup routine needed to push a fixup task onto the
PJSIP channel's serializer. Unfortunately, if the serializer was also
processing a task that needed to lock the channel, you get deadlock.
* Added a new control frame that is used to notify the channels that a
masquerade is about to start and when it has completed.
* Added the ability to query taskprocessors if the current thread is the
taskprocessor thread.
* Added the ability to suspend/unsuspend the PJSIP serializer thread so a
masquerade could fixup the PJSIP channel without using the serializer.
ASTERISK-24356 #close
Reported by: rmudgett
Review: https://reviewboard.asterisk.org/r/4034/
........
Merged revisions 424471 from http://svn.asterisk.org/svn/asterisk/branches/12
........
Merged revisions 424472 from http://svn.asterisk.org/svn/asterisk/branches/13
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@424473 65c4cc65-6c06-0410-ace0-fbb531ad65f3
........
r399887 | dlee | 2013-09-26 10:41:47 -0500 (Thu, 26 Sep 2013) | 1 line
Minor performance bump by not allocate manager variable struct if we don't need it
........
r400138 | dlee | 2013-09-30 10:24:00 -0500 (Mon, 30 Sep 2013) | 23 lines
Stasis performance improvements
This patch addresses several performance problems that were found in
the initial performance testing of Asterisk 12.
The Stasis dispatch object was allocated as an AO2 object, even though
it has a very confined lifecycle. This was replaced with a straight
ast_malloc().
The Stasis message router was spending an inordinate amount of time
searching hash tables. In this case, most of our routers had 6 or
fewer routes in them to begin with. This was replaced with an array
that's searched linearly for the route.
We more heavily rely on AO2 objects in Asterisk 12, and the memset()
in ao2_ref() actually became noticeable on the profile. This was
#ifdef'ed to only run when AO2_DEBUG was enabled.
After being misled by an erroneous comment in taskprocessor.c during
profiling, the wrong comment was removed.
Review: https://reviewboard.asterisk.org/r/2873/
........
r400178 | dlee | 2013-09-30 13:26:27 -0500 (Mon, 30 Sep 2013) | 24 lines
Taskprocessor optimization; switch Stasis to use taskprocessors
This patch optimizes taskprocessor to use a semaphore for signaling,
which the OS can do a better job at managing contention and waiting
that we can with a mutex and condition.
The taskprocessor execution was also slightly optimized to reduce the
number of locks taken.
The only observable difference in the taskprocessor implementation is
that when the final reference to the taskprocessor goes away, it will
execute all tasks to completion instead of discarding the unexecuted
tasks.
For systems where unnamed semaphores are not supported, a really
simple semaphore implementation is provided. (Which gives identical
performance as the original taskprocessor implementation).
The way we ended up implementing Stasis caused the threadpool to be a
burden instead of a boost to performance. This was switched to just
use taskprocessors directly for subscriptions.
Review: https://reviewboard.asterisk.org/r/2881/
........
r400180 | dlee | 2013-09-30 13:39:34 -0500 (Mon, 30 Sep 2013) | 28 lines
Optimize how Stasis forwards are dispatched
This patch optimizes how forwards are dispatched in Stasis.
Originally, forwards were dispatched as subscriptions that are invoked
on the publishing thread. This did not account for the vast number of
forwards we would end up having in the system, and the amount of work it
would take to walk though the forward subscriptions.
This patch modifies Stasis so that rather than walking the tree of
forwards on every dispatch, when forwards and subscriptions are changed,
the subscriber list for every topic in the tree is changed.
This has a couple of benefits. First, this reduces the workload of
dispatching messages. It also reduces contention when dispatching to
different topics that happen to forward to the same aggregation topic
(as happens with all of the channel, bridge and endpoint topics).
Since forwards are no longer subscriptions, the bulk of this patch is
simply changing stasis_subscription objects to stasis_forward objects
(which, admittedly, I should have done in the first place.)
Since this required me to yet again put in a growing array, I finally
abstracted that out into a set of ast_vector macros in
asterisk/vector.h.
Review: https://reviewboard.asterisk.org/r/2883/
........
r400181 | dlee | 2013-09-30 13:48:57 -0500 (Mon, 30 Sep 2013) | 28 lines
Remove dispatch object allocation from Stasis publishing
While looking for areas for performance improvement, I realized that an
unused feature in Stasis was negatively impacting performance.
When a message is sent to a subscriber, a dispatch object is allocated
for the dispatch, containing the topic the message was published to, the
subscriber the message is being sent to, and the message itself.
The topic is actually unused by any subscriber in Asterisk today. And
the subscriber is associated with the taskprocessor the message is being
dispatched to.
First, this patch removes the unused topic parameter from Stasis
subscription callbacks.
Second, this patch introduces the concept of taskprocessor local data,
data that may be set on a taskprocessor and provided along with the data
pointer when a task is pushed using the ast_taskprocessor_push_local()
call. This allows the task to have both data specific to that
taskprocessor, in addition to data specific to that invocation.
With those two changes, the dispatch object can be removed completely,
and the message is simply refcounted and sent directly to the
taskprocessor.
Review: https://reviewboard.asterisk.org/r/2884/
........
Merged revisions 399887,400138,400178,400180-400181 from http://svn.asterisk.org/svn/asterisk/branches/12
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@400186 65c4cc65-6c06-0410-ace0-fbb531ad65f3
The pimp_my_sip branch is being merged at this point because
it offers basic functionality, and from an API standpoint, things
are complete.
SIP work is *not* feature-complete; however, with the completion
of the SUBSCRIBE/NOTIFY API, all APIs (except a PUBLISH API) have
been created, and thus it is possible for developers to attempt
to create new SIP work.
API documentation can be found in the doxygen in the code, but
usability documentation is still lacking.
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@386540 65c4cc65-6c06-0410-ace0-fbb531ad65f3
This patch adds the ability to create a serializer from a thread pool. A
serializer is a ast_taskprocessor with the same contract as a default
taskprocessor (tasks execute serially) except instead of executing out
of a dedicated thread, execution occurs in a thread from a
ast_threadpool. Think of it as a lightweight thread.
While it guarantees that each task will complete before executing the
next, there is no guarantee as to which thread from the pool individual
tasks will execute. This normally only matters if your code relys on
thread specific information, such as thread locals.
This patch also fixes a bug in how the 'was_empty' parameter is computed
for the push callback, and gets rid of the unused 'shutting_down' field.
Review: https://reviewboard.asterisk.org/r/2323/
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@381326 65c4cc65-6c06-0410-ace0-fbb531ad65f3
* Clarify some documentation
* Change copyright date of taskprocessor files
* Address potential issue of creating taskprocessor with listener if
taskprocessor with that name exists already
git-svn-id: https://origsvn.digium.com/svn/asterisk/team/mmichelson/threadpool@379124 65c4cc65-6c06-0410-ace0-fbb531ad65f3
Now user data is allocated by the creator of the taskprocessor
listener and that user data is passed into ast_taskprocessor_listener_alloc().
Similarly, freeing of the user data is left up to the user himself. He can
free the data when the taskprocessor shuts down, or he can choose to hold
onto it if it makes sense to do so.
This, unsurprisingly, makes threadpool allocation a LOT cleaner now.
git-svn-id: https://origsvn.digium.com/svn/asterisk/team/mmichelson/threadpool@379120 65c4cc65-6c06-0410-ace0-fbb531ad65f3
* Remove extraneous whitespace
* Bump up debug levels of messages and add identifying info to messages.
* Account for potential failures of ao2_link()
* Add additional test and some more test data
* Add some comments in places where they could be useful
* Make threadpool listeners and their callbacks optional
git-svn-id: https://origsvn.digium.com/svn/asterisk/team/mmichelson/threadpool@378652 65c4cc65-6c06-0410-ace0-fbb531ad65f3
Unfortunately, this required a taskprocessor listener change that makes listener allocation
utterly silly. I'm going to change the scheme so that allocation of taskprocessor listeners
is done internally within taskprocessor code. This will make it parallel with threadpool
code, which is a good thing.
git-svn-id: https://origsvn.digium.com/svn/asterisk/team/mmichelson/threadpool@377687 65c4cc65-6c06-0410-ace0-fbb531ad65f3
Mike Bradeen
Alexander Traud
Kevin Harwell
Ben Ford
Matthew Fredrickson
Richard Mudgett
George Joseph
Corey Farrell
Alexei Gradinari
Mark Michelson
Jonathan Rose
Joshua Colp
Matt Jordan
Kinsey Moore
David M. Lee