Enhancement to IMMND restart invisible to IMMA

[mahesh]

Please find the attached mails for reference .

It has take default it seems.

[anders]

Date: Thu, 16 Apr 2009 10:28:47 +0200
From: Anders Bjornerstedt <anders.bjornerstedt@…>
To: mahesh.alla@…, Mario Angelic <mario.angelic@…>,

Hans Feldt <hans.feldt@…>

Cc: devel@…
Subject: [devel] Analysis of "Enhancement to IMMND restart invisible to

IMMA" (Enhancement Ticket #496)

I have done some thinking about the "transparency for handles of IMMND
restarts" issue,
which is in essence equivalent to the BAD_HANDLE error case as seen by
the IMM client.
[The BAD_HANDLE error code can in principle be returned also for some
client errors,
but the bottom line for the client is that all such cases should be
handled by discarding the
current handle and (optionally) initializing a new handle. If there are
any cases currently

mapped by the IMM to BAD_HANDLE, that I discover can not be handled by
the client
in this way, then they will be remapped to ERR_LIBRARY.]

My short conclusion is that to completely eliminate the BAD_HANDE error
case,
for all kinds of IMM handles, is not realistic.

By not realistic I mean it is not worth the cost in implementation,
increased complexity,
and degraded performance for all normal processing, to *eliminate* the
case.

I would rather put time into making the IMMND less likely to crash in
the first place.
There is something inherently suspicious about adding complexity to an
implementation
to compensate for poor quality of that implementation.

The IMMND may in some cases be restarted for reasons outside its
control. But even for
such cases I am not convinced that it is a good idea to attempt
elaborated local recovery.
For one thing, it is typically not possible for the restarted IMMND to
know why it was
restarted. For another, compensating recovery behavior by the IMMND may
increase the
risk of producing cyclic restarts, escalating the problem to a node
restart.

The cost of removing this error case must also be weighed against the
degree of difficulty
caused for the IMM client developer, by having to deal with the
BAD_HANDLE case,
or the consequences of the client application escalating, if they decide
not to handle it (i.e. to terminate).

On the other hand, reducing the frequency of the BAD_HANDLE case is
realistic, particularly
for some types of IMM handles.

By reducing frequency I dont just mean reducing the frequency of IMMND
restarts. That should
always be the main priority and a low frequency event anyway. What I
mean is rather that there
are some relatively simple techniques for the IMMA library to detect how
"heavily loaded" a
particular handle happened to be when the restart of the IMMND occurred.
Handles that happen to be "idle", or lightly loaded, may be trivial to
re-initialize transparently
from the IMMA library code. Essentially then, the library would
internally perform the same
re-initialization of the handle that currently the client is expected to
do. The IMMND server side
would not see any difference.

Degree of difficulty for client developer
================================

The degree of difficulty for the client depends on the type of handle
and the context it is used.
As I see it, the handle transparency problem is only a "hard" problem
for imm clients that are in
a dispatch loop. In practice we are talking about the IMM-OI dispatch
loop. Exactly the problem
encounterded by the LOG-OI (ticket #466).

The IMM-OM interface also has one dispatch case, for handling replies to
asyncronous admin-op
invocations. But the OM-client that is worried about loss of connection
between invocation and reply,
should use the new continuation API added in A.02.01 (That API is
actually not yet implemented,
but it will be as soon as we decide we need it).

All other imm-client use cases are of the type client request followed
by syncronous reply to the
client. These cases have to be equiped anyway to handle the failure of
that request for a variety of
error codes. For a one-shot request, the consequences for the client
developer of also dealing with
the BAD_HANDLE case should not add significantly to the burden.

For requests tied to a larger context (CCBs and iterators) it may not be
so obvious.

Any client using CCBs has to deal with the entire CCB failing for
reasons outside their control anyway.
The BAD_HANDLE error case is just one case among several (which should
already be there).
BAD_HANDLE means they need to reinitialize their handles. Here it will
be much easier
(than in the dispatch case) since the context is unwinded anyway. No
need for a separate thread etc.
I am against coding for recovery of CCBs that are not in a critical
state. On of the big benefits
of CCBs/Transactions is that they are designed to allow abort at any
point before the apply/commit
has been accepted by the system.

For iterations, non-transparency has a somewhat bigger impact.
Iteration will have to restart from the beginning. The application can
not just refresh the
iterator handle and expect to be able to invoke a 'next-call' to
continue iterating.
Iterations will in this sence becaome more similar to CCBs in that it is
the entire iteration
which ahs to be restarted.
But if there is any case that is the least realistic to provide
transparency for, it is iterations.
Iterations are currently a read only operation and handled node local
IMMA<->IMMND.
That is a great advantage from a performance and accessibility point of
view. Iterations
are allowed when the IMM service is write locked for sync. To make
iterations transparent
for IMMND restart would impact both performance and make them "mutating"
operations
in the sense that the iteration progress has to be remembered
redundantly, i.e. making
progress in iteration is now in essence a non-local and state rich
operation. There are other
solutions: shared memory, checkpoint etc, but since the IMMND crashed,
can I really
trust what it wrote to that shared memory/checkpoint anywway?

Consequences of client escalating.
=========================
The client developer can decide that the degree of difficulty in
handling the BAD_HANDLE case
is not worth it and simply terminate, i.e. escalate and forward the
non-transparency to the next level.
If the process that is doing this is restartable, i.e. does not cause
node restart, then the consequence
should not be too serious. Again, I belive the serious problem is object
implementers for
service-server type of implementations (non restartable).

Transparency at one interface may cause problems at another.
=============================================
One thing that must be remebered about the IMM service is that it has
two interfaces. Providing
transparent IMMND restart towards OI-implementers at that node, can
translate to a different
kind of non-transparent problem on the OM side. By insisting that the
OI/IMMND connection must
survive an IMMND restart, we are forcing OM clients doing anything that
needs to reach the
implementer to wait (delayed reply or by replying with TRY_AGAIN). In
the current situation, when
the OI connection is severed, the OM client (at another node!) will get
NOT_EXIST immediately.
This until the implementer reconnects somewhere.
This is the same reply they will get if the OI had crashes and has not
yet recovered,
i.e. this case they will need to deal with even if IMMND restart is made
transparent.
The delay in reply for the OM side may or may not be a problem depending on
how long the delay is. The duration will depend on the time it takes to
sync the restarted IMMND.
If the sync time is long (more than a few seconds) the transparent OI
handles causes a new problem
on the OM side: indefinite wait with TRY_AGAIN. The application may here
prefer to restart their
OI at another node immediately, which is possible in the non-transparent
OI case, but not possible
(blocked) by the fancy transparent OI case.

The conclusion here is that *if* I am to attempt to provide OI handles
that are transparent to IMMND
restarts, then that transparency has to have a TIMEOUT anyway (!) and if
that timeout expires we get
back anyway to the non transparent case which is again exposed
(non-transparent) to the OI client.

Another way of formulating this is that long sync-times for restarted
IMMNDs will be painfull for
IMM clients. With non-transparent handles, that pain at least stays
local to the node where the IMMND
restarted. With handle transparency, the pain (ERR_TRY_AGAIN, or
ERR_TIMEOUT) can get
propagated to other nodes in the cluster.

Conclusion
=========
Complete elimination of the BAD_HANDLE case from restarted IMMNDs is
neither practical nor desirable.

However, trying to recover handles that are not state-heavy should be
done and will in many cases succeed.
This is where the situation is benign. The handle was not associated
with any significant state, or that
state is simple to recover or re-create by the reconnecting library.
For example: the library can attempt to re-connect an idle implementer
handle and re-claim an
implementer role, without making this vissible to the IMMA OI client.
If the implementorship is not occupied then this will succeed. If it is
occupied (presumably by some
other process in the cluster that detected the non-existence of the
implementer) then the attempt will
fail and result in the BAD_HANDLE case towards the IMMA OI client.

Recovery of the handles that are state-heavy will not be attempted. For
example, if the implementer
handle is accoiated with one or more on-going ccbs, that is it has
received ccb related upcalls but not
the terminating ccb-apply or ccb-abort upcall, then BAD_HANDLE will be
generated.
In theory one could generate abort upcalls for the ccb(s) instead, but
we prefer to let the OI-client to get the general
picture BAD_HANDLE, instead of late abort upcalls.

The main consequence of this is that at least IMM clients that are
non-restartable and that are executing on a controller
node should be designed to handle the BAD_HANDLE case without restarting.

_
devel mailing list
devel@…
http://list.opensaf.org/maillist/listinfo/devel

[anders]

Only part 1.
http://list.opensaf.org/archives/devel/2009-September/005126.html

[anders]

Part 1, resurrection for OI handles checked in to opensaf-staging.
Remaining parts will not be included in OpenSAF Release 4 Beta2.

changeset: 664:95d12c38f5b1
tag: tip
user: Anders Bjornerstedt <anders.bjornerstedt@…>
date: Thu Sep 17 12:16:20 2009 +0200
summary: immsv:IMMND restart less visible to IMMA part1 (#496)

[anders]

Part 2.
http://list.opensaf.org/archives/devel/2009-October/005362.html

A third part will deal with recovery of ccb result
This is for OM saImmOmCcbApply and for OI generating ccbApply/ccbAbort upcalls
after restarted IMMND.

[anders]

Part2, resurrection of OM handles, aborting of non critical CCBs.

changeset: 689:76b543ee75d0
user: Anders Bjornerstedt <anders.bjornerstedt@…>
date: Fri Oct 16 14:42:04 2009 +0200
summary: immsv:IMMND restart less visible to IMMA part2 (#496)

[anders]

http://list.opensaf.org/archives/devel/2009-November/005582.html

[anders]

Final Part 3
opensaf-staging:
changeset: 698:33c73ef6e84c
user: Anders Bjornerstedt <anders.bjornerstedt@…>
date: Wed Nov 11 16:41:30 2009 +0100
summary: immsv: Recovery of CCB outcome for OI (#496 part 3)

changeset: 697:73fc1cf85ddc
user: Anders Bjornerstedt <anders.bjornerstedt@…>
date: Fri Nov 06 16:23:19 2009 +0100
summary: immsv: Recovery of CCB outcome for saImmOmCcbApply (#496 part 3)

changeset: 696:bd976e985fe0
user: Anders Bjornerstedt <anders.bjornerstedt@…>
date: Thu Nov 05 10:31:57 2009 +0100
summary: immsv: Retain CCB outcome for recovery (#496 part 3)