Planning and execution of plans (termed orchestrator) are performed by a single mobile robot. The plans involve high level plans like where the robot should go to, opening doors, picking up certain objects and so on. It does not involve minute details such as how the robot should open doors through the coordination of servos and visual input. In this work, when a robot is expected to execute a plan, the robot is already in an environment that is conducive or possible to execute such plans. The planner and the orchestrator will execute a sequence of plans to prepare such an environment. For example, when the robot needs to switch on a light, the planner will first tell the robot to go near the corresponding switch before switching it on with its manipulators.
For this work, a few basic movements of the robot are assumed, which are, mobility, tuning up/down, opening and closing, switching on and off, picking and putting objects. It is assumed that the robot can hold something while performing open/close and tuning up/down actions. Planning and execution will revolve around these basic movements to achieve certain goals. Planner module is responsible for the planning and passing commands to the robot for its execution. Given a certain goal, planner module will compose a sequence of plans for the robot to execute such that after the plans are completed, the goal is achieved. For example, given that the goal is to place a canned soft drink in the fridge and that all cabinet needs to be closed, the robot will first find where the soft drink is. If there is one that happens to be in the kitchen cabinet, the robot will approach it, open the cabinet and pick up the soft drink, and then close it. It will subsequently approach and open the fridge, after which it will place the canned drink before closing it. No prior programming is required for the plan.
This work emphasizes on generating plans, thus, time to achieve this is assumed to be acceptable. It is also assumed that the syntactic statement used by the planner for the planning problem describes the actual environment. Therefore, failure or success of robot in achieving its task is reflected in the environment of planning problem.
Planner module
Figure 1 shows the flow chart of the planning and execution process. Three important components of the planner module are the variables, services and goals. Variables record the current state of robot and the environment. Services are activities that the robot can perform, which have preconditions and effects. Precondition contain a list of conditions that need to be fulfilled by the variables before the plan can be executed, whereas effect is the changes that will occur after the service is implemented. The goal is the final condition that needs to be met by the variables. Planner module consists of a planner and an orchestrator. Planner will compose a sequence of plans according to current variable state. If a solution is found by the planner, the plan will be passed on to the orchestrator to execute the plan. Orchestrator’s role is to pass commands to the robot to execute the plans accordingly.
Given the available variables, services and goals, the planner will compose a sequence of plans such that, after implementation, the goal will be fulfilled. The sequence of plans can be likened to theorem proving [22]. Every subsequent plans will have their preconditions met, and will impose changes for the next plan to drive the variables to what is required by the goal.
The following is an explanation of the flow chart in Fig. 1. When the program starts, the state will be at state -1 (note that the use of the word ’State’ here is only for explanation in Fig. 1. State -1 only occurs once when the program starts. During this state, start-up booting will occur as well as extracting crucial information from the knowledge base. The first goal is also obtained at this stage. Planning will occur to devise a sequence of plans to fulfill the goal. If a solution is found, the program will proceed to state 1. Otherwise, it will go to state 4, signifying a failure in finding solution. The purpose of state 1 is to obtain the next service in the plan for execution, after which will be executed at state 2. State 2 implements as the orchestrator. It first checks whether preconditions of the current activity are met or not. If not, the program will proceed to state 0, or else, the service will be executed and returns to state 1 to pick up the next service. State 0 resembles state -1, just that it doesn’t require start up phase. The transition from state 2 to state 0 enables dynamic planning. This is important to deal with uncertain situation, where variables cannot be confirmed except during run-time. In this case, re-planning can occur at state 0 if expectation is wrong. After plan execution, state 3 will re-check whether the goal is fulfilled, upon which is yes, it will proceed to state 5. If the goal cannot be achieved, the program will proceed to state 4. The final state is state 6 which records a log for future reference or learning. The program will restart at state 0 after selecting the next goal.
General domain description
Domain description will be based on the work of [15]. We denote \(\vartheta \) = term set (list of variables). \(\vartheta \) contains V terms, which consists of knowledge, effect, dynamic response and static response terms confined by their own domain. Term can be a variable, constant or function. Response terms represent information that can only be obtained from objects, information that comes from sources not within \(\vartheta \). During planning stage, static response terms remain the same throughout all planning sequence and represent an unknown value (thus, initialization constraint is imposed on them), whereas dynamic response can change for every sequence. They can take on whatever value to facilitate constraint satisfaction during planning that employs an optimistic approach (value taken on by response terms are considered true). Dynamic response terms can also be determined or having their domain constrained by current, past or future variables via the use of rules (In this work, only past and current variables are considered). Response term usage will be made more evident in subsequent sections. Knowledge terms record information for future reference. Effect terms are used for external control.
A state is a tuple of values to terms at a particular plan implementation sequence index t that is denoted as \(X_t=(X_t^1,X_t^2...X_t^V)\) where \(X_t^1,X_t^2...X_t^V\in {\vartheta }_t\), confined by their domains denoted by \(D^1,D^2...D^V\). As there is a finite limit to the number of sequence per plan being planned denoted as K, thus, \(0\le t<K\). For the current work, domains of the variables of the terms remain unchanged over time.
\(\alpha \) is the set of activities, where \(a=(id(a),\)
\(precond(a),effect(a))\in \alpha \). id(a) is the identifier of the activity. There is an additional activity in \(\alpha \) that does nothing. It has no pre-conditions and effects, termed as Nop.
precond(a) is the pre-condition that need to be met before the activity can be executed, such as the robot needs to near the cabinet before attempting to open it. Precondition of an activity can be described as follows:
$$\begin{aligned} precond(a)::=prop|precond(a)\wedge precond(a)|precond(a) \nonumber \\\vee precond(a)| \lnot precond(a)|precond(a) \nonumber \\ \rightarrow precond(a) \end{aligned}$$
(1)
$$\begin{aligned} prop::=var\bullet var|var\bullet val|(var\odot var)\\\bullet var| (var\odot var)\bullet val|nPred \end{aligned}$$
(2)
where \(var\in \vartheta \), val is a constant, \(\odot \in \{+,-\}\) is a binary operator, \(\bullet \in \{=,<,>,\ne ,\le ,\ge \}\) is a relational operator, and nPred is an n-ary predicate.
effect(a) is the changes that will be induced after the activity is completed. It emulates the state transition given the activity a, such that its logical formulation can be used to impose constraints on subsequent sequence of the plan for activity planning. It should be emphasized that the actual object manipulates variables during run-time after planning instead of the effect(a) formulation (which is only used for planning). Effect of an activity can be formulated as or a combination of the following: \(var_{t+1}=val\), \(var_{t+1}=var_t\), \(var_{t+1}=f(v_1,v_2)\) where \(v_1,v_2\in {\vartheta }_t\) or \(v_1,v_2\) are constants, and f is the sum, subtraction and Boolean operation.
For simplification, when necessary, we will denote the above relations as \(var_{t+1}=effects_t(a)\). This relation is read differently between the planner and the orchestrator. During planning, \(var_{t+1}=effects_t(a)\) means the truth statement that: (\(effects_t(a)\) includes an effect towards the variable var) implies \((var_{t+1}=effects_t(a))\) holds true.
On the contrary, for orchestrator, it is seen as \(var_{t+1}:=effects_t(a)\), which indicates that the term var is being modified according to \(effects_t(a)\). Therefore, depending on whether planner or orchestrator is referred to, the correct interpretation has to be made. There is no arrow of time for planner, thus, the relation is seen as equality. For orchestrator, it is seen as an assignment.
Though this work only use the specified effects, more sophisticated effects, such as conditional effects, can be used as shown in [15]. That said, the extension to previous work [15] is shown in Fig. 2, indicated by the red bounding box. The extension is the use of dynamic response terms and the rules for their transition. This extension enables inference capabilities and the ability to make choices to be realized as part of planning process. As dynamic response terms and their corresponding rules are dependent on the activity the robot is performing, detailed explanation will be given in “Design of services” section.
Planning as constraint satisfaction problem
Given goals, which are represented as propositions, activity planning can be obtained to fulfill the goal by representing the problem as CSP and solve it. A CSP is a triple \(CSP=\langle \chi ,D,\zeta \rangle \), where \(\chi \) is a set of terms, D is the set of domains of the variables of the terms in \(\chi \), and \(\zeta \) is a set of constraints over \(\chi \). A solution to a CSP is an assignment of values to the terms in \(\chi \) such that the values fall within D and all constraints in \(\zeta \) are satisfied. In this work, D is unchanged throughout the activity sequence. It is considered determined when a goal is passed to the planning process flow in Fig. 1, and will stay that way until the goal is achieved.
In terms of the ISS planner, \(\chi =\{X_1,X_2...X_K\}\cup \{A_1,A_2...A_{K-1}\}\cup R\), R is a set of response terms, and \(A_t\) is the chosen goal at sequence index t. Unlike X and A, variables in R remain the same throughout the planning sequence.
\(\zeta \) consists of constraints imposed by a chosen activity at t from activity pre-conditions and effects, inertia law, initial and final variable state and maintenance of achieved goal constraints. Initial variable state is just a constraint that dictates the values of all variables (obtained from object state module) before any planning. Final state constraint consists of the goal proposition that needs to hold at sequence index K.
Constraints from activity pre-conditions:
\((A_t=a)\rightarrow precond(a)\) where \(\forall a\in \alpha \)
Constraints from activity effects:
\((A_t=a)\rightarrow [(var_{t+1}=effects_t(a))\wedge Fr]\) where \(\forall a\in \alpha \)
where Fr is the inertia law constraint, which indicates that for every other variables var (excluding those from R) not affected by \(effects_t(a)\), \(var_{t+1}=var_t\).
Maintenance of achieved goal constraint:
This constraint dictates that whenever a goal is achieved at sequence index \(\bar{t}<K\):
\(A_t=Nop\) where \(\bar{t}<t<(K-1)\)
Maintenance of achieved goal constraint is just one of the goals specified in [15], though it is sufficient for the current work.
Constraints are fed to a solver to obtain a sequence of A, which are the activities that need to be implemented to fulfill the given goals. Z3, which is a state of the art SMT solver, is used to obtain the plan [23]. The plan will be solved by continually increasing K until the constraints are satisfied.
Design of terms
The robot in this work is intended to fetch and place objects, as well as opening/closing and switching on/off switches. It needs to be able to move around to enable it to perform the tasks. There are three types of objects, which are, movable objects, non-movable objects and location.
Movable objects consist of objects that can be moved around by the robot like towel and cans. Non-movable objects are objects that are fixed, but they may be able to be operated by the robot. Examples of non-movable objects are bed, doors, cabinet and switches. The robot itself is also considered a non-movable object for convenience during planning which will be shown later on. Human is also considered a non-movable object as the assumption is that the human remain stationary during planning. If the human moves, due to the ability to re-plan as discussed previously, the issue can be easily solved. Location object consists of regions on the home such as the living room, bedroom and kitchen.
Apart from integer and boolean datatype, five new data types are introduced to support the mentioned objects, namely, NOType, MOType, Location, NOStateType, MOStateType, NOID and MOID. NOType defines the type (ex: cabinet, door, switch) of non-movable object with ID defined in data type NOID. Likewise, MOType defines the type (towel, paper, cup) of movable object with ID defined in data type MOID. Location is the datatype of location.MOStateType and NOStateType are the datatype that defines the state of a movable and non-movable object respectively.
There are two static functions (functions where the output values remain the same throughout all planning sequence), which are, \(FNOType:NOID\rightarrow NOType\) and \(FMOType:MOID\rightarrow MOType\). A constant is HumanLocation, and a static predicate is NOLocation(NOID, Location).
FNOType maps a particular non-movable object to its type (For example, mapping an object as a door). Like wise, the same applies to FMOType, but that it applies to movable object. The predicate NOLocation(a, b) states the truth value whether a non-movable object a is in location b. This is especially important for objects like doors. HumanLocation stores where the human is at in datatype Location
The subsequent terms explained in this subsection can have their values changed in the course of planning sequence. This means, given a term A, for a plan with K sequence, there will be \(A\times K\) number of variable A, each for every sequence, where each has a unique definition \(A_1, A_2 ... A_K\).
The function \(DYStateNO_t{:}\,NOID\rightarrow NOStateType\) outputs the state of the non-movable object. \(DYStateNO_t(a)=b\) states that a is in a state of b at sequence t. The same applies to \(DYStateNO_t{:}MOID\rightarrow MOStateType\), which is for movable objects.
The function \(DYTune_t{:}\,NOID\rightarrow Int\) outputs the numerical value (in integer Int type) associated with the non-movable object at sequence t.
As movable objects will always be placed at a non-movable object (ex: cup on a table, book with a human), the function \(DYAtMO_t{:}\,MOID\rightarrow NOID\) maps a movable object to a non-movable object it is placed at.
Three variables are included, namely, \(RobotLocation_t\), \(RobotApproach_t\) and \(RobotHold_t\). \(RobotLocation_t\) stores the location of the robot (with datatype Location) at sequence t. \(RobotApproach_t\) records the non-movable object the robot is approaching. \(RobotHold_t\) is a boolean variable determining whether the robot is holding something or not.
For proper functionality, three dynamic response variables are introduced, which are, \(Approachres_t\), \(MOIDres_t\) and \(RobotHoldres_t\). The role of these dynamic response variables will be made more evident during the discussion of the services.
Design of services
This subsection describes, but not limited to, three types of services the robot is expected to perform. Although only three types are listed, more services can be included depending on the application. The four types are open/closing service, tuning up/down service, mobility, and put/pick service.
Open/closing service consists of two services, which are opening and closing. These two services applies to opening/closing of doors and switching on/off switches.
It has a precondition:
\(NOLocation(RobotApproach_t,RobotLocation_t)=true\)
and effect:
\(DYStateNO_{t+1}(RobotApproach_t){:}=Open/Close \, \mathrm{respectively}\)
The precondition has to make sure the robot is approaching object stored in \(RobotApproach_t\) and that the robot is in location RobotLocation, before it can open/close the non-movable object by manipulating DYStateNO at sequence \(t+1\).
Tuning up/down service deals with numerical manipulation that consists of (1) tuning up (increasing by 1), and (2) tuning down (decreasing by 1).
It has a precondition:
\(NOLocation(RobotApproach_t,RobotLocation_t)=true\)
and effect:
\(DYTune_{t+1}(RobotApproach_t){:}=DYTune_{t}(RobotApproach_t) {+/-} 1\) respectively
Put/Pick service consists of robot fetching and placing a movable object.
The preconditions for picking up objects are:
\(DYStateNO_t(RobotApproach_t)=Open\)
\(DYAtMO_t(MOIDres_t)=RobotApproach_t\)
\(NOLocation(RobotApproach_t,RobotLocation_t)=true\)
\(RobotHold_t=false\)
and effects:
\(DYAtMO_{t+1}(MOIDres_t){:}=NRobot\)
\(RobotHold_{t+1}{:}=true\)
The preconditions make sure that the movable object the robot is trying to fetch is located in a non-movable object with state Open through the predicate \(DYStateNO_t(RobotApproach_t)\),and that the robot is not holding anything via \(RobotHold_t=false\). They also make sure the object the robot is trying to fetch (\(MOIDres_t\)) is located in non-movable object \(RobotApproach_t\). \(MOIDres_t\) is a dynamic response variable, where its value is freely determined by the planner to aid optimistic planning which is inherent in planning via solving CSP. If the preconditions are met, the robot can pick object \(MOIDres_t\) through setting \(DYAtMO_{t+1}(MOIDres_t)\) to NRobot, where NRobot is the non-movable object ID under datatype NOID for the robot. RobotHold is set to true to indicate that the robot is holding something.
The preconditions for putting objects are:
\(DYStateNO_t(RobotApproach_t)=Open\)
\(DYAtMO_t(RobotHoldres_t)=NRobot\)
\(NOLocation(RobotApproach_t,RobotLocation_t)=true\)
\(RobotHold_t=true\)
and effects:
\(DYAtMO_{t+1}(RobotHoldres_t){:}=RobotApproach_t\)
\(RobotHold_{t+1}{:}=false\)
For putting objects, the preconditions also specify that the intended non-movable object be opened. It requires the robot to hold movable object \(RobotHoldres_t\), which is indicated by having NRobot as the output for function \(DYAtMO_t\). Just like \(MOIDres_t\), \(RobotHoldres_t\) is a dynamic response variable that stores the current object the robot is holding. \(RobotHold_t=true\) is the constraint where the robot is holding something. With the preconditions met, \(RobotHoldres_t\) will be placed at \(RobotApproach_t\) through \(DYAtMO_{t+1}\).
Mobility service consists of two services, that is, movement within a room, and movement between rooms.
Movement within a room has the following precondition:
\(NOLocation(Approachres_t,RobotLocation_t)=true\)
and effect:
\(RobotApproach_{t+1}{:}=Approachres_t\)
The robot will always be going towards a non-movable object, as it is practically meaningless to go towards nothing. Therefore, the precondition makes sure that a non-movable object the robot is going to is within the current room, where the dynamic response variable \(Approachres_t\) stores the object the robot is approaching.
Movement between rooms is required for every doors. One can write a precondition as the following:
$$\begin{aligned}&\exists z(NOLocation(z,RobotLocation_t)\wedge NOLocation(z,res_t)\wedge (res_t\ne RobotLocation_t)\wedge \\&DYStateNO_t(RobotApproach_t)\wedge (RobotApproach_t=z)\wedge (FNOType(z)=Door) ) \end{aligned}$$
This precondition with existential quantifier takes a longer time to plan from preliminary test. Another alternative is to build a service for every doors. The precondition is shown as follows:
\(DYStateNO_t(V1)=true\)
\(RobotLocation_t=V2\)
and effects:
\(RobotLocation_{t+1}{:}=V2\)
Services with above mentioned preconditions and effects are built for every door and for every side. It means that if there are two doors altogether, there are a total of four such services, where each door takes up two for the robot to move through the door in both directions. From the preconditions and effects, V1 is the door object ID and V2 is the location the robot goes to after passing the door.