MODULE
DYNAMIC PROGRAMMING
LEARNING
OBJECTIVES
After completing this module, module, students will be able to: 1. Understand Understand the overall approach approach of dynamic programming. 2. Use dynamic programming to solve the shortest-route problem. 3. Develop dynamic programming stages. 4. Describe important dynamic programming terminology. 5. Describe the use of of dynamic programming programming in solving knapsack problems.
MODULE
.
OUTLINE
M2.1
Introduction
M2.2
Shortest-Route Problem Solved by Dynamic Programming
M2.3
Dynamic Programming Terminology
M2.4
Dynamic Programming Notation
M2.5
Knapsack Problem
Summary • Glossary • Key Equations • Solved Problem • Self-Test • Discussion Questions Questions and Problems • Case Study: Unit United ed Trucking Trucking • Internet Case Study • Bibliography
2
M2-2 M2.1
MODULE MODU LE 2 Dynam Dynamic ic Programm Programming ing
INTRODUCTION
Dynamic programming breaks a difficult problem into subproblems.
Dynamic programming is programming is a quantitative analysis technique that has been applied to large, complex problems problems that have have sequences of decisions to be be made. Dynamic programming programming divides problems into a number of decision of decision stages ; the outcom outcomee of a decision decision at one stage stage affects the decision decision at each of the next stages. The technique is is useful in a large number of multiperiod business problems, such as smoothing smoothing production employment, allocating capital funds, allocating salespeople to marketing areas, and evaluating investment opportunities. Dynamic programming differs differs from linear programming programming in two ways. ways. First, there is no algorithm (like the simplex method) that can be programmed to solve all problems. Dynamic programming is, instead, a technique that allows us to break up difficult problems into a sequence sequence of easier subproble subproblems, ms, which are then evaluated evaluated by stages. Seco Second, nd, linear programming is a method that gives single-stage (one time period) solutions. Dynamic single-stage (one programming has the power to determine the optimal solution over a one-year time horizon by breaking the problem into 12 smaller one-month horizon problems and to solve each of of these optim optimally ally.. Henc Hence, e, it uses uses a multistage approach. multistage approach. Solving problems with dynamic programming involves four steps:
Four Steps of Dynamic Programming 1. Divide the original problem into subproblems called stages. 2. Solve the last stage of the problem for all possible conditions conditions or states. 3. Working backward from the last stage, solve each intermediate intermediate stage. This is done by determining optimal policies policies from that stage to the end of the problem (last stage). 4. Obtain the optimal solution for the original problem by solving all stages sequentially. In this module we show how how to solve two types of dynamic programming problems: network and nonnetwork. The shortest-route problem is a network problem that can be solved solv ed by dynamic programming programming.. The knapsack knapsack problem is an example of a nonnetwork nonnetwork problem that can be solved using dynamic programming. M2.2
SHORTEST-ROUTE PROBLEM SOLVED BY DYNAMIC PROGRAMMING George Yates is about about to make make a trip from Rice, Rice, Georgia (1), to Dixieville, Dixieville, Georgia (7). George would would like like to find the shortest shortest route. route. Unf Unfortunat ortunately ely,, there are are a number of small towns between Rice and Dixieville. His road map is shown in Figure M2.1. M2.1. The circles circles on the map, map, calle called d nodes , repr represent esent citi cities es such such as Rice Rice,, Dixie Dixieville, ville, Brown, and so on. on. The arrows arrows,, cal called led arcs , repr represent esent highways highways between between the cities. cities. The distance distance in miles is is indicated indicated along along each arc. arc. This problem problem can, can, of cour course, se, be solved solved by inspecti inspection. on. But seeing how dynamic programming can be used on this simple problem will teach you how to solve larger and more complex problems.
The first step is to divide the problem into subproblems subproblems or stages.
subproblems or stages. Figure M2.2 Step 1: The 1: The first step is to divide the problem into subproblems reveals reve als the stages stages of this problem. problem. In dynamic dynamic programming, programming, we usually usually start with the last part of the problem problem,, stage 1, 1, and work work backward backward to the the beginning beginning of the problem problem or or network, which is stage 3 in this problem. problem. Table M2.1 summar summarizes izes the arcs and arc distances distances for each stage.
The second step is to solve the last stage—stage 1.
Step 2: W 2: We next solve solve stage stage 1, the last part part of the network. network. Usu Usually ally,, this is trivial. trivial. We find the shortest shorte st path to the end end of the network, network, node 7 in this this problem. problem. At stage stage 1, the shortest shortest paths, paths,
M2.2: Shortest-Route Problem Solved by Dynamic Programming FIGURE
M2.1
Highway Map between Rice and Dixieville Lake City
Athens 10 Miles
4
s l e i M 4
Brown 5 Miles
1 4 M i l e s
s i l e M 1 2
s i l e 4 M
1
5
3
7 Dixieville
Rice
2 M i l e s
6 M i l e s 10 Miles
2
6
Hope
FIGURE
s l e i M 2
Georgetown
M2.2
Three Stages for the George Yates Problem
A Node
5
4
1
10
4
A Branch
14
12 3 6
5 2
7 2
4
10
6
2 Stage 3
TABLE M2.1
Distance Along Each Arc
Stage 2
Stage 1
STAGE
ARC
ARC DISTANCE
1
5–7
14
6–7
2
4–5
10
3–5
12
3–6
6
2–5
4
2–6
10
1–4
4
1–3
5
1–2
2
2
3
M2-3
M2-4
MODULE 2 Dynamic Programming
FIGURE
M2.3
Solution for the One-Stage Problem
10
4 4 1
Minimum Distance to Node 7 from Node 5
14 5 14
12 3 6
5 2
7 2
4
10
6
Minimum Distance to Node 7 from Node 6
2 2
from node 5 to node 6, to node 7 are the only paths. Also note in Figure M2.3 that the minimum distances are enclosed in boxes by the entering nodes to stage 1, node 5 and node 6. The objective is to find the shortest distance to node 7. The following table summarizes this procedure for stage 1. As mentioned previously, the shortest distance is the only distance at stage 1. STAGE 1 BEGINNING NODE
Step 3 involves moving backward solving intermediate stages. FIGURE
SHORTEST DISTANCE TO NODE 7
ARCS ALONG THIS PATH
5
14
5–7
6
2
6–7
Step 3: Moving backward, we now solve for stages 2 and 3. At stage 2 we will use Figure M2.4.
M2.4
Solution for the Two-Stage Problem
Minimum Distance to Node 7 from Node 4
24
4 10 Minimum Distance to Node 7 from Node 3
14
8 12
4
1
5 2
3
5
7
6 2 4
10
6
2 2 Minimum Distance to Node 7 from Node 2
14
12
M2.2: Shortest-Route Problem Solved by Dynamic Programming
M2-5
If we are at node 4, the shortest and only route to node 7 is arcs 4–5 and 5–7. At node 3, the shortest route is arcs 3–6 and 6–7 with a total minimum distance of 8 miles. If we are at node 2, the shortest route is arcs 2–6 and 6–7 with a minimum total distance of 12 miles. This information is summarized in the stage 2 table: STAGE 2 BEGINNING NODE
SHORTEST DISTANCE TO NODE 7
ARCS ALONG THIS PATH
24
4–5
4
5–7 3
8
3–6 6–7
2
12
2–6 6–7
The solution to stage 3 can be completed using the accompanying table and the network in Figure M2.5. STAGE 3 BEGINNING NODE
SHORTEST DISTANCE TO NODE 7
ARCS ALONG THIS PATH
13
1–3
1
3–6 6–7
FIGURE
M2.5
Solution for the ThreeStage Problem
24 Minimum Distance to Node 7 from Node 1
10
4
14 8
13 4 1
5 14 5 2
12 3 6
7 2
4
10
6
2 2 12
M2-6
MODULE 2 Dynamic Programming
The fourth and final step is to find the optimal solution after all stages have been solved.
M2.3
Step 4: To obtain the optimal solution at any stage, all we consider are the arcs to the next stage and the optimal solution at the next stage. For stage 3, we only have to consider the three arcs to stage 2 (1–2, 1–3, and 1–4) and the optimal policies at stage 2, given in a previous table. This is how we arrived at the preceding solution. When the procedure is understood, we can perform all the calculations on one network. You may want to study the relationship between the networks and the tables because more complex problems are usually solved by using tables only.
DYNAMIC PROGRAMMING TERMINOLOGY Regardless of the type or size of a dynamic programming problem, there are some important terms and concepts that are inherent in every problem. Some of the more important follow: 1. Stage: a period or a logical subproblem. 2. State variables: possible beginning situations or conditions of a stage. These have also been called the input variables. 3. Decision variables: alternatives or possible decisions that exist at each stage. 4. Decision criterion: a statement concerning the objective of the problem. 5. Optimal policy: a set of decision rules, developed as a result of the decision criteria, that gives optimal decisions for any en tering condition at any stage. 6. Transformation: normally, an algebraic statement that reveals the relationship between stages. In the shortest-route problem, the following transformation can be given: distance from the beginning of a given stage to the last node
=
distance from the beginning of the previous stage to the last node
+
distance from the given stage to the previous stage
This relationship shows how we are able to go from one stage to the next in solving for the optimal solution to the shortest-route problem. In more complex problems, we can use symbols to show the relationship between stages. State variables, decision variables, the decision criterion, and the optimal policy can be determined for any stage of a dynamic programming problem. This is done here for stage 2 of the George Yates shortest-route problem.
IN ACTION
Dynamic Programming in Nursery Production Decisions
Managing a nursery that produces ornamental plants is difficult. In most cases, ornamental plants increase in value with increased growth. This value-added growth makes it difficult to determine when to harvest the plants and place them on the market. When plants are marketed earlier, revenues are generated earlier and the costs associated with plant growth are minimized. On the other hand, delaying the harvesting of the ornamental plants usually results in higher prices. But are the additional months of growth and costs worth the delay? In this case, dynamic programming was used to determine the optimal growth stages for ornamental plants. Each stage was associated with a possible growth level. The state variables included acres of production of ornamental plants and carryover plants from previous growing seasons. The objective of the
dynamic programming problem was to maximize the after-tax cash flow. The taxes included self-employment, federal income, earned income credit, and state income taxes. The solution was to produce one- and three-gallon containers of ornamental plants. The one-gallon containers are sold in the fall and carried over for spring sales. Any one-gallon containers not sold in the spring are combined into three-gallon container products for sale during the next season. Using dynamic programming helps to determine when to harvest to increase after-tax cash flow. Source: Stokes, Jeffrey et al., “Optimal Marketing of Nursery Crops from Container-Based Production Systems,” American Journal of Agricultural Economics (February 1997): 235–246.
M2.3: Dynamic Programming Terminology
M2-7
1. State variables for stage 2 are the entering nodes, which are (a) (b) (c)
node 2 node 3 node 4
2. Decision variables for stage 2 are the following arcs or routes: (a) (b) (c) (d) (e)
4–5 3–5 3–6 2–5 2–6
3. The decision criterion is the minimization of the total distances traveled. 4. The optimal policy for any beginning condition is shown in Figure M2.6 and the following table:
FIGURE
GIVEN THIS ENTERING CONDITION
THIS ARC WILL MINIMIZE TOTAL DISTANCE TO NODE 7
2
2–6
3
3–6
4
4–5
M2.6
State variables are the entering nodes.
Stage 2 from the ShortestRoute Problem
24
10
4
14 8 5 1
12 3 6 4
Decision variables are all the arcs.
7 10
6
2 2 The optimal policy is the arc, for any entering node, that will minimize total distance to the destination at this stage.
12
Stage 2
M2-8
MODULE 2 Dynamic Programming
Figure M2.6 may also be helpful in understanding some of the terminology used in the discussion of dynamic programming. M2.4
DYNAMIC PROGRAMMING NOTATION
An input, decision, output, and return are specified for each stage.
The input to one stage is the output from another stage.
In addition to dynamic programming terminology, we can also use mathematical notation to describe any dynamic programming problem. This helps us to set up and solve the problem. Consider stage 2 in the George Yates dynamic programming problem first discussed in Section M2.2. This stage can be represented by the diagram shown in Figure M2.7 (as could any given stage of a given dynamic programming problem). As you can see, for every stage, we have an input, decision, output and return . Look again at stage 2 for the George Yates problem in Figure M2.6. The input to this stage is s 2, which consists of nodes 2, 3, and 4. The decision at stage 2, or choosing which arc will lead to stage 1, is represented by d 2. The possible arcs or decisions are 4–5, 3–5, 3–6, 2–5 and 2–6. The output to stage 2 becomes the input to stage 1. The output from stage 2 is s 1. The possible outputs from stage 2 are the exiting nodes, nodes 5 and 6. Finally, each stage has a return. For stage 2, the return is represented by r 2. In our shortest-route problem, the return is the distance along the arcs in stage 2. These distances are 10 miles for arc 4–5, 12 miles for arc 3–5, 6 miles for arc 3–6, 4 miles for arc 2–5 and 10 miles for arc 2–6. The same notation applies for the other stages and can be used at any stage. In general, we use the following notation for these important concepts: s n
=
input to stage n
(M2-1)
dn
=
decision at stage n
(M2-2)
rn
=
return at stage n
(M2-3)
You should also note that the input to one stage is also the output from another stage. For example, the input to stage 2, s 2, is also the output from stage 3 (see Figure M2.7). This leads us to the following equation: sn −1
FIGURE
=
output from stage n
(M2-4)
M2.7
Input, Decision, Output, and Return for Stage 2 in George Yates’s Problem
Decision d 2
Stage 2 Input s 2
Output s 1
Return r 2
M2.5: Knapsack Problem A transformation function allows us to go from one stage to another.
M2-9
The final concept is transformation . The transformation function allows us to go from one stage to another. The transformation function for stage 2, t 2, converts the input to stage 2, s 2, and the decision made at stage 2, d 2, to the output from stage 2, s 1. Because the transformation function depends on the input and decision at any stage, it can be represented as t 2 (s 2, d 2). In general, the transformation function can be represented as follows: tn
=
transformation function at stage n
(M2-5)
The following general formula allows us to go from one stage to another using the transformation function: sn −1
The total return function allows us to keep track of profits and costs.
M2.5
= t n (sn , dn )
(M2-6)
Although this equation may seem complex, it is really just a mathematical statement of the fact that the output from a stage is a function of the input to the stage and any decisions made at that stage. In the George Yates shortest-route problem, the transformation function consists of a number of tables. These tables show how we could progress from one stage to another in order to solve the problem. For more complex problems, we need to use dynamic programming notation instead of tables. Another useful quantity is the total return at any stage. The total return allows us to keep track of the total profit or costs at each stage as we solve the dynamic programming problem. It can be given as follows: fn
=
total return at stage n
(M2-7)
KNAPSACK PROBLEM The “knapsack problem” involves the maximization or the minimization of a value, such as profits or costs. Like a linear programming problem, there are restrictions. Imagine a knapsack or pouch that can only hold a certain weight or volume. We can place different types of items in the knapsack. Our objective is to place items in the knapsack to maximize total value without breaking the knapsack because of too much weight or a similar restriction.
Types of Knapsack Problems There are many kinds of problems that can be classified as knapsack problems. Choosing items to place in the cargo compartment of an airplane and selecting which payloads to put on the next NASA Space Shuttle are examples. The restriction can be volume, weight, or both. Some scheduling problems are also knapsack problems. For example, we may want to determine which jobs to complete in the next two weeks. The two-week period is the knapsack, and we want to load it with jobs in such a way as to maximize profits or minimize costs. The restriction is the number of days or hours during the two-week period.
Roller’s Air Transport Service Problem The objective of this problem is to maximize profits.
Rob Roller owns and operates Roller’s Air Transport Service, which ships cargo by plane to most large cities in the United States and Canada. The remaining capacity for one of the flights from Seattle to Vancouver is 10 tons. There are four different items that Rob can ship between Seattle and Vancouver. Each item has a weight in tons, a net profit in thousands of dollars, and a total number of that item that is available for shipping. This information is presented in Table M2.2. Problem Setup Roller’s Air Transport problem is an ideal problem to solve using dynamic programming. Stage 4 will be item 1, stage 3 will be item 2, stage 2 will be item 3, and stage
M2-10
MODULE 2 Dynamic Programming
TABLE M2.2
Items to Be Shipped
➠ MODELING IN THE REAL WORLD Defining the Problem
Developing a Model
ITEM
WEIGHT
PROFIT/UNIT
NUMBER AVAILABLE
1
1
$3
6
2
4
9
1
3
3
8
2
4
2
5
2
Reducing Electric Production Costs Using Dynamic Programming
The Southern Company, with service areas in Georgia, Alabama, Mississippi, and Florida, is a major provider of electric service, with about 240 generating units. In recent years, fuel costs have increased faster than other costs. Annual fuel costs are about $2.5 billion, representing about one-third of total expenses for the Southern Company. The problem for the Southern Company is to reduce total fuel costs.
To deal with this fuel cost problem, the company developed a state-of-the-art dynamic programm ing model. The dynamic programming model is embedded in the Wescouger optimization program, which is a computer program used to control electric generating units and reduce fuel costs through better utilization of existing equipment.
Acquiring Input Data
Data were collected on past and projected electric usage. In addition, daily load/generation data were analyzed. Load/generation charts were used to investigate the fuel requirements for coal, nuclear, hydroelectric, and gas/oil.
Developing a Solution
The solution of the dynamic programming model provides both short-term planning guidelines and long-term fuel usage for the various generating units. Optimal maintenance schedules for generating units are obtained using Wescouger.
Testing the Solution
Analyzing the Results
Implementing the Results
To test the accuracy of the Wescouger optimization program, Southern used a real-time economic dispatch program. The results were a very close match. In addition, the company put the solution through an acid test, in which seasoned operators compared the results against their intuitive judgment. Again, the results were consistent.
The results were analyzed in terms of their impact on the use of various fuels, the usage of various generating units, and maintenance schedules for generating units. Analyzing the results also revealed other needs. This resulted in a full-color screen editing routine, auxiliary programs to automate data input, and software to generate postanalysis reports.
The Southern Company implemented the dynamic programming solution. Over a seven-year period, the results saved the company over $140 million. Source: S. Erwin et al. “Using an Optimization Software to Lower Overall Electric Production Costs for Southern Company,” Interfaces 21, 1 (January–February 1991): 27–41.
M2.5: Knapsack Problem TABLE M2.3
ITEM
STAGE
1
4
2
3
3
2
4
1
Relationship Between Items and Stages
M2-11
1 will be item 4. This is shown in Table M2.3. During the solution, we will be using stage numbers. Roller’s Air Transport problem can be represented graph ically (see Figure M2.8). As you can see, each item is represented by a stage. Look at stage 4, which is item 1. The total weight that can be used is represented by s 4. This amount is 10 because we haven’t assigned any units to be shipped at this time. The decision at this stage is d 4 (the number of units of item 1 we will ship). If d 4 is 1, for example, we will be shipping 1 unit of item 1. Also note that r 4 is the return or profit at stage 4 (item 1). If we ship 1 unit of item 1, the profit will be $3 (see Table M2.2). As mentioned previously, the decision variable d n , represents the number of units of each item (stage) that can be shipped. Looking back at the original problem, we see that the problem is constrained by the number of units. This is summarized in the following table: STAGE
MAXIMUM VALUE OF DECISION
4
6
3
1
2
2
1
2
The Transformation Functions Next, we need to look at the transformation function. The general transformation function for knapsack problems follows: sn −1
=
( an
×
sn )
+
( bn
×
dn )
+
cn
Note that a n , b n , and c n are coefficients for the problem and that d n represents the decision at stage n . This is the number of units to ship at stage s n . FIGURE
Decisions
M2.8
d 4
Roller’s Air Transport Service Problem s 4
Stage 4 (Item 1)
r 4
d 3
s 3
Stage 3 (Item 2)
d 2
s 2
r 3
Stage 2 (Item 3)
r 2
Returns
d 1
s 1
Stage 1 (Item 4)
r 1
s 0
M2-12
MODULE 2 Dynamic Programming
The following chart shows the transformation coefficients for Rob Roller’s transport problem:
COEFFICIENTS OF TRANSITION FUNCTION STAGE
a n
b n
c n
4
1
Ϫ1
0
3
1
Ϫ4
0
2
1
Ϫ3
0
1
1
Ϫ2
0
First note that s 4 is 10, the total weight that can be shipped. Because s 4 represents the first item, all 10 tons can be utilized. The transformation equations for the four stages are as follows: s3
=
s4
− 1d 4
stage 4
(a)
s2
=
s3
−
4d 3
stage 3
(b)
s1
=
s2
−
3d 2
stage 2
(c)
s0
= s1 −
2d 1
stage 1
(d)
Consider stage 3. Equation b reveals that the number of tons still available after stage 3, s 2, is equal to the number of tons available before stage 3, s 3, minus the number of tons shipped at stage 3, 4d 3. In this equation, 4d 3 means that each item at stage 3 weighs 4 tons. The Return Functions Next, we will look at the return function for each stage. This is the general form for the return function: rn
=
( an
×
sn )
+
( bn
×
dn )
+
cn
Note that a n , b n , and c n are the coefficients for the return function. Using this general form of the return function, we can put the return function values in the following table:
COEFFICIENTS OF RETURN FUNCTION
DECISIONS STAGE
LOWER
UPPER
a n
b n
c n
4
0
≤
d n ≤
6
0
3
0
3
0
≤
d n ≤
1
0
9
0
2
0
≤
d n ≤
2
0
8
0
1
0
≤
d n ≤
2
0
5
0
M2.5: Knapsack Problem
M2-13
The lower value for each decision is zero, and the upper value is the total number available. The b n coefficient is the profit per item shipped. The actual return functions are: r4
=
3d 4
r3
=
9d 3
r2
=
8d 2
r1
=
5d 1
Stage-by-Stage Solution As you would expect, the return at any stage r n , is equal to the profit per unit at that stage multiplied by the number of units shipped at that stage, d n . With this information, we can solve Roller’s Air Transport problem, starting with stage 1 (item 4). The following table shows the solution for the first stage. You may wish to refer to Figure M2.8 for this discussion.
STAGE 1
STAGE 1 S 1
d 1
r 1
s 0
f 0
f 1
S 1
d 1
r 1
s 0
f 0
f 1
0
0
0
0
0
0
7
0
0
0
0
0
1
0
0
0
0
0
1
5
0
0
5
2
0
0
0
0
0
2
10
0
0
10
1
5
0
0
5
0
0
0
0
0
0
0
0
0
0
1
5
0
0
5
1
5
0
0
5
2
10
0
0
10
0
0
0
0
0
0
0
0
0
0
1
5
0
0
5
1
5
0
0
5
2
10
0
0
10
2
10
0
0
10
0
0
0
0
0
0
0
0
0
0
1
5
0
0
5
1
5
0
0
5
2
10
0
0
10
2
10
0
0
10
0
0
0
0
0
1
5
0
0
5
2
10
0
0
10
3 4
5
6
We consider all possibilities.
f 1 is the total return at the first stage. The total return before the first stage is f 0.
8
9
10
Because we don’t know how many tons will be available at stage 1, we must consider all possibilities. Thus the number of tons available at stage 1, s 1, can vary from 1 to 10. This is seen in the first column of numbers for stage 1. The number of units that we ship at stage 1, d 1, can vary from 0 to 2. We can’t go over 2 because the number available is only 2. For any decision we compute the return at stage 1, r 1, by multiplying the number of items shipped by 5, the profit per item. The profit at this stage will be 0, 5, or 10, depending on whether 0, 1 or 2 items are shipped. Note that the total return at this stage, f 1, is the same as r 1 because this is the only stage we are considering so far. Also note that the total return before stage 1, f 0, is 0 because this is the beginning of the solution and we are shipping nothing at this point.
M2-14
MODULE 2 Dynamic Programming
The solution for stage 1 shows the best decision, the return for this stage, and the total return given all possible number of tons available (0 to 10 tons). Using the results of stage 1, we can now proceed to stage 2. The solution for stage 2 is as follows:
STAGE 2
STAGE 2
S 2
d 2
r 2
s 1
f 1
f 2
S 2
d 2
r 2
s 1
f 1
f 2
0
0
0
0
0
0
7
0
0
7
10
10
1
0
0
1
0
0
1
8
4
10
18
2
0
0
2
2
5
2
16
1
0
16
3
0
0
3
5
5
0
0
8
10
10
1
8
0
0
8
1
8
5
10
18
0
0
4
10
10
2
16
2
5
21
1
8
1
0
8
0
0
9
10
10
0
0
5
10
10
1
8
6
10
18
1
8
2
5
13
2
16
3
5
21
0
0
6
10
10
0
0
10
10
10
1
8
3
5
13
1
8
7
10
18
2
16
0
0
16
2
16
4
10
26
4 5 6
8
9
10
The solution for stage 2 is found in exactly the same way as for stage 1. At stage 2 we still have to consider all possible number of tons available (from 0 to 10). See the s 2 column (the first column). At stage 2 (item 3) we still only have 2 units that can be shipped. Thus d 2 (second column) can range from 0 to a maximum of 2. The return for each s 2 and d 2 combination at stage 2, r 2, is shown in the third column. These numbers are the profit per item at this stage, 8, times the number of items shipped. Because items shipped at stage 2 can be 0, 1, or 2, the profit at this stage can be 0, 8, or 16. The return for stage 2 can also be computed from the return function: r 2 = 8d 2. Now look at the fourth column, s 1, which lists the number of items available after stage 2. This is also the number of items available for stage 1. To get this number, we have to subtract the number of tons we are shipping at stage 2 (which is the tonnage per unit times the number of units) from the number of tons available before stage 2 (s 2). Look at the row in which s 2 is 6 and d 2 is 1. We have 6 tons available before stage 2, and we are shipping 1 item, which weighs 3 tons. Thus we will have 3 tons still available after this stage. The s 1 values can also be determined using the transformation function, which is s 1 = s 2 – 3d 2. The last two columns of stage 2 contain the total return. The return after stage 1 and before stage 2 is f 1. These are the same values that appeared under the f 1 column for stage 1. The return after stage 2 is f 2. It is equal to the return from stage 2 plus the total return before stage 2. Stage 3 is solved in the same way as stages 1 and 2. The following table presents the solution for stage 3; look at each row and make sure that you understand the meaning of each value.
M2-15
M2.5: Knapsack Problem STAGE 3
STAGE 3
s 3
d 3
r 3
s 2
f 2
f 3
s 3
d 3
r 3
s 2
f 2
f 3
0
0
0
0
0
0
7
0
0
7
18
18
1
0
0
1
0
0
1
9
3
8
17
2
0
0
2
2
5
0
0
8
21
21
3
0
0
3
8
8
1
9
4
10
19
4
0
0
4
10
10
0
0
9
21
21
1
9
0
0
9
1
9
5
13
22
0
0
5
13
13
0
0
10
26
26
1
9
1
0
9
1
9
6
16
25
0
0
6
16
16
1
9
2
5
14
5 6
8 9 10
Now we solve the last stage of the problem, stage 4. The following table shows the solution procedure: STAGE 4 s 4
d 4
r 4
s 3
f 3
f 4
10
0
0
10
26
26
1
3
9
22
25
2
6
8
21
27
3
9
7
18
27
4
12
6
16
28
5
15
5
13
28
6
18
4
10
28
The first thing to note is that we only have to consider one value for s 4, because we know the number of tons available for stage 4; s 4 must be 10 because we have all 10 tons available. There are six possible decisions at this stage, or six possible values for d 4, because the number of available units is 6. The other columns are computed in the same way. Note that the return after stage 4, f 4, is the total return for the problem. We see that the highest profit is 28. We also see that there are three possible decisions that will give this level of profit, shipping 4, 5, or 6 items. Thus we have alternate optimal solutions. One possible solution is as follows: FINAL SOLUTION STAGE
OPTIMAL DECISION
OPTIMAL RETURN
4
6
18
3
0
0
2
0
0
1
2
10
Total
8
28
M2-16
MODULE 2 Dynamic Programming
We start with shipping 6 units at stage 4. Note that s 3 is 4 from the stage 4 calculations, given that d 4 is 6. We use this value of 4 and go to the stage 3 calculations. We find the rows in which s 3 is 4 and pick the row with the highest total return, f 3. In this row d 3 is 0 items with a total return ( f 3) of 10. As a result, the number of units available, s 2, is still 4. We next go to the calculations for stage 2 and then stage 1 in the same way. This gives us the optimal solution already described. See if you can find one of the alternate optimal solutions.
SUMMARY Dynamic programming is a flexible, powerful technique. A large number of problems can be solved using this technique, including the shortest-route and knapsack problems. The shortest-route problem finds the path through a network that minimizes total distance traveled, and the knapsack problem maximizes the value or return. An example of a knapsack problem is to determine what to ship in a cargo plane to maximize total profits given the weight and size constraints of the cargo plane. The dynamic programming technique requires four steps: (1) Divide the original problem into stages, (2) solve the last stage first, (3) work backward solving each subsequent stage, and (4) obtain the optimal solution after all stages have been solved.
Dynamic programming requires that we specify stages, state variables, decision variables, decision criteria, an optimal policy, and a transformation function for each specific problem. Stages are logical subproblems. State variables are possible input values or beginning conditions. The decision variables are the alternatives that we have at each stage, such as which route to take in a shortest-route problem. The decision criterion is the objective of the problem, such as finding the shortest route or maximizing total return in a knapsack problem. The optimal policy helps us obtain an optimal solution at any stage, and the transformation function allows us to go from one stage to the next.
GLOSSARY Decision Criterion. A statement concerning the objective of a dynamic programming problem.
Stage. A logical subproblem in a dynamic programming problem.
Decision Variable. The alternatives or possible decisions that exist at each stage of a dynamic programming problem.
State Variable. A term used in dynamic programming to describe the possible beginning situations or conditions of a stage.
Dynamic Programming. A quantitative technique that works backward from the end of the problem to the beginning of the problem in determining the best decision for a number of interrelated decisions.
Transformation. An algebraic statement that shows the relationship between stages in a dynamic programming problem.
Optimal Policy. A set of decision rules, developed as a result of the decision criteria, that gives optimal decisions at any stage of a dynamic programming problem.
KEY EQUATIONS (M2-1) s n
=
Input to stage n
The input to stage n . This is also the output from stage n + 1. (M2-2) n
=
Decision at stage n
The decision at stage n . (M2-3) rn
=
Return at stage n
The return function, usually profit or loss, at stage n . (M2-4) s n −1
=
Output from stage n
The output from stage n. This is also the input to stage n Ϫ 1.
(M2-5) t n
=
Transformation function at stage n
A transformation function that allows us to go from one stage to another. (M2-6) s n −1
=
t n (s n , dn )
The general relationship that shows how the output from any stage is a function of the input to the stage and the decisions made at that stage. (M2-7) f n
=
Total return at stage n
This equation gives the total return (profit or costs) at any stage. It is obtained by summing the return at each stage,r n .
Solved Problem
M2-17
SOLVED PROBLEM Solved Problem M2-1 Lindsey Cortizan would like to travel from the university to her hometown over the holiday season. This road map from the university (node 1) to her home (node 7) is shown in Figure M2.9. What is the best route that will minimize total distance traveled?
FIGURE
M2.9
4
Road Map for Lindsey Cortizan
39
20 6 13
28 18
1
3
7
36 10
22
18
5
2
Solution The solution to this problem is identical to the one presented earlier in the chapter. First, the problem is broken into three stages. See the network in Figure M2.10. Working backward, we start by solving stage 3. The closest and only distance from node 6 to 7 is 13, and the closest and only distance from node 5 to node 7 is 10. We proceed to stage 2. The minimum distances are 52, 41, and 28 from node 4, node 3, and node 2 to node 7. Finally, we complete stage 3. The optimal solution is 50. The shortest route is 1–2–5–7 and is shown in the following network. This problem can also be solved using tables, as shown following the network solution.
FIGURE
Problem type
minimization network
Number of stages Transition function type
3 s n –1 = s n – d n
Recursion function type
f n = r n + f n –1
M2.10
52
Solution for the Lindsey Cortizan Problem
4
39
13
20 50 1
41 18
6 13
28
3
7
36
10
22 18
5
2 10
28 Stage 3
Stage 2
Stage 1
M2-18
MODULE 2 Dynamic Programming
STAGE
STAGE
NUMBER OF DECISIONS
3
3
2
4
1
2
STARTING NODE
ENDING NODE
RETURN VALUE
1
2
22
1
3
18
1
4
20
2
5
18
2
6
36
3
6
28
4
6
39
5
7
10
6
7
13
3
2
1
STAGE 1 s 1
d 1
r 1
s 0
f 0
f 1
6
6–7
13
7
0
13
5
5–7
10
7
0
10
STAGE 2 s 2
d 2
r 2
s 1
f 1
f 2
4
4–6
39
6
13
52
3
3–6
28
6
13
41
2
2–6
36
6
13
49
2–5
18
5
10
28
STAGE 3 s 3
d 3
r 3
s 2
f 2
f 3
1
1–4
20
4
52
72
1–3
18
3
41
59
1–2
22
2
28
50
FINAL SOLUTION STAGE
OPTIMAL DECISION
OPTIMAL RETURN
3
1–2
22
2
2–5
18
1
5–7
10
Total
50
Self-Test
➠
M2-19
SELF-TEST I
Before taking the self-test, refer back to the learning objectives at the beginning of the module, the notes in the margins, and the glossary at the end of the module.
Use the key at the back of the book to correct your answers. I Restudy pages that correspond to any questions that you answered incorrectly or material you feel uncertain about. I
1. Dynamic programming divides problems into a. nodes. b. arcs. c. decision stages. d. branches. e. variables. 2. Possible beginning situations or conditions of a dynamic programming problem are called a. stages. b. state variables. c. decision variables. d. optimal policy. e. transformation. 3. The statement concerning the objective of a dynamic programming problem is called a. stages. b. state variables. c. decision variables. d. optimal policy. e. decision criterion. 4. The first step of a dynamic programming problem is to a. define the nodes. b. define the arcs. c. divide the original problem into stages. d. determine the optimal policy. e. none of the above. 5. In working a problem with dynamic programming, we usually a. start at the first part of the problem and work forward to the next parts. b. start at the end of the problem and work backward. c. start at the most expensive part of the problem. d. start at the least expensive part of the problem. 6. An algebraic statement that reveals the relationship between stages is called a. the transformation. b. state variables. c. decision variables. d. the optimal policy. e. the decision criterion. 7. In this chapter, dynamic programming is used to solve what type of problem? a. quantity discount b. just-in-time inventory c. shortest-route d. minimal spanning tree e. maximal flow 8. In dynamic programming terminology, a period or logical subproblem is called a. the transformation. b. a state variable.
9.
10.
11.
12.
13.
14.
c. a decision variable. d. the optimal policy. e. a stage. The statement that the distance from the beginning stage is equal to the distance from the preceding stage to the last node plus the distance from the g iven stage to the preceding stage is called a. the transformation. b. state variables. c. decision variables. d. the optimal policy. e. stages. In dynamic programming, s n is a. the input to the stage n . b. the decision at stage n . c. the return at stage n . d. the output of stage n . e. none of the above. The relationship that the distance from the beginning stage is equal to the distance from the preceding stage to the last node plus the distance for the given stage to the preceding stage is used to solve which type of problem? a. knapsack b. JIT c. shortest-route d. minimal spanning tree e. maximal flow In using dynamic programming to solve a shortest route problem, the distance from one point to the next would be called a a. state. b. stage. c. return. d. decision. In using dynamic programming to solve a shortest route problem, the decision variables at one stage of the problem would be a. the distances from one node to the next. b. the possible arcs or routes that can b e selected. c. the number of possible destination nodes. d. the entering nodes. In using dynamic programming to solve a shortest route problem, the entering nodes would be called a. the stages. b. the state variables. c. the returns. d. the decision variables.
M2-20
MODULE 2 Dynamic Programming
DISCUSSION QUESTIONS AND PROBLEMS Discussion Questions
M2-10 Solve the shortest-route problem of Figure M2.11. M2-11 Solve the shortest-route problem of Figure M2.12. M2-12 Mail Express, an overnight mail service, delivers mail to customers throughout the United States, Canada, and Mexico. Fortunately, Mail Express has additional capacity on one of its cargo planes. To maximize profits, Mail Express takes shipments from local manufacturing plants to warehouses for other companies. Currently, there is room for another 6 tons. The following table shows the items that can be shipped, their weights, the expected profit for each, and the number of available parts. How many units of each item do you suggest that Mail Express ship?
M2-1 What is a stage in dynamic programming? M2-2 What is the difference between a state variable and a decision variable? M2-3 Describe the meaning and use of a decision criterion. M2-4 Do all dynamic programming problems require an optimal policy? M2-5 Why is transformation important for dynamic programming problems?
Problems M2-6 Refer to Figure M2.1. What is the shortest route between Rice and Dixieville if the road between Hope and Georgetown is improved and the distance is reduce to 4 miles? M2-7 Due to road construction between Georgetown and Dixieville, a detour must be taken through country roads (Figure M2.1). Unfortunately, this detour has increased the distance from Georgetown to Dixieville to 14 miles. What should George do? Should he take a different route? M2-8 The Rice Brothers have a gold mine between Rice and Brown. In their zeal to find gold, they have blown up the road between Rice and Brown. The road will not be in service for five months. What should George do? Refer to Figure M2.1. M2-9 The Rice Brothers (Figure M2.1) wish to determine the potential savings from using the shortest route from Rice to Dixieville rather than just randomly selecting any route. Use dynamic programming to find the longest route from Rice to Dixieville. How much farther is this than the shortest route?
FIGURE
ITEMS TOBE SHIPPED ITEM
WEIGHT (TONS)
PROFIT/UNIT
NUMBER AVAILABLE
1
1
$3
6
2
2
9
1
3
3
8
2
4
1
2
2
M2-13 Leslie Bessler must travel from her hometown to Denver to see her friend Austin. Given the road map of Figure M2.13, what route will minimize the distance that she travels? M2-14 An air cargo company has the following shipping requirements. Two planes are available with a total capacity of 11 tons. How many of each item should be shipped to maximize profits?
M2.11
4
(for Problem M2-10)
2 4
10 2
5 1
7
4
6
12 5 6
4
9
11 3
4 8
6
10 6
Discussion Questions and Problems FIGURE
M2.12
3
7
6
5
(for Problem M2-11)
10
2
3
M2-21
6
5
5
9
1
3 2
3
4
2
3
8
2
11
4
2
3
4 7
ITEMS TOBE SHIPPED ITEM
WEIGHT (TONS)
PROFIT/UNIT
NUMBER AVAILABLE
1
1
$3
6
2
2
9
1
3
3
8
2
4
2
5
5
5
5
8
6
6
1
2
2
M2-18 In Problem M2-16, what is the shortest distance from node 6 to the ending node? How does this change if the road between node 6 and node 11 is no longer in service (see Problem M2-17)? M2-19 Paige Schwartz is planning to go camping in Big Bend National Park. She will carry a backpack with food and other items, and she wants to carry no more than 18 pounds of food. The four possible food items she is considering each have certain nutritional ingredients. These items and their weights are as follows:
M2-15 Because of a new manufacturing and packaging procedure, the weight of item 2 in Problem M2-14 can be cut in half. Does this change the number or types of items that can be shipped by the air transport company? M2-16 What is the shortest route through the network of Figure M2.14? M2-17 The road between node 6 and node 11 is no longer in service due to construction. (Refer to Problem M2-16). What is the shortest route g iven this situation?
FIGURE
ITEM
WEIGHT (POUNDS )
NUTRITIONAL UNITS
A
3
600
B
4
1,000
C
5
1,500
D
2
300
How many of each item should Paige carry if she wishes to maximize the total nutritional units?
M2.13
(for Problem M2-13)
4
3
3
9
2
6 2
3
3
4 1
1
11
12 3
2
4 5
3
3 2
8
4
3
3 10
2 7
M2-22
MODULE 2 Dynamic Programming
FIGURE
M2.14
13
3
(for Problem M2-16) 3
8
16 12
3 4 4 1
2
4
2
4
18
7 11
15 6
19 2
4 14
2
3
20
17
5
4
2
2
3 10
5
2
3
3 3
3
4
3 4
3
4
9
➠ CASE STUDY United Trucking Like many trucking operations, United Trucking got started with one truck and one owner—Judson Maclay. Judson is an individualist and has always liked to do things his way. He was a fast driver, and many people called the 800 number on the back of his truck when he worked for Hartmann Trucking. After two years with Hartmann and numerous calls about his bad driving, Judson decided to go out on his own. United Trucking was the result. In the early days of United Trucking, Judson was the only driver. On the back of his truck was the message: How do you like my driving? Call 1-800-AMI-FAST. He was convinced that some people actually tried to call the number. Soon, a number of truck operators had the same or similar messages on the back of their trucks. After three years of operation, Judson had 15 other trucks and drivers working for him. He traded his driving skills for office and management skills. Although 1-800-AMI-FAST was no longer visible, Judson decided to never place an 800 number on the back of any of his trucks. If someone really wanted to complain, they could look up United Trucking in the phone book. Judson liked to innovate with his trucking company. He knew that he could make more money by keeping his trucks full. Thus he decided to institute a discount trucking service. He gave substantial discounts to customers that would accept delivery to the West Coast within two weeks. Customers got a great price, and he made more money and kept his trucks full. Over time,
Judson developed steady customers that would usually have loads to go at the discounted price. On one shipment, he had an available capacity of 10 tons in several trucks going to the West Coast. Ten items can be shipped at discount. The weight, profit, and number of items available are shown in the following table: WEIGHT (TONS)
PROFIT/UNIT
AVAILABLE
1
1
$10
2
2
1
10
1
3
2
5
3
4
1
7
20
5
3
25
2
6
1
11
1
7
4
30
2
8
3
50
1
9
1
10
2
10
1
5
4
ITEM
Discussion Questions 1. 2.
What do you recommend Judson should do? If the total available capacity was 20 tons, how would this change Judson’s decision?
INTERNET CASE STUDY See our Internet home page at www.prenhall.com/render for this additional case study: Briarcliff Electronics.
Bibliography
M2-23
BIBLIOGRAPHY Bellman, R. E. Dynamic Programming, Princeton, NJ: Princeton University Press, 1957.
Howard, R. A. Dynamic Programming . Cambridge, MA: The MIT Press, 1960.
Bourland, Karla, “Parallel-Machine Scheduling with Fractional Operator Requirements,” IEE Transactions (September 1994): 56.
Ibarake, Toshihide and Yuichi Nakamura,“A Dynamic Programming Method for Single Machine Scheduling,” European Journal of Operations Research (July 1994): 72–82.
Elmaghraby, Salah, “Resource Allocation via Dynamic Programming,” European Journal of Operations Research (January 1993): 199–215. El-Rayes, Khaled et al. “Optimized Scheduling for Highway Construction,” Transactions of AACE International (February 1997): 311–315. Hillard, Michael R. et al.“Scheduling the Operation Desert Storm Airlift: An Advanced Automated Scheduling Support System,” Interfaces 21, 1 (January–February 1992): 131–146.
Idem, Fidelis and A. M. Reisman.“An Approach to Planning for Physician Requirements in Developing Countries Using Dynamic Programming,” Operations Research (July 1990): 607–18. Stokes, Jeffrey et al. “Optimal Marketing of Nursery Crops from Container-Based Production Systems,” American Journal of Agricultural Economics (February 1997): 235–246.
